Edit documentation in TW and use Github API to create PRs directly?

[saqimtiaz]

Edit (27-10-2021): there is a working demo

This is a bit of a pie in the sky idea at present prompted by syncers & savers related thoughts and the FileUploads plugin.

It would be worth investigating if the following is possible:

• User has forked the Github repo and starts with a local copy of TiddlyWiki.com and fills out Github saver credentials.
  • We might even be able to fork the repo for them in one of the later steps as long as they have Github credentials.
  • We could also have a “clone TW.com on TiddlyHost or other hosting” one click option to avoid the need for a local copy
• User edits one or several documentation tiddlers
• User selects via a UI the documentation tiddlers to submit as a PR, and writes a message for the PR and hits submit:
  • These tiddlers are mapped to corresponding files in the git directory hierarchy (tw.com already does this via the ViewToolbar => Info panel => Sources tab)
  • Via the Github API a new branch is automatically created and the files saved using the FileUploads plugin or something similar
  • A PR request is initiated on Github on behalf of the user using the text/message the user specified.

I think all the pieces exist for this to be possible. See for example:

GitHub - gr2m/octokit-plugin-create-pull-request: Octokit plugin to create a...

[saqimtiaz]

Unless I am missing something, this is very much doable and not more than a long weekend’s project assuming one has one of those to spare.

[Mark_S]

@saqimtiaz

These tiddlers are mapped to corresponding files in the git directory hierarchy (tw.com already does this via the Info => Sources tab)

Where is this? What tab are you referring to?

@saqimtiaz

Via the Github API a new branch is automatically created and the files saved using the FileUploads plugin or something similar

Is this the same API used for the GitSaver ? Or is there something new?

Thanks!

[saqimtiaz]

@Mark_S

Where is this? What tab are you referring to?

Sorry, view toolbar for any tiddler on tiddlywiki.com, open the info panel and go to the Sources tab.

@Mark_S

Is this the same API used for the GitSaver ? Or is there something new?

Same underlying API, we are currently just using a fraction of it. Plugins/libraries like the one I linked above make using it to do complex things pretty easy.

[saqimtiaz]

I have confirmed that the concept is sound and that we just need to connect all the pieces. This PR was generated from TiddlyWiki:

pull request title

It would even be possible to work directly on TiddlyWiki.com, saving temporarily with local storage and generating a PR. Being able to generate your own copy of tiddlywiki.com to work from would be easier for more complex changes.

[saqimtiaz]

The one missing piece, and this would be useful regardless of this concept, is to setup the $:/config/FileSystemPaths tiddler for the tw5.com edition so that new documentation tiddlers are mapped to the appropriate directory. This is currently a pain even when working with node.js as new tiddlers have to be manually moved to the correct directory.

[pmario]

@saqimtiaz

This is currently a pain even when working with node.js as new tiddlers have to be manually moved to the correct directory.

Yea, I did think the same thing. … There has been some contribution from a community member for the German translation. And having all the translated tiddlers in 1 directory was a real pain point.

I needed to search, where the original tiddlers have been and move them into a similar position.
The problem with $:/config/FileSystemPaths is, that most tiddlers at tiddlywiki.com have several tags. … So it’s very hard to define, where the tiddlers end up.

My path settings look like this atm: TiddlyWiki5/FileSystemPaths.txt at german-translation · pmario/TiddlyWiki5 · GitHub … We add new paths as they are needed. …

IMO it would be nice if tiddlywiki-com would define a “reference setting”, which covers ALL tiddlers in the docs.

This would also make it easier for new users. If they use the right tags, their tiddlers will automagically end up in the right subdirectory.

[saqimtiaz]

Ok so I think these are the pieces we need:

• A UI that asks for:
  • the personal access token (this is unavoidable and will be the hardest part for new contributors)
  • a PR title and body
  • allows you to select a list of tiddlers to include from docs tiddlers. Would be helpful perhaps to show which tiddlers have been edited by the user.
• A submit button that:
  • Sends a widget message, let’s call it tm-makepr with a list of tiddlers as a parameter in title list format (so a filter).
  • A root widget handler that intercepts this message and for each title specified:
    • calculates the file path for the tiddler. For existing tiddlers we can get this from $:/config/OriginalTiddlerPaths
    • generates the content for that file in .tid format
    • initiates a request using the Create Pull Request plugin for Octokit with the file data as payload.
    • either displays an error message, for example if the access token was expired. Or shows a link to the PR on Github.

I only had a quick look but I did not find a packaged version of the Octokit library. I am sure we could bundle it if we tried, but loading it from CDN on demand is also feasible (as I did for testing) and would keep the plugin light. After all users will need to be online to create a PR anyway.

[pmario]

@saqimtiaz

I only had a quick look but I did not find a packaged version of the Octokit library. I am sure we could bundle it if we tried, but loading it from CDN on demand is also feasible (as I did for testing) and would keep the plugin light. After all users will need to be online to create a PR anyway.

That’s right. The user has to be online, if they want to create a PR. So a CDN should be fine.

[pmario]

@saqimtiaz

A UI that asks for:

• the personal access token (this is unavoidable and will be the hardest part for new contributors)

I think, this would be a 3 minute video, to show how it will be used. … As a side effect it will also help users to use the GitHub saver.

[saqimtiaz]

Here is what the really ugly prototype looks like:

As an aside, it would be nice to have a wikitext affordance to check if a password has been set in local storage. Only to verify if it exists, not to retrieve it.

[TW_Tones]

@saqimtiaz this is a very good idea. I am sure we can get a sufficient number of developers and enthusiasts to utilise this method. This will itself help build the documentation.

However I am not sure a novice or even regular users/designers will adopt this. For this reason I think a/the discourse Documentation category could be a place for contributors to post updated tiddlers that those with this ability may load into their Git activated wiki and submit on behalf of the contributor.

• Thus Being able to annotate such PR’s with the contributor or even a talk.tiddlywiki.org link makes sense.
• To make it easy share instructions how to export a modified tiddler or tiddlers for application to the documentation.
• Consider the possibility of being able to submit changes to other wiki repositories/trees.
  ** Eg a links repository, extended documentation or references, self submitted references etc…

Perhaps in the future a publicly accessible GuitHub could allow “anon” contributions into a repository from a website. Still demanding a review before actioned.

Good work @saqimtiaz, looking out for community needs.

[saqimtiaz]

@pmario @jeremyruston @boris and @everyone

This is what a PR generated via TiddlyWiki looks like:
(edit: replaced with a better example)

Docs: remove erroneous tag from enlist-input operator.

What I did:

• Edited the tiddler “enlist-input Operator” in TiddlyWiki
• In the interface shown below:
  • Added a PR title and PR message
  • Added the tiddler title
  • clicked on “Make a PR”
  • smiled like crazy and realized I am up way too late.

I think even I would find this an easier way to make docs PRs than my current node.js workflow.

Good night!

Edit: if the user does not already have a fork of the TiddlyWiki repo it is automatically created.

[saqimtiaz]

@pmario asks on Github:

Is it possible to add changes (new commits) to those tiddlers later on, or is that a new PR?

With the current code it would be a new PR. However, it is possible to implement the ability to add new commits to the same PR as well.

We are using 2 dependencies, a low level library (octokit) that covers the entire Github API and a plugin for it that makes it very easy to create PRs. So either we use the low level library to update the PR or we find a plugin for it.

The real catch is going to be the user side of things and the copy of tw.com that they use.

• we need to make sure they always start with a fresh copy of the site
• if we want to allow updating a PR, then they would need to have saved their changes in TW somewhere
  • unless we use Github actions to generate a TW file for every PR, and then that file can be used to further update the same PR as long as it hasn’t fallen far behind the tw.com branch in terms of changes.
• Or we have a one click option to clone tw.com to say TiddlyHost (with their permission and consideration of cost) as starting point for each PR. The wiki persists at TH while the PR is open.

[pmario]

@saqimtiaz

The real catch is going to be the user side of things and the copy of tw.com that they use.

OK. I haven’t had a look at the dependences yet. …

For the start I would say “keep it simple” and iterate often. So we can deal with several commits to the same PR later.

[saqimtiaz]

@pmario I’ll post a demo soon that everyone can play with. It will create PRs but it will target my fork so as to not spam the main repo while testing

[saqimtiaz]

I have posted a demo:

Demo: updating docs and creating a PR from within TiddlyWiki

A lot of room for refinement as well as constraints in terms of what kinds of PRs are allowed etc, but the basic workflow works. You will need to create a Github Personal Access Token Demo: TiddlyWiki — a non-linear personal web notebook Background: Edit documentation in TW and use Github API to create PRs directly?

[pmario]

@saqimtiaz

@pmario I’ll post a demo soon that everyone can play with. It will create PRs but it will target my fork so as to not spam the main repo while testing

I don’t want to spam your account either. … What do you think, should we create a “throw away” organisation, that we can “spam” with test PRs without collecting too much garbage in our own repos?

[pmario]

@pmario

OK. I haven’t had a look at the dependences yet. …

I did have a closer look at octokit.js … It seems once a PR is created, it will automatically pick up new commits to files in the same “feature branch”. …

So instead of creating a new PR we just need to update the file. … The only thing which seems to be important is the “SHA of the parent commit” … So it may be important what the “create PR workflow” returns. … So a bit more testing is needed.

[saqimtiaz]

@pmario

I don’t want to spam your account either. … What do you think, should we create a “throw away” organisation, that we can “spam” with test PRs without collecting too much garbage in our own repos?

We can, though it really isn’t a problem to spam my fork. I can simply delete my fork and recreate it to get rid of the noise. When I don’t have open PRs I consider forks of public repos to be disposable.

I do have a throwaway account I created for testing CI for twlinks, I can look up the details for that later if need be.

@pmario

I did have a closer look at octokit.js … It seems once a PR is created, it will automatically pick up new commits to files in the same “feature branch”. …

So instead of creating a new PR we just need to update the file. … The only thing which seems to be important is the “SHA of the parent commit” … So it may be important what the “create PR workflow” returns. … So a bit more testing is needed.

Yep if we use the low level octokit it is all doable. The create PR plugin makes creating a PR easier but only returns the PR number. So either we need to use that and the branch name to fetch the SHA for the parent so we can update the same branch, or just handle PR creation ourselves with octokit and not use the plugin. Both would work.

Anyway, I do think we shouldn’t get too caught up on that at the moment and consider that as Step 2, with Step 1 being getting a decent workflow in place for creating a PR in one go (without facilities for updates).

(An important bit with being able to update a PR is that you will need to have the TiddlyWiki you previously used to create it.)