RFI: GitHub Actions Static Publishing Script

oeyoews · November 24, 2022, 8:45am

Sttot · November 24, 2022, 8:58am

The CPL I designed before was a JSON file generator using a CI/CD script.

I recently developed another solution myself to generate HTML static blogs (with metadata) using TiddlyWiki content, the generated blogs are at gk0wk.github.io (all written in Chinese, nothing to see here, just wanted to serve as an example) - the related code I haven’t open sourced yet, because I don’t know in what form to publish it.

BTW: it’s SSR(Server side render) not CSR (client side render)

boris · November 24, 2022, 3:31pm

There are many SSG scripts. I currently use jekyll on my personal site.

Roughly this means putting TW and node in a Docker container and passing the correct switches to TiddlyWiki into an output folder.

The LogSeq Publish action uses a Docker image as starting point logseq-publish/action.yml at main · pengx17/logseq-publish · GitHub

This Jekyll Build action uses no Docker and just JS and Ruby so closer to what TW needs jekyll-action-ts/action.yml at master · limjh16/jekyll-action-ts · GitHub

Mohammad · November 24, 2022, 4:07pm

Great stuff @Sttot. Thank you for sharing.

saqimtiaz · November 25, 2022, 2:22pm

@oeyoews @Sttot we are discussing creating a Github Action or script for TiddlyWiki that can be used to build a static site every time new content is pushed to the repo. We can try to fund raise for this a little (perhaps via Open Collective) to provide a small thank you and some motivation for the developer.

Would either of you, or anyone you know, be interested in working on this?

oeyoews · November 25, 2022, 2:47pm

@saqimtiaz Tiddlywiki repo looks like there are already some scripts to build static site.

which repo?

saqimtiaz · November 25, 2022, 2:52pm

The idea is to create an action that anyone can use in their GitHub repo, to build a static site from their TiddlyWiki and publish it with GitHub Pages.

This is meant to be something for end users to be able to use, and not for the official TW repo.

oeyoews · November 25, 2022, 2:57pm

That would be easy to do if just used scripts, but not as user-friendly as github actions

saqimtiaz · November 25, 2022, 2:58pm

Scripts can be the starting point. If we have a good documented script that accepts different arguments or environmental variables to customize the static rendering (which tiddlers to include etc), it can later be used to create a GitHub Action.

Edit: another option is a template repository with scripts configured that the user can clone to create their own repository with static site building configured.

oeyoews · November 25, 2022, 3:07pm

Some configurations can be placed in the tiddlywiki.info file, such as building multiple html at the same time, custom directories, and tiddlywiki commands(It’s just that this file does not currently support comments, and the readability is not very friendly)

saqimtiaz · November 25, 2022, 3:13pm

I use this approach myself, but it is very unfriendly for non-technical users.

Here is my idea of how to create a script that is user friendly for non-technical users:

In TiddlyWiki, have a tag that identifies tiddlers that contain static file rendering configurations, for example one configuration for HTML files, one for RSS files and one for JSON files. Each configuration tiddler would include:
- template to use
- tiddlers to render
Have a script that is executed on push via Github Actions workflow that:
- extracts the information from the configuration tiddlers
- for each configuration tiddler, it invokes the tiddlywiki command to create the static files
All static files are published to GitHub pages.

This way, a non-technical user can make all changes in their wiki, and push or save to Github and the static site will be automatically created. The user never has to configure anything in the script or actions.

oeyoews · November 25, 2022, 3:25pm

For non-technical users, They often use tiddlywiki (nodejs or html) with other apps, maybe they want a button to publish

saqimtiaz · November 25, 2022, 3:30pm

Absolutely. However, implementing this on the client side is considerable work and will require some rather large core improvements to do it properly. However, if we create a script or action as I described above, users can use it today. And in the future the core can provide a UI based mechanism that replaces the need for the script/action, but uses the same configuration tiddlers and templates. So users can easily transition from the script solution to a UI based one.

So the script is a potentially easy way to get non-technical users started with publishing static sites, and creating better templates for them, until the core has better browser based support for static publishing.

oeyoews · November 25, 2022, 3:34pm

Combined with the template, is there any specific example, is it similar to the theme of hugo?

saqimtiaz · November 25, 2022, 3:41pm

I think the default would be to use the same template as the static version of TiddlyWiki.com. See:

(Note that the above documentation uses the deprecated rendertiddlers command and instead we should user the render command.)

Users could specify a different template in the static site configuration and the script would use that instead.

boris · November 25, 2022, 3:56pm

I am closer to having this working. I’m using the defaults from “empty” edition.

@saqimtiaz I couldn’t find examples with render, do you have some?

Progress

Here’s where I’m at:

Couldn’t get it working at all

From single file wiki to output. Just fails silently? Verbose mode does nothing?

Solution: you must have a `tiddlywiki.info` file

You can grab the default one from empty edition.

Looking at that file was helpful. eg. I changed the name of static to index.html instead, and deleted the commands around alltiddlers.html.

Load from a single file wiki and build a static site

tiddlywiki --load twgroceries.html --build static

Open Questions

This outputs to /output/

how to change this output folder? Specifically, I don’t want a static.html AND a subfolder of /static/, I’d like them all in one folder
also I’d actually like to do the current folder I’m operating in, but --output ./ or similar don’t work

The “front page” gets rendered differently (without the custom Notebook theme), but all the individual tiddlers do get rendered with it. I guess I need to include the Notebook theme in the tiddlywiki.info? I don’t know what the path is for that theme.

Helpful links

I am working on this here https://github.com/bmann/twgroceries

I’ve also enabled git saving, which is really nice. I can just hit save on my desktop and it pushes it up (still have to pull down the code to get an updated local copy, but decent for a single working session).

I think I can use the LogSeq Publish plugin and the “publish to gh-pages” branch after I get things building correctly.

Right now, if I were to check in the output folder, I could set that as the subfolder to build from, and I’d be done. But that’s completely manual and what we’re trying to fix

saqimtiaz · November 25, 2022, 4:28pm

See tw5-feeds/plugins/feeds at main · saqimtiaz/tw5-feeds · GitHub

Place the output command before the build command, for example:
tiddlywiki --load twgroceries.html --output . --build static

Check the last two lines for the static build target and the arguments passed therein which refence the path static. Not sure if you can set the path to the current folder but you could set the index.html for the wiki to also be in the same folder.

boris · November 25, 2022, 6:13pm

Thanks @saqimtiaz!

I see your feed examples for render – and will want feeds! – I will attempt to replace these rendertiddler ones with render using your examples.

GROAN order dependent command line switches…

(and also silent failing with no errors…)

OK this worked – index.html in root, all other files in /static/

tiddlywiki --load twgroceries.html --output ./ --build static

And, editing the scripts:

"--rendertiddlers","[!is[system]]","$:/core/templates/static.tiddler.html","./","text/plain",

This works … but it deletes all the contents of the current root folder you run it from . Since I was operating in git and had just checked everything in, I just checked it out again so nothing lost. So, does need to go into some sub folder.

Whenever you run the main static tiddler flow, it deletes / recreates the folder, so it has to go first, and then you can generate an index file and it will end up in that same static folder:

"--rendertiddlers","[!is[system]]", "$:/core/templates/static.tiddler.html","static","text/plain",
"--rendertiddler", "$:/core/templates/static.template.html","static/index.html","text/plain",
"--rendertiddler", "$:/core/templates/static.template.css","static/static.css","text/plain"]

But! the index.html assumes a static subfolder, so all the relative paths there are wrong (appends an extra “static”). I guess that means spelunking in these core templates. I don’t want this to be example.com/static/sometiddler.html/, which is why I’m doing this – we should be able to have nice clean URLs like example.com/sometiddler.html.

So, feels like I’m making progress. Now just need to auto-run these commands via GH Actions.

saqimtiaz · November 25, 2022, 6:55pm

See PR with build script for static rendering: Added static build script by saqimtiaz · Pull Request #1 · bmann/twgroceries · GitHub

tiddlywiki.info file is not needed.
TWCrossLinks plugin removed due to issues with static rendering (this is where browser rendering would be superior)
renamed TWGroceries tiddler (and adjusted default tiddlers to suit) so that if you copy the static files to the same folder as the wiki, the tiddler file does not clobber the wiki file
edited static template to remove /static prefix. Future improvement, don’t overwrite the template and override the link generation by providing a variable on the command line.
uncomment last two lines in the build script if you want the static files in the same directory as the wiki, or you can copy the wiki to the same directory as the static files and publish that directory to GH Pages/Fission.

#!/bin/sh

#install tiddlywiki
npm install tiddlywiki

#clean up any remnants from last build
rm -fr wiki
rm -fr static

#unpack single file wiki into wiki folder
node node_modules/tiddlywiki/tiddlywiki.js --load twgroceries.html --savewikifolder ./wiki

#render each non-system tiddler as a static HTML file
node node_modules/tiddlywiki/tiddlywiki.js ./wiki --output ./ --render '[!is[system]]' '[encodeuricomponent[]addprefix[static/]addsuffix[.html]]'  'text/plain' '$:/core/templates/static.tiddler.html'

#render the css as an external file
node node_modules/tiddlywiki/tiddlywiki.js ./wiki --output ./ --render '$:/core/templates/static.template.css' 'static/static.css'  'text/plain'

#render the index page
node node_modules/tiddlywiki/tiddlywiki.js ./wiki --output ./ --render '$:/core/templates/static.template.html' 'static/index.html' 'text/plain'

#remove the wiki folder
rm -fr wiki

#uncomment below lines to copy static files to root directory of repo, warning if a tiddler has the same name as the wiki file it will be overwritten
#cp -a static/. .
#rm -fr static

boris · November 25, 2022, 7:24pm

Thanks. I need to translate this into a GH Actions. Appreciate the examples!

Next is pushing the actions folder out into a separate repo so it can be re-used by anyone.

I’ll get to that and post as I continue.