Wonky Dev Question: Can I create a wiki from within a JS app using tiddlers stored in memory?

Scott_Sauyet · February 23, 2025, 2:21am

I’m trying to dynamically generate a number of single-file wikis from within a JS application. They will each have a little over 32K (very small) tiddlers. I’m able to quickly generate the tiddlers. But trying to use them to build a wiki is more difficult than I would hope. Is there a way to build a wiki from within Node without writing the tiddlers to files?

For now, I create several intermediate formats, and it’s easy enough to write these to disk.

I found it frustrating that a JSON file containing an array of JSON-formatted tiddlers doesn’t seem to work with the --load command. That format works fine for dragging onto an empty wiki. But I don’t want any manual intervention here. I would like this to all run from a Node script, perhaps eventually on a CI/CD server.

I did find a work-around to that. Just wrap those tiddlers like this

<script class="tiddlywiki-tiddler-store" type="application/json">
  [ ... tiddlers here ... ]
</script>

and put them in an .html file. Then I can call something like this¹:

const $tw = require('../node_modules/tiddlywiki/boot/boot.js').TiddlyWiki();

// ...

$tw.boot.argv = [
  wikiFolder,
  `--init`, `server`, 
  `--load`, tiddlerFile,
  `--render`, '$:/plugins/tiddlywiki/tiddlyweb/save/offline', wikiFile, 'text/plain'
]

// ...

$tw.boot.boot()

But that `load` call takes forever! I didn't time it, but I just ran it and it must have taken at least 40 minutes, perhaps as much as an hour. The `render` isn't quick, but I'm pretty sure it's under five minutes, which I can live with for such large wikis.

Another work-around occurs to me: I can manually write all these tiddlers to this temporary tiddlers folder. I’m sure I can write 32K small files in a small handful of minutes. I will probably try that next.

But I would much, much rather not use the filesystem for this at all. I have these tiddlers as objects in memory. Is there a straightforward way to use them?

¹ That leads to another question: I feel as though I’m digging too far into the internals with this method of setting argv and calling $tw.boot.boot(). Even requiring TW through node_modules/.../boot.js seems wrong-headed, even if it works. Is there a more ergonomic technique for calling TW tools from a Node app? (This is less critical to me, as my current technique works. But it feels far too hacky.)

TW_Tones · February 23, 2025, 2:26am

Did you look at leveraging the inner wiki plugin?

Give named tiddlers you can generate an inner wiki based on empty plus the nominated tiddlers.

Similarly what you may be able to use is a selective save or download similarly how one can save empty.html on tiddlywiki.com however it now includes your custom tiddlers and not the rest of the wiki.

Otherwise you may want to use the zip function to zip the specified tiddlers and or core.

Of course these suggestion are not within JS but theoretically can be triggered from within JS.

just some possible leads.

Scott_Sauyet · February 23, 2025, 3:19am

Thank you for your feedback. No, I didn’t. Please correct me if I’m wrong, but I don’t think that would help. I’ll discuss my scenario more completely below, but first:

The problem is not one of sheer size. There are a lot of tiddlers, but the generated wikis will be well below 20MB in size. They should load in a few seconds.

While running in JS is not essential, and I could write this in Python, Scheme, Haskell, Java, or a number of different languages I use, JS is my main language, and it certainly seems the best fit for working with TW, which itself is written in JS/Node.

Thank you very much! I always appreciate your feedback.

Background

This is a follow-up to the King James Bible wiki work I discussed some months back. I always planned on expanding this idea to cover other versions of the bible beyond King James. This is part of that effort.

Bible Super Search has nearly a hundred translations of the Bible, in dozens of languages. They are available in JSON, in a format which, aside from a small bit of metadata, consist of Verse objects that look like this:

    ...
    {
      "book_name": "1 Corinthians",
      "book": 46,
      "chapter": 13,
      "verse": 11,
      "text": "When I was a child, I spake as a child, I understood as a child, I thought as a child: but when I became a man, I put away childish things."
    },
    {
      "book_name": "1 Corinthians",
      "book": 46,
      "chapter": 13,
      "verse": 12,
      "text": "For now we see through a glass, darkly; but then face to face: now I know in part; but then shall I know even as also I am known."
    },
    {
      "book_name": "1 Corinthians",
      "book": 46,
      "chapter": 13,
      "verse": 13,
      "text": "And now abideth faith, hope, charity, these three; but the greatest of these [is] charity."
    },
    ...
]

The code I’ve been writing uses these tiddlers and a small amount of additional metadata to generate Book, Chapter, and Verse tiddlers plus a very small number of infrastructure tiddlers. The goal is to use these to create a single-file wiki similar to my King James one but for different translations. (Right now I only handle English translations, but it should not be too hard to add other languages.)

So far my process has done everything in JS. My usage of TW is slightly odd JS, but it’s workable. I am just hoping there’s an easier way for the final step of creating a wiki than writing all these tiddlers to disk.

pmario · February 23, 2025, 3:26am

The tiddler store that you mention here is designed to be concatenated with an empty wiki. eg:

<script class="tiddlywiki-tiddler-store" type="application/json">
  [ ... tiddlers here ... ]
</script>

You can create tiddler stores as above and then concatenate it with an empty.html like this.

Windows PowerShell

Get-Content test.store, empty.html | Set-Content joined.html

Unix

cat test.store empty.html > joined.html

You can add several stores at the front of an empty.html file. When the wiki is opened and than stored all concatenated stores are combined and added into the standard wiki store as we know it.

Hope that helps.
-mario

Scott_Sauyet · February 23, 2025, 3:35am

Well that sounds a lot easier than my current attempt!

I’ll try that right now. Thank you very much!

Scott_Sauyet · February 23, 2025, 6:24am

That worked perfectly. Thank you very much!

You can see what I’m working on at https://github.com/CrossEye/TW5-Bibles. I haven’t done a README yet, so here’s the 30-second overview:

A version consists of a bible formatted as JSON verses downloaded from Bible Super Search, and a config function which returns those verse objects and various metadata.
The script scripts/makeWiki.js converts into a nested format with books containing Chapters and chapters containing Verses, adding a fair bit of metadata as necessary, converts that into a set of tiddlers, adds a few additional tiddlers, and generates a wiki using all those tiddlers.
```
node scripts/makeWiki Coverdale
```
will look in the versions/Coverdale folder for that config module, run it, and create these files using the data it contains:
- raw.json, the original file;
- enhanced.json, that file with additional information about books and chapters;
- tiddlers.json, a list of tiddlers suitable for dragging into another wiki;
- index.html, a complete wiki containing that version of the bible.
Links to these are (currently hardcoded) on GitHub Pages.

There are still some bugs in the output I need to trace down (mostly, ironically, in the King James version.)

After a short round of cleanup, next up will be an attempt to internationalize this, allowing for non-English translations. And then… who knows?

Thank you again @pmario. This is so much simpler!

pmario · February 23, 2025, 1:24pm

You are welcome.

There is only one thing left, just to be sure. The SCRIPT stores are handled one by one. If there are tiddlers with the same title, the last store that is handled wins.