Rendering and processing tiddlers

Developers
, ,
#1

I have an experiment running that may or may not end up being a project. I am trying to build a wiki with my own data by working with individual files in either .tid or .tiddler format.

I’m processing text files, so I’m intrigued by the slicer wiki type that the tiddlywiki --init NodeJS command can generate, but I don’t understand how the utility works, especially since it appears to be simply a wiki with its own tiddlers. But that’s neither here nor there; I can find another way to slice up text files.

I have one TiddlyWiki served on my localhost using Tiddly Server (tiddlyserver --config mypath/mysettings.json). Then, I have a totally separate project in a separate directory and even a separate NodeJS ecosystem. (I’m using local NodeJS packaging; so I’m prefixing the commands with npx in my terminal environment.) In my second project directory I have the tiddlywiki NodeJS program installed and do not have tiddlyserver installed.

From the second directory, I’ve been trying to import the tiddlers from the wiki running on localhost through the tiddlyserver instance from the first directory. I’ve read the help documentation in the tiddlyserver command and have tried a pretty basic use of the –render option:

npx tiddlywiki --verbose --fetch files “http://127.0.0.1:8080/wiki.html” “[all[tiddlers]]” application/x-tiddler ./tiddlers/

The command works! The output says it imported 152 tiddlers. But it doesn’t do what I was expecting. I thought this would populate the ./tiddlers/ directory inside of the current folder (which was generated by tiddlywiki --init server) with *.tid files for all the tiddlers from the wiki file I have hosted in localhost through tiddlyserver.

Instead, it only produced one file, with this content:

created: 20240611153228728
modified: 20240620151246540
title: application/x-tiddler

Dropdown Template

I was trying a basic command from a careful reading of the help docs, but I don’t understand how to operate this functionality. I’d actually like to be able to select specific plugins and tiddlers to import, but there’s no way I’m going to understand how to do so. Getting the files to actually appear will be the first step.

The intermediate goal is to be able to develop a wiki by working with a set of plain text files, so that I can operate TiddlyWiki almost like a static site generator. I was confused about this difference between the tiddlywiki and tiddlyserver NodeJS programs for this purpose, and I basically started working through tutorials for both of them at the same time:

The end goal is to process text files with metadata and import into a wiki, using a sparce pre-made wiki with 5 plugins as a model and/or starting point. I notice that even though tiddlyserver does not seem to allow me to work with individual tiddler files, it is an efficient way to work on a wiki, since it saves efficiently and doesn’t get frozen or slow like my browsers and TiddlyDesktop often do. So, I might skip to the end goal and just do a bunch of external text processing and then import into the wiki that I have hanging out on localhost.

But I’d like to be able to understand the Node utilities, particularly what production methods workflows the tiddlywiki program allows.

#2

Have a closer look at the –fetch docs … It says the 3rd parameter is <transform-filter>, which says:

For all variants, the <transform-filter> parameter specifies an optional filter that transforms the titles of the imported tiddlers. For example, [addprefix[$:/myimports/]] would add the prefix $:/myimports/ to each title.

So you basically did rename all your tiddlers to application/x-tiddler … So it’s only 1 tiddler.

#3

Okay, so this at least fetches all the tiddlers and properly renders a *.tid file for each one:

npx tiddlywiki --verbose --fetch files “http://127.0.0.1:8080/wiki.html” “[all[tiddlers]]” “[all[$title]]” application/x-tiddler ./tiddlers/

With the naming filter being [all[tiddlers]].

#4

I do this fairly often, often as a one-off, but sometimes as repeatable processes.

I tend to write this code in Node, generate an array of (flat) JS objects, and write the resulting array as JSON. Then I drag the JSON file atop my mostly-empty wiki. As long as I’m not changing titles, I can do this repeatedly as I work out the correct structure for my tiddlers, with subsequent imports just overlaying earlier ones. When I want a repeatable process to use in a build step, I do the same manipulations into JS objects, then write each to a .tid file.

You can see a typical JSON process in http://link.sauyet.com/107, which includes information cut-and-pasted from Wikipedia, with the headers modified manually, and then some code to create a JSON representation of this data. You can see something more sophisticated in the creation of .tid files in my Periodic Table wiki: buildContent.js, which collects the information from the two files in RawData and uses it to create most of the information in the Plugins folder.

So you can work with individual files; here I create them from external sources. But you can also edit them. The only thing you will need to do – because of local caching – is to restart the server after you make such changes; that’s usually a quick process though.

1 Like
#5

You can see a typical JSON process in http://link.sauyet.com/107

Thank you for those examples. It took me a while to discern what you’re doing; but yes, your Ramda code pen thing shows something rather similar to what I would like to accomplish. So, your solutions are valuable to me. I’m using a different workflow. I take it you’re executing this on your command line using Node?

Just finished a wiki with some data that I ended up more or less manually entering after doing a simple conversion from markdown to wikitext with a text editor macro (which I had to make). I’m tempted to try to make One TiddlyWiki Python Script To Rule Them All, chopping up my data files based on configurable sequences and outputting *.tid files or JSON.

I have a question—anyone, Can you transclude tiddlers from markup in *.tid files?

#6

My first example was just hacked together to demonstrate the process, but yes, it would run under the command line in Node, if it were for some repeatable process. The other example already does run that way, whenever I want to alter my data tiddler format.

I’m not quite sure what you’re asking. If you want to your generated tiddlers to include bits that transclude others, then sure: Just make sure your template includes the transclusion syntax. If you want to transclude the generated content in other tiddlers, again, just do do; once they’re generated and imported, it will just show up in the right place. Or you can do both. I’ve used TW for documentation wikis generated entirely – outside a few templates and such – from external data sources; there’s a large deal of transclusion included, mostly in the templates, but in a few cases in the tiddlers themselves.

If however, you want to use TiddlyWiki transclusion to generate content that is then included directly in other generated tiddlers, while I’m sure it’s possible, I’ve never tried it, and I would discourage it. Using data tiddlers instead is akin to database normalization: a very good practice for most cases.