The philosophy and design of the single HTML file wiki

#1

Context: I was fiddling with the idea of using TiddlyWiki in restricted environments that lack Node.js, so only the single HTML file option is available, watch the file for changes and automatically commit each change to a version control repository, to get a detailed history of atomic small changes. While an autocommit workflow would lack valuable info like human friendly summaries of changes, having a timestamped history of changes still keeps the option of spotting breaking changes via bisection.

Since the Node.js option is out, it’s not possible to manage (version control wise) tiddlers in separate files, one has to deal with a single big file.

Which brings me to the topic line. In TiddlyWiki’s philosophy, what is the status of the single HTML file that wikis get saved as? Is it merely a container for the data stored in user made tiddlers? Is it rather a first class source code (HTML + JS + JSON) file that is diff friendly, as in two versions of it produce diffs of minimized size?

Of course my hopes were high for the latter, but after doing a very simple test, I’m not so sure, hence I’m asking here.

So I took an empty wiki (a ~2.5MB file), added a couple of tiddlers with one word in text field, one additional field (so nothing fancy or big), saved the wiki and diffed the versions. Diff size is 200KB. The offenders seem to be the very long context lines containing theme tiddlers. Did anybody think about ways to deal with this? I know the problem lies deeper - JSON isn’t friendly to multiline strings, but still. Or am I just misunderstanding something and making a big fuss over a non issue here?

#2

I did a test and it seems the big “vanilla theme” changes are only there the very first time.
Afterwards the changes are small. eg:

-> git diff -U0

diff --git a/empty.html b/empty.html
index f9a81d6..3cffe32 100644
--- a/empty.html
+++ b/empty.html
@@ -162,0 +163,2 @@ Error message and password prompt
+<li>aaa</li>
+
@@ -184 +186 @@ Error message and password prompt
-{"title":"$:/StoryList","text":"","list":"asdf test GettingStarted"},
+{"title":"$:/StoryList","text":"","list":"aaa asdf test GettingStarted"},
@@ -186,0 +189 @@ Error message and password prompt
+{"created":"20241121122011140","text":"aaa","tags":"","title":"aaa","modified":"20241121122017332"},

So if you save a wiki and then run git diff -U0 which is the shortcut for git diff --unified=0, you’ll get a change log, which only contains the changes.

If you save this output into a “diff.txt” file git diff -U0 --output=diff.txt and commit it with the single file wiki, you will have a good overview what actually changed in the diff.txt file.

just a thought
-mario

#3

This is still interesting to investigate, because that should not really happen.

#4

I have some details on a similar approach that may help you:

I’ve been keeping a single-file HTML wiki in git with a script (and Siri Shortcuts) to auto-sync it for the past ~year. My wiki has plugins (including big ones), but assets are external. I had them as internal for a few months until I moved to the External Attachments plugin and migrated them to an assets/ folder.

My HTML file is 10MB. The folder my wiki and assets are in is ~24MB (not counting the contents of .git. The total size of the folder is ~130MB. I did run a garbage collection pass this past weekend (git gc --aggressive --prune=now - your milage may vary) that helped get the size down (it was closer to ~1GB before garbage collection).

Hope this helps - please share your approach if you end up going down this path.