MultiWikiServer and SQLite

Developers
,
#21

I did not take your comment that way. I recognise that moving away from plain text tiddler files will be a big change for users, and I’m keen to take the chance to try to explain the rationale.

You would need an export command. Assuming the wiki is already running, the easiest way to get the data would be to use curl to request it from the API. As now, there will be an endpoint that can return an array of all the tiddlers in a bag or recipe as a big blob of JSON.

You’d probably want to commit .tid files. There’s an immediate problem of course in that not all tiddlers can be expressed as a .tid file so the design would have to fallback to using JSON files in those cases. Care might need to be taken to ensure that a tiddler moving between formats would be correctly handled.

The workflow mechanism could be used to automate things. You’d initiate the workflow by typing your commit message and clicking a button. The server would then execute the workflow, which would involve exporting tiddlers to a temporary file directory, and then invoking a script to call GitHub, finally deleting the temporary files.

1 Like
#22

Thanks. I will play with this and see if there’s a variant that works for me.

Well, I’m looking at an older documentation project that’s getting some new changes this week. My tiddlers folder contains 2 .css files, 1 .ico file, 2 .js files, 25 .json files, 39 .md files, 3 .svg files, 16 .png files, 1 file with no extension (hmm), the .meta files for each of those, and 307 .tid files.

They are all committed to a local git repository synced to our in-house GitLab instance. Will an export maintain a similar structure?

I know I could make most of those into .tid files, and would not be bothered if that happened automatically. But I don’t think that’s possible for the .ico and .png files.

#23

Hi @Scott_Sauyet such an export operation would probably reuse the existing core functions for controlling the naming and subdirectories of tiddler files. You would only have to worry about it if you were trying to track a tiddler as it mutated through different file formats.

#24

Perfect. Thanks.

That’s not a concern for me. Git for me is mostly a sensible backup facility. “I know that this was working Tuesday afternoon. Let’s see what changed since then.” Or, “Let’s revert to that version while I investigate.” Since I tend to have small commits (fewer than ten files, often just one or two), it’s really easy to see replacements even when git doesn’t recognize it as a rename.

BTW, TW has been amazing to use for this workflow. Because I’m always using a working system, it’s extremely easy to find good points to check in recent work, knowing that they won’t likely break anything. Even when modifying my own JS plugins, a restart is so fast it never interrupts the flow. As always, thank you for this wonderful tool!

#25

This opens many doors. SQLite is almost certainly the best answer; it remains as portable an option as I can think of in many respects. SQLite’s ability to be self-contained and useful within the browser while also easily and performantly wielded on bare metal fits the Tiddlywiki ethos, too. Both tools are somehow both web and desktop first; they compatibilize many layers.

If this were about software outside the browser, distributed databases might also be worth considering here (it would probably be more work up front). Decentralized multidevice and multipersonal synchronization remains a crucial yet hard problem. Picking a database designed to solve at least some hard parts of some of these problems may be worthy. That also could just be an offshoot evolution that SQLite makes somewhat more accessible, might plug right in. The ability to broadly intermingle and scale multiple wiki infrastructures together in a graceful way may hinge upon something like this approach (though, I’ll argue the portability of SQLite is handling other problems here better than distributed databases could dream). Given that this is so deeply invested in the browser and working within potatoes (something I admire*), SQLite is surely the best move forward.

I’m still concerned about ensuring that someone with a mere text editor would be able to make quick and valuable use of a Tiddlywiki through either the html file or the highly legible .tids without needing many specialized tools (which isn’t to say that sqlite doesn’t enable its own, highly desireable accessibility features for developers and powerusers either). Working on natural language structured plaintext files should never become a second-class citizen for TW. That’s doable, as pointed out. That democratizing function is a moral virtue of Tiddlywiki. Though there are trade-offs to consider, we can all prize highly accessible on-ramps and the open simplicity of migrating into and out of TW.

#26

I’ve been having visions of a new interface for TiddlyWiki Either 3D or 2.5D. Think a crossbreed between Cecily, Tiddly Map and the interface of the supercomputer from the “Minority Report” film. With some flavours.

I was wondering if a new interface can be layered on top of the main TiddlyWiki core. API rings true if I want to achieve anything close to that. But the choice of technology which would support the viewer is so critical, that I’m stuck on it.

To be honest, I feel less and less prepared to realise that, given how little I know about the direction TiddlyWiki is going. I thought, if I begin by getting information at the source, it may eventually untangle.

#27

One key piece of information about WASM is that WASI still has more features coming. It hasn’t reached version 1.0 yet. WASI brings in offline, system-level capabilities for any combination of desktop functionalities that mix with the web ones. Just thought I would mention it even though I have no idea what the plans are.

They’ve done WASI version 0.2.0, which they call “WASI Preview 2”. I think this has been this way since July 2023. They still have WASI Preview 3. Looks like it could be another 18 months or more until 1.0 comes out.

From the way it sounds, these are less like Alpha, Beta, RC releases, and more like feature-set releases, although the Preview 2 is still receiving many updates.

WASM is built in to browsers, whereas WASI is installed on the OS.

  • I mention all this, because depending in part, on what you’re doing, WASI (not just WASM) is really what is supposed to give developers wings. (Their snail pace in development has held WASM back.)
3 Likes
#28

I salute the adoption of Sqlite (wasm or not).
It really seems to become the lowest common denominator between many dockerized selfhosted solutions (I guess because it’s the best combi of fast/easy to setup).
It also became the easiest solution for horizontally scaling a db with marmot:

Just a sqlite file and a marmot oneliner on a server (no kubernetes, no containers)…let that sink in.

1 Like
#29

Sounds great, how might version control be possible if everything is stored in an sqlite db?

Nick

#30

We use Fossil in combination with TiddlyWiki to store versions of our tiddlers. For example, for our TTRPG Wiki

image
image1897×890 206 KB

we have this Fossil repository

image
image1228×891 98.1 KB

where all our tiddlers and other resources are stored in STON format a JSON like inspired format but with several additions that make it more suitable for wider use cases.

In that way we can collaborate on our interpersonal wikis without fearing destructive changes when synchronizing them.

By the way, I like the idea o having SQLite behind TW, as I’m a huge fan of the philosophy behind it and its related communities (SQLite, Fossil). Now that I’m reluctantly entering the JS server side of TW, but with Bun instead of NodeJS (which for my newbies eyes has a lot of historical complexity behind), do you, @jeremyruston, have the opportunity of comparing the SQLite integration in Bun vs Node?