Announcing the MultiWikiServer Plugin

#1

Introduction

I have started a major new area of core development. It is about enhancing the capabilities of TiddlyWiki under Node.js to their full potential:

  • Hosting multiple wikis at once, with the “bags and recipes” mechanism for sharing data between them
  • Improved handling of file uploads and attachments, allowing gigabyte video files to be uploaded and streamed
  • Instantaneous synchronisation of changes between the server and all connected clients
  • Workflow processing on the server, for example to automatically compress images, or to archive webpages
  • Robust authentication and authorisation options

These are deep changes that require significant reengineering of existing code. To avoid breaking backwards compatibility, everything is packaged as an optional new core plugin called “multiwikiserver”. Users that do not install the plugin will not notice any changes from how TiddlyWiki works today.

I’ve started work on the hosting of multiple wikis, and am currently working on the handling of file uploads and attachments. The code can be seen over at GitHub, which is also the best place for technical discussion about the implementation.

In this thread I wanted to focus on the implications of these enhancements for users.

I am really excited about the possibilities. I personally use TiddlyWiki pretty intensively in the form of single file wikis for each of notes, recipes and guitar sheet music. My guitar wiki is around 100MB because it includes lots of images; it performs OK on my hardware but feels like it is getting closer to the limit of acceptable performance. There are several other areas of my life that I would love to get into TiddlyWiki where the obstacle is the need to handle large quantities of media in the form of images and videos. I would like to be able to drop whatever is relevant into my wikis, without having to stop and think about the performance implications.

Usage

MultiWikiServer installation and usage will be almost identical to today. The initial screen resembles TiddlyDesktop.

There is a list of wikis at the top, shown with their “favicon”, name and description. Clicking on a wiki opens it in a new browser tab. As you work on it, changes are synced as individual tiddlers to the server; you do not need to remember to save changes, and you are not slowed down while saving takes place.

image
image1260×2346 347 KB

The screenshot has many mysterious references to bags, which we will explore in the next section.

Bags and Recipes Model

The idea of bags and recipes is very simple. Tiddlers are stored in named “bags” which act like folders. Wikis are constructed from a list of bags called a “recipe”, putting together the constituent tiddlers to make a single wiki.

So, in my own case, not only would I have bags called “notes”, “guitar” and “recipes” that make up wikis of the same name, I might also have a bag called “customisations” that is added to each wiki so that I can have uniform customisations across all of my wikis that I can update in a single place.

The semantics of bags and recipes can be expressed more formally in a small number of propositions:

  • Bags are containers that can contain zero or more tiddlers
  • Bags have names
  • Bags cannot contain more than one tiddler with the same title
  • Recipes are lists of bags in a specified order
  • Recipes have names
  • Recipes cannot contain the same bag more than once
  • The tiddlers within a recipe are those that result from overlaying the tiddlers from each bag in the recipe in turn. This means that where there is more than one tiddler with the same title, the one in the bag that is highest in the recipe is the one that “wins”
  • Wikis are constructed from the tiddlers in a particular recipe
  • Tiddlers that are modified from the wiki are always saved to the topmost bag in the recipe

The key concept is the way that bags overlay one another in a recipe. It is very similar to the way that ordinary tiddlers overlay shadow tiddlers, hiding their value. Just like shadow tiddlers, if the tiddler in a higher bag is deleted it will reveal any tiddlers with the same title in lower bags.

In the screenshot above, you can see that each wiki has an associated list of bags. The highest bag is highlighted to show that it it is the one that will receive new tiddlers.

This work is the culmination of many years of thinking and planning. The idea of bags and recipes dates back to TiddlyWeb, more than 15 years ago. It has been tried and tested through the TiddlySpace years. It is intensively used in Anna Freud’s Manuals, a collection of nearly 1,000 intertwingled wikis with over 1,000 users and a total of over 1GB of data. You can see it at https://manuals.annafreud.org/. The original version of TiddlyWiki 5 in 2012/2013 included a partial implementation of bags and recipes that is still present today, but until now it has not been fleshed out enough to make it useful.

Final Thoughts

There is quite a lot to take in here. I intend to make more posts about other aspects of the work, but in the meantime questions and comments are very welcome.

20 Likes
MultiWikiServer and SQLite
Creating a chat/messaging environment
¿How does TW compare to SAP solutions?
What's a bag? And what's a TiddlyWebAdaptor?
My ExternalContent, once loaded, keeps getting saved
Tiddlywiki (Node.js) on iphone?
#2

Today I use subdomains and initiate multiple instances of tiddlywiki. Will I initiate once instance tomorrow that utilizes the folder structure rather than subdomains?

#3

2 posts were merged into an existing topic: MultiWikiServer and SQLite

#4

Hi @john.edw_gmail.com yes, you will initiate a single instance of TiddlyWiki that will serve multiple wikis. However, tiddlers are stored in the database rather than directly in the file system. You’ll be able to import your existing wikis, and you will be able to sync changes back to the file system if you want to stick with one-file-per-tiddler.

#6

Just to say well done Jeremy! I was the earliest user from Anna Freud Centre, where I used TW to develop a method of working with complex young people at high risk (“AMBIT”) that is now used in adapted ways by teams around the world. The bags and recipes multiwiki arrangement has since those early days been an amazing way to allow multiple teams to own their own locally-customisable version of a centrally developed set of principles and evidence-based practices. Can’t think quite how else we could have developed things in the way we did - so a huge vote of thanks to TiddlyWiki and Jeremy’s hard work

5 Likes
#7

Exceptional introduction, @jeremyruston

If time permits, it would be informative (and fun!) to read a report about your adventures while transferring and configuring your guitar wiki to work with the new system and setup.

Killer reason for adoption, right there. :clap:

Aside, and ICYMI: Serverless -- you know, like it was supposed to be

2 Likes
#8

The guitar wiki doesn’t use external images because that makes it easier to airdrop it to others. That means that it’s just a matter of drag and drop to import it. It just takes a few seconds to sync the data back up to the server, and results in a .sqlite file just slightly larger than the original HTML file.

#9

Will tiddlers load on demand similar to lazyload when not part of the core/setup or have the lazyload capability in the new world? Today I initially send ~12mb on initial load rather than the full 250+mb of tiddlers.

#10

Hello

This sounds like a great extension of the Philosophy of Tiddlers into a wider ecosystem.

With regard to your final proposition about bag and recipe semantics:

… to continue your analogy, I wonder whether “drawstrings” might also be attached to the bags (for example, a tag, field, or combination of these) to draw saved tiddlers back to the bag where they belong (for instance, so a guitar-related tiddler produced from a recipe gets saved back to the guitar wiki, while a note gets saved back to the note wiki). If so, perhaps “drawstrings” could be specified on the initial MultiWikiServer screen, with the topmost bag being used to save anything not marked as belonging in one bag or the other.

I’m not sure whether this is a helpful suggestion or not; either way I’m looking forward to experimenting with this set-up; at the moment, I have one single-file TW for all my notes, and it might be interesting to split these up and recombine them in different ways.

All best wishes, and thanks very much indeed for making an app which is both super helpful and super enjoyable to use!

1 Like
#11

Very exciting!

Does this mean that I would need to be running node.js in order to take advantage of these changes? Or might it be possible (at least in theory, eventually) for a server-based solution like TiddlyHost to offer complex bags-and-recipes MultiWikiServer-style accounts for users who access TiddlyWiki on web-based connections?

I see that this is an elegant default for interface-specific changes that should stay “shallow” to any given wiki, or for laying the groundwork for multi-user permission structures where write-access to “deeper” bags can be restricted…

Like @srbt1 I expect it would be very helpful (at least in a single-author world) to be able to save “back to the source” under certain conditions. For example, when I realize I need to fix an error in a definition-tiddler, that’s the kind of edit that I’d always want to lock in back at the “bag of definitions” shared across many projects, without having to shift to another wiki interface to make that edit. Of course, a frequent maintenance-routine of scanning my top-level bags for tiddlers that fit various conditions (such as being tagged “definition”) could ensure things end up where they belong. Still, I could imagine various workflow hiccups that would result from making it impossible, from within any given wiki interface, to save directly to bags other than the topmost.

I’m aware how long this has been in development, and am chuffed to see this announcement!

1 Like
#12

Hi @john.edw_gmail.com the plan is to maintain the same support for lazily loaded tiddlers as we have today, but with some modest improvements (for example, I would like to avoid having to expose the _is_skinny field that we currently use to mark a skinny tiddler that has not yet been loaded, and instead offer an is[skinny] filter).

I did experiment with early versions of Xememex working in that way, but we found that it was confusing for users because it means that you cannot predict the bag to which a tiddler will be saved unless you know the bag that it came from. In contrast, with the present arrangement tiddlers are always consistently saved to the same bag within a recipe. That means that you can look at the address bar of the browser and know where your changes will be saved, which only requires familiarity with the recipe, and doesn’t require inspection of a specific tiddler to see where it is saved to.

With a shared system, inadvertently saving something into the wrong bag might have implications for the privacy of data (for example, imagine a teacher sending a note intended for a specific student but having it visible to the entire class). So we want to be conservative, and minimise the chance of surprise.

Having said all of that, I would like to expose the logic that chooses the bag to use to save a tiddler so that it is hackable, and users can experiment with other configurations.

To take advantage of these changes does indeed mean that either you would have to be running node.js, or you could use a hosted version that was running on a server operated by someone else.

I think it is still highly beneficial to have the option of both ways of working. Nothing here will make TiddlyHost any less useful.

It is very tricky to get these semantics right. It is easy to make the rules complex, but the cost is that the user has to run the algorithm in their head every time they save a tiddler.

My hope is that we can make the rules for saving to a recipe cascade be brutally simple, and compensate for any loss of flexibility by providing good tooling for moving tiddlers between bags.

The workflow processing that I referred to above might also be relevant. In essence, this is the ability to perform server-side actions when a condition is met. For example, one might have a very simple workflow that looked for tiddlers in one bag that met a certain criteria and automatically move them to another bag.

3 Likes
#13

@jeremyruston puts up a new profile pic sporting a distinguished beard and this TW thing is what everyone is excited about?!! :wink:

(I am also excited about this TW thing)

4 Likes
#14

@jeremyruston - cool stuff! I really liked TiddlySpace so this is very exciting.

Q: Does this at all deal with “multi users” i.e to allow for several users to edit a shared wiki(s)? Or is that just a totally different matter?

Thank you!

#15

I’m so excited to hear this! I’ve been thinking about figuring how I could achieve just a few of these things myself, and been overwhelmed. These two:

make me think we could have that seamless multi-user system that I’ve been dreaming about.

Would the core get a similar treatment? I would love to be able to include all wiki editing capabilities in a separate bag from the viewing ones.

This will be essential, I’m quite sure.

#16

Thank you! I hope it lives up to expectations.

Multiple users editing the same wiki has pretty good support with the existing TiddlyWiki on Node.js setup, but if there are shortcomings I would certainly hope to address them.

Hopefully so. It will be satisfying to fill in the gaps and get things working properly.

The core doesn’t split neatly along those lines. For example, if one considers the set of core widgets and tries to separate the ones that are concerned with editing one might choose the <$edit-*> widget for a starting point, but of course it is also used to present the search user interface. There are certainly templates that are only concerned with editing, but they don’t make up much of the bulk of the core.

#17

I have been following your progress on this on Github. It has made me completely re-think how I want to structure the live-multiplayer server. Mahalo (thanks) for starting this thread so we can talk about intent/data-structure/etc I also see the need for a “Publish” UI on a tiddler that could move a copy of it to another bag (and optionally delete the copy in the topmost bag to expose the new tiddler in the lower-bag). That probably needs hashed out more thoroughly, etc. Excited to see where this goes.

1 Like
#18

Great, thank you @joshuafontany I am glad it is hitting the spot.

I think that sounds like the workflow mechanism I am envisioning. I’ve done a few attempts over the years at state machines modelled as interlinked action tiddlers, and now I’m hoping to tie that work together into a robust mechanism we can use on the server.

A workflow step would have a condition (expressed as a filter) that, when satisfied, would trigger the execution of the associated actions. The available actions would be pretty much what we have today, but we would need to add support for triggers such as:

  • Elapsed time interval (like cron)
  • Successful completion of another workflow step

The trickiest aspect seems to be exposing enough of the inner workings interactively to enable users to understand what is going on, and be able to debug when things don’t work as expected. It’s also remarkably easy to set up infinite loops, or thrashing between two opposing actions.

4 Likes
#19

Can we still keep .tid file in some way while use this plugin? Can filesystem-syncadaptor still be used to sync changes from SQLite to fs, so SQLite is just a cache?

Since TidGi is a Tiddlywiki based Obsidian replacement, so I want to keep it .tid/.md files based, like Obsidian.

#20

It will be possible to sync from the tiddler store to an external file directory. But you couldn’t treat the .tid files as the master copy because they would lack the metadata needed for syncing discussed above.

I am sure we will be able to give users lots of options for getting data out of MWS in whatever format they want.

#22

The biggest issue of how the current sync ‘server’ edition is linked into the boot process via boot.files which requires a reboot to load tiddlers updated from external sources.

@jeremyruston Am assuming that if an external source updates tiddlers in the SQLite database they will take effect immediately?