Hi guys!
Thanks @stobot for letting me know about this thread! I’ll definitely look into the issue you had with installation next week (this week is a bit busy).
First of all, thank you guys for trying out / discussing tw5-firebase! It gives me a warm feeling that others find it useful (or at least the concept interesting).
TW5-firebase is an ambitious project. Unfortunately I’ve had to put it on hold for the last couple of months, but I do plan to continue.
TL;DR: Jump to “Current Status”
Let me give you some insight into the whys and hows.
What’s the point?
A lot of us really like TiddlyWiki as a personal notebook. I’ve used Evernote, Joplin, etc. and they all have their strong points, but none of them offer the level of extensibility that TiddlyWiki does. In my opinion there’s several reasons for this:
- Everything is a tiddler - Macros, plugins, themes (what’s traditionally code) can be added to the wiki the same way articles and images (content).
- Anything can be (and is meant to be) customized. Any shadow tiddler can be overridden, no exceptions (icons, text - even the wikitext parsing logic).
- The entire UI is constructed out of TiddlyWiki macros. It’s turtles all the way down. The macro syntax can be a bit confusing initially, but it’s a lot more non-dev friendly than javascript. Combined with the previous points, this means one could in theory change anything they want about the way tiddlywiki works.
The result is a substrate that can be used to build all sorts of text-centric applications (TiddlyMap, GTD tiddlywikis, etc). As tiddlywiki provides a nice text editing experience, the application can be used to modify itself in a relatively non-programmer friendly way (eg: using tags to make a new menu item appear). Users must understand the specific tiddlywiki well, but not necessarily programming in general. In such an application adding a button or fixing a typo does not require a software developer’s participation.
The majority of text-based applications (including the original wiki functionality) become more valuable when collaboration is possible. TiddlyWiki has grown to become a fantastic tool as a personal wiki. Imagine the possibilities if we could make it a collaborative experience!
The road to collaboration
I believe a theoretical collaborative multi-user TiddlyWiki’s ideal users are small teams (less than a dozen individuals). Even in such a small, relatively high-trust scenario, certain guarantees are needed:
- Editing a tiddler should not ever result in accidentally overwriting another user’s simultaneous changes.
- It shouldn’t be possible for a user of the wiki to run un-sandboxed code in other users’ browsers (more on this in a bit).
- Administrators of the wiki should be able to add / remove users.
- There should be some form of access control (not everyone should be able to read / edit every tiddler).
- The complexity and cost of installing and operating the wiki should be modest.
It would be great if we could use an existing system for storing tiddlers. Unfortunately, I didn’t find anything that worked. The first point is difficult to achieve if the entire wiki is a single file. Solutions like the -otherwise excellent- TiddlyDrive https://tiddlydrive.github.io/ and TiddlyWiki in the Sky https://twcloud.github.io/tw5-dropbox/ thus don’t scale to the dozen-user teams I targeted. Storing each tiddler as a separate file in Google Drive or Dropbox would lead to very slow startup times and edit conflicts would be difficult to handle.
Collaborative web apps traditionally store their state in a transactional database to solve the problems around simultaneous user actions. Unfortunately installing and operating traditional web apps has been complex and time-consuming.
Firebase as a user-friendly backend
Installing and operating a database like postgres is not something a non-developer should have to do. Building on supabase as @boris mentioned is indeed an option, but at the time I started tw5-firebase, it wasn’t yet announced. Even today, it still lacks many of the components offered by firebase. Fission might be the perfect platform for a collaborative tiddlywiki server someday, but until then, we have Firebase.
Firebase attempts to provide solutions for all the traditional needs of a collaborative web apps while providing user-friendly dashboards. Namely:
- Authentication (firebase auth integrates with over a dozen identity providers)
- Static asset hosting (with integrated CDN)
- Database (firestore)
- User file hosting (different from static asset hosting, uses google storage)
- Server-side functions (run server-side javascript without managing servers)
All this is bundled up and nicely integrated. The various components know about each other. Firebase is well documented, with plenty of examples.
Of course, there are disadvantages:
- I’m just as disillusioned with Google as everyone else. The original “do no evil” mantra has seemingly disappeared - perhaps unavoidable at their scale, but still disappointing.
- Cloud functions requires users to opt into billing by providing their credit card. It gave me a pause too, I’m sure a lot of people would hesitate to proceed at that point.
- Firestore is not my favorite database. It’s impossible to run the real thing locally during development, performance is hard to analyze (no
EXPLAIN). The backup / restore tools are pretty lame. The way indexes work is hacky. There’s a surprisingly low ceiling to document size. It’s not horrible, but it’s not great either.
- Cloud Functions are the only supported way of executing custom server-side logic, which means that long-running processes with lots of in-memory state don’t work. This brings us to our next point:
Why not use the built-in node.js server?
TiddlyWiki5 comes with a built-in webserver. Projects like Bob https://github.com/OokTech/TW5-Bob build on it. It works really well locally. Unfortunately, as I mentioned at the end of the previous section, it’s architecture is not very firebase friendly. On startup, the TW5 node app reads all tiddlers. There is no further reading of tiddlers from the filesystem. Once loaded, the “source of truth” for tiddlers is the node process’ memory.
If we were to naively move this to Firebase (and swap the filesystem for firestore), serving every single request would require reading all tiddlers before it even got to processing the request itself. Also, firebase cloud functions run in parallel, which would lead to users accidentally overriding each others’ edits. So we need something else.
Note that this isn’t really a firebase-specific issue: collaborative web apps generally have a central ACID-compliant database storing application state, and -potentially several- stateless web server processes running the app’s code. The web server processes can (and in the case of Cloud Functions do) get replaced frequently; the contents of their memory should not need to survive multiple requests. AWS Lambda, supabase, etc would all require following the same architecture.
The built-in node.js server is built the way it is because it’s meant to be super-compact, treating node.js as just another execution environment. It’s brilliant but doesn’t scale. Since TiddlyWiki is designed to be a personal wiki, it doesn’t need to.
Current TW5-firebase architecture
The current architecture (what’s currently in master:
- A static HTML file containing a very minimal customized TiddlyWiki loads and authenticates the user using firebase auth.
- Once authenticated, tiddlers are read from firestore via HTTP request to a firebase function.
- Updating tiddlers also happens via HTTP requests to firebase functions (which then write to firestore).
- Importing non-text tiddlers uploads them to firestore storage. A tiddler with a
_canonical_uri field points to the real URL of the uploaded file.
This is already useful, but it doesn’t do all that’s required for collaboration:
- No user-friendly way to add users (although a CLI tool for this is included).
- No user-friendly way to set permissions. Access control is managed by placing tiddlers in
bags. Individual tiddlers don’t have rights. Moving tiddlers between bags should be easy for a user and atomic. They’re not.
- Currently, only admin users can add / update code in the wiki to protect users against other (malicious / playful) users. The wiki has access to firebase credentials, however, so an admin could impersonate the user and delete tiddlers in another wiki they have access to.
Next Architecture
- Firestore supports listening to changes in the DB. This means when one user makes an edit to a tiddler (and save the draft), any other user editing the same tiddler could be instantly modified. They would “merge” their versions of the tiddler before the second user saves, hopefully eliminating most “conflict on save” situations.
- In order to isolate user code within the wiki from firebase credentials, the wiki itself should run within a sandboxed iframe, while the “outer” frame communicates with firebase.
Current Status
In short, TW5-firebase is a (mostly) working prototype. It shouldn’t eat your data, but it is definitely not read for collaboration.
The master branch of tw5-firebase should work. I plan on fixing the issues @stobot faced with installation. Please note that the optimistic locking logic is not perfect: it does occasionally detect conflicts when it shouldn’t. This is not handled well by the current UI (the “save error” dialog appears). They can be resolved by deleting the draft tiddlers in question, but it’s not pretty.
The tiddler-version-manager at https://github.com/neumark/tw5-firebase/tree/tiddler-version-manager branch is an unfinished attempt at introducing the changes I mentioned in the “next architecture” section. I hope to continue someday, but haven’t been able to work on it for the last couple of months.
PS: I’m a bit upset that I’m only allowed to post 2 links!
