[TW5-Yjs] Real Time Sync Brainstorm

[joshuafontany]

Hi everyone,

I am very close to releasing a public beta of my “live multiplayer” plugin: TW5-Yjs

I started out by writing an update to the Bob server sync mechanism, but it quickly turned out that re-writing the underlaying logic was turning the tool into its own “thing”.

As such, I wanted to touch base with the community on the key features I am focusing on, and ask if there are any ideas/suggestions/etc for tools along these lines.

::Aside:: After watching Saq’s excellent presentation on the File-Uploads plugin, and studying the Yjs library further, I think I will end up splitting this up into a Yjs “core” and a set of “provider” (the Yjs term) plugins that hook everything together. This current prerelease under discussion would be TW5-Yjs and TW5-YjsWebsockets.

Key Features

• Multi User Server
  • MultiServer extends the native TW5 Server
  • adds an “admin” Authorization type that controls who has administrative access to the server functions
  • and can be configured to Simple or Header based Authentication
  • and can support randomized Anonymous Usernames for Anon users
  • and has a secure “Read-Only” mode with changes to Yjs docs from “readers” ignored by the server.
• Multi Wiki Server
  • with each wiki considered a “room” with a unique server path
  • with a main “RootWiki” at the root server path that allows admin users to:
    • create and delete Wikis
    • modify any served wiki’s “tiddlywiki.info” file from the RootWiki
• LIVE Multiplayer experience
  • All updates to a wiki are sent to all connected users live via a WebSocket Client-Server model
  • “Updates” trade Yjs info about the snapshot/state of the wiki document and the “Confict Free Replicated Datatype” logic ensures that “all users sync to the same state… eventually”
  • All fields handled as “text strings”. For now, field values in the Yjs-doc “copy” of the tiddler are of “string type”, and replaced by the incoming value. Eventually, the full “Yjs Text type” will be used, which will allow sharing of individual insert/delete actions, tracking Undos, showing active User’s cursors with different colors, etc. Eventually, other types might be added, with field-name prefixes allowing a “grow-only/field-name” as a grow-only integer, etc. This is advanced Yjs/CRDT stuff.

Some things to consider

• FileWatcher system?
  • Bob has a functional system to watch the filesystem for changes, and update a wiki-in memory.
• Yjs files as “source of truth”
  • Can we simply trust a “server address + $:/SiteTitle” to identify a unique wiki?
    • Bob uses an internal $tw.WikiName variable which becomes part of the “path” to the wiki and is exposed in a $:/status/WikiName tiddler.
    • Is this a good place for a UUID?
  • While there are good examples on how to save “Yjs Documents” to persistent storage, they are mostly Database focused (browser & server).
  • This means that any wiki loaded from another source (a single-file or a node-folder) would need to be compared to a Yjs document, with the “incoming” wiki changes being used to update the Yjs doc (this in turn updates all connected clients). Server-restarts, loading from a single-file (or other source, say Fission) but connecting to a WebsocketServer, etc. This all needs examining.

There are many technical details about this setup, but feel free to just shout out ideas.

One pain point noticed so far: The Storylist UI does not differentiate from “close this open tiddler” and “discard/delete this open draft”. “Closing” an open draft tiddler immediately deletes it from the wiki, which can interrupt others editing their own drafts. A simple update of the icon for the “discard/delete this draft” to say, a trash-can, and adding a “miniminze/remove draft from storylist” button with the “X” would be a good start (thanks @pmario for this suggestion).

::Another Aside:: Would this be a set of features that the community would get behind as an OpenCollective task?

[boris]

I think you’re going to want to try and write a “user features” post that is completely separate from the technical details if you want the best user feedback.

I think the technical details are super interesting because of course I’m working on similar things

Even I’m having trouble parsing what you’re saying here.

Can you do a screencast of a demo? That might help too.

[stobot]

Unfortunately most of your post is way over my head, but I want to continue to offer encouragement and excitement around progress on multi-user. After using both the standard server functionality and many versions of BOB, I can submit a bit of user experience in those areas in case it’s helpful. What might be additionally helpful from your standpoint is a comparison to BOB, so that those of us that use BOB might understand the differences more clearly. (Note: I didn’t for this to be so long when I started )

First, from your post a few comments in the order you discuss them:

Multi user
I don’t really understand most of what you discuss here, and am not familiar with Yjs in general, but from a practical/functional standpoint, a 3 tier system is what I tried to force BOB into:

1. Admin (standard TiddlyWiki): What the main owner of the wiki can do - with password
2. Member: Can use functionality like buttons etc. provided by the Admin/author, but can’t for instance edit a tiddler from a UI standpoint
3. Guest: Read-only, can’t edit, and all buttons (and equivalent) are in disabled status

In BOB I try to accomplish this via wikitext alone which works fine since I’m the only one that understands TiddlyWiki. I have a hard-coded data-tiddler with usernames, passwords, and levels. This is obviously not secure at all, but good enough for team purposes. Admin is easy, Guest is relatively easy as it’s all just adding <$list> widgets around all of my buttons etc. Member is a bit tougher, and I end up having to hide the edit button globally, and then contiditionally show another one near the top for admins. Clunky, but it works.

Multi Wiki
This sounds mostly like BOB which works well. This is obviously a huge pain point in non-BOB, but the way BOB handles it is pretty great. Much easier than TiddlyServer by comparison.

Live Experience
I don’t understand much of your technical detail, but if this is like BOB, it’s magic and hugely useful. I know others here find that “serial editing” is good enough, and i’m sure it certainly has it’s use cases, but for me, the real-time nature of BOB is critical. Much of the use I have is to collect information from my team, often around scoring, ranking, and discussion. We will all co-edit in real time during meetings.

File Watcher
For me I’m constantly torn between the folder and file versions of TiddlyWiki. In general the folder versions have two main benefits: Quicker saving (per-tiddler vs. per-wiki) and integrations. I work in data analytics, and therefore have other processes as a result of batch files, Power Automate or other automations that now ingest information, and then translate / push into .tid files for BOB to injest. In this way it’s like a whole “connections” ecosystem that becomes available.

I don’t really understand much of the rest of that - Yjs / StoryList etc. so I can’t comment there.

Secondly, generally around the multi-user paint points generally: Bob is currently the best in all areas functionally so far, but here are my current concerns:

1. BOB seems to be much worse at re-connecting without data loss. I work in an office where I’m about 50% at my desk (on ethernet), but 50% of the day I disconnect with may laptop and switch to wifi around the building, then come back and plug back in. In the normal server version I’ve yet to have a problem, but with BOB it’s fairly common that I’m getting the red messages and when I use the sync-back-up it outright crashes BOB completely, kicking everyone off. That’s pretty painful.
2. I worry considerably about it being a non-core function and the risk that comes from that. Jed (I believe) is on hiatus, and he and the limited resources supporting it (including you) could abandon it at any time. While true of TiddlyWiki more generally, I worry much less about the core server version being abandoned.

Lastly, I’ve pre-emptively mentioned my willingness to sign on and continue supporting via open collective, though I do worry that the pool of people passionate about multi-user is smaller (which I continue to find surprising). I will say that if a product was out there robust enough, and with guaranteed support, people like me could support it via business expense, which would be a rich revenue stream if realized.

[joshuafontany]

@boris @stobot @everyone

Thanks so much for the quick feedback.

The state of my post is no accident, the project is showing early signs of extreme awesomeness, but starting out as a Bob fork and then having most of Bob quickly trimmed away has left it a bit muddled. And thus has left my thoughts on the project a bit muddled

I will try to get a clean single-wiki example going, and then start a new thread about the technical features and adding multi-wiki support.

[TW_Tones]

As a promoter of serial editing, I do so to enable single file wikis. In part because there is no simple secure way to publish bob to the internet. Bob.exe is fantastic, we just need to take it a little further, or see what @joshuafontany comes up with.

Can we host node server safely anywhere on the internet?

Perhaps someone could build;

• a bob/proxy docker container as a start.
• A safe internet facing version of Joshua’s Real time sync

[TW_Tones]

I am excited like you, but must admit parts make it hard for me to contribute. Perhaps separate features and technical methods a little more.

[knarfix]

Hi guys, I am a new writer here, but want to a my 5 ct to another multi user related question. But first I want to say that I am very happy about your project, @joshuafontany .

Given the attempts to create a multi-user capable server are successful (and I believe they will be), what else will be needed in TW to allow successfull shared use, and maintain as much as possible of the TW core values, like hackability?

I have thought about this for a while and would be happy to discuss in a seperate thought.
Let me try :

• a more elaborate and working “include wiki” functionality to allow users to add their own individual “decoration” on top of a shared wiki. Decoration includes convencience functions.
• conventions or better enforced rules what belongs in the “commons” wiki and what is kept private. Stripping off macros from uploads and more like this.

this would be a a thread and a task of its own, certainly, needed when real time sync and authorizations are there.

Cheers, F

[JanJo]

Hi @joshuafontany ,
this sound really great, but where can I find it?
Best wishes Jan