Solutions for syncing the same TiddlyWiki across many devices?

#23

For the last couple of months, I have been using a set up in which I have copies of a single file TiddlyWiki for notes on my desktop, tablet and phone, and the three copies of the same wiki synchronize with each other on a tiddler level.

The sync is via an in-development plugin (Harmony) that synchronizes tiddlers to a private GitHub repo which essentially serves as a free sync server, with the user not requiring any knowledge of git. Any changes to a wiki configured to sync to a given repo are automatically propagated to other wikis syncing to the same repo.

Essentially this is the same model used by apps such as Evernote, where you have different instances of the app on different devices that can all work separately and offline, but they also sync with each other via a server. Here, the private GitHub repo plays the role of a no configuration necessary free server, and you retain full control over your data. Creating a private GitHub repo is very easy even for non-technical users, whereas setting up your own server securely can be a complex undertaking.

This setup allows for use cases like synchronizing a wiki between devices at the tiddler level; ensuring that changes in one tab aren’t overwritten by an instance of the same wiki open in another tab or device; or being offline for an extended period and syncing any changes made to other devices once online.

The driver for this design choice is that I need offline access to my wikis at all times, so online hosting solutions are a non-starter for me. Were online hosting an option, I would likely choose TiddlyPWA.

Currently the plugin works robustly enough that I have been using it on a daily basis since February. It would need a rewrite to make it usable for a wider audience, which I will admit that I am currently struggling to find motivation for given that it already works very well for my very specific use cases, and I am not sure that many others would find this useful.

PS: a useful though unplanned benefit of this approach is that you get a per tiddler history of changes in the GitHub repo and it would theoretically be possible to revert a wiki back to a specific time in the history of changes.

3 Likes
#24

Looking forward to see this however;

I was hoping MWS Multi-wiki server could do this to an internet hosted SQL Server, that can be placed on any apatche/cpanel host.

1 Like
#25

You would also need to deploy MWS TiddlyWiki on node.js on the server to interact with the SQL database.

1 Like
#26

Wow @saqimtiaz - that’s almost my exact need as well and I haven’t been able to solve it. I’m offline such a big chunk of the time as I move between work and home (long train commute) and trying to keep things all in sync is very difficult. If that ever became public I would jump all over it. I’m not very technical, but do have a github that I’ve just used for TiddlyWiki page hosting etc.

3 Likes
#27

Do the actual contents of each commit, and the diff from the previous commit, look sensible? And does it matter? And what would the commit message be??

They’re pretty atomic. I save the Wiki every time TiddlyDesktop loses key window status (so if I’m switching apps), which generally is a good time to commit the wiki and push it. My commit message is “auto commit from ___”, which fills in what device (phone / computer) did the commit. I forgot to mention that this script is also auto-git pulling at a similar interval.

Does this Git-based solution allow dealing with sync conflicts?

It does, but only superficially. I also have TiddlyDesktop quit when I sleep my machine (and relaunch when I wake it), because I’ve had issues like this:

  1. Edit something on my phone
  2. Go back to my computer, where TiddlyDesktop is still running with contents prior to (1)
  3. Edit the wiki, which saves, and clobbers the phone changes from (1)
1 Like
#28

I understand this is still in development. But once ready, it sounds like it could be ideal as a multi-user solution (for cases where a wiki is shared among familiar trusted users who cannot connect on the same local network). Is that right? If it requires all devices to interact over the same GitHub account, that could be mildly inconvenient (if at least one user also wants to do other things with GitHub that would not be shared), but not necessarily a deal-breaker.

I’d love to see multi-user experiments of as many kinds as possible. And any per-tiddler sync solution is going to have a great advantage (in minimizing potential conflicts) over single-html solutions. Reduced bandwidth demand is also very attractive!

2 Likes
#29

I imagine it doesn’t. A GitHub repo allows you to name collaborators. Each user would need a GH id, and be given permission to the repository, but that should do it.

2 Likes
#30

I knew someone would make the leap to multi-user setups :slight_smile:
I am actually unsure how well that would work. The present sync logic assumes that multiple devices are not editing the same wiki in parallel, but rather one after another as is the case when a user switches between devices. There are certain limitations to what the GitHub API allows in terms of retrieving data and how often one can talk to the API, so while improvements in this area might be possible, it isn’t something I have investigated at present.

2 Likes
#31

One of the challenges here is that if you switch from device A to device B when moving from the office to the train, you would need to open the wiki on the device B to let it sync before going offline. Otherwise you would risk editing having edited a tiddler on device A and then without that change propagating to device B, editing the same tiddler again and losing earlier changes from device A.

I would be happy to discuss this further privately or in another thread to understand if Harmony would be useful for you to test out.

1 Like
#32

That’s very informative thanks @session. I’ll be sure to check out /siteinfo when exploring Hubzilla instances. Hopefully Hubzilla is still actively developed/maintained that it’s around for the long term!

#33

As the others have said, this sounds exciting especially if it enables multiuser edits. Please share in case you every decide to publish this plugin!

#34

As far as I know, TiddlyPWA only requires online access the first time you access it from a given browser/device. After that it can always be accessed offline. It has a service worker for offline access of the app wiki and a local copy of the tiddlers are stored in the browser’s indexdb. Are your offline requirements even more strict than that?

1 Like
#35

Past experience has shown me that I cannot trust browsers to persist data over longer periods of time, which is a deal breaker for my particular needs. In addition to that the ability to easily move the wiki to another device while offline has come in handy in the past as well. For others with a bit more flexibility in this regard, TiddlyPWA would indeed be an option for offline usage as well.

2 Likes
#36

Neat, I have also been considering using GitHub as the “tiddler store backup” for my sync cases.

I’m also still exploring Yjs for real-time sync. With Yjs local-first apps, you would need the internet to load the initial “wiki/page/room”, but then all edits once there are “non-deterministically conflict free” with all other copies of that doc ID. While there could be “have a REALLY old copy of a tiddler open somewhere, and edit that after a bunch of newer edits elsewhere” race-condition, but at least you would have a clean backup-history if it was also persisted in plain text to GitHub.

The final sync to github could be at a slower rate, if you are saving all live edits to browser indexeddb AND broadcasting to any listening wikis (local or remote networks). Consider real-time multi-user mouse-cursor tracking, like on https://kinopio.club - where certain things like mouse position, or tiddler-card-being-dragged xyz-position, should be streamed in realtime to other clients, but should NOT be persisted to an edit-history/backup.

Great conversation. Would love to check out Harmony when you feel ready.

1 Like
#37

you find the code here: hubzilla / core · GitLab

Version 9.0.2 was just published few days ago

1 Like
#38

Problem when use both clold sync adaptor and filesystem adaptor User Interface - SyncAdaptor - Local DB/FS - Some Custom Sync Mechanism · Jermolene/TiddlyWiki5 · Discussion #7720 · GitHub

Also you need to handle deletion bi-directional properly Can we sync "delete" event after client reloaded · Jermolene/TiddlyWiki5 · Discussion #7194 · GitHub

Are you writing a github sync-adaptor?

1 Like
#39

5 posts were split to a new topic: How do you add encryption to wikis in TiddlyHost?

#41

Got it. Thanks! Do you by chance know if you can use Quine2 with Tiddlyhost? I would like to view my wiki on an iOS devices. (I could just use the Safari app, if it isn’t possible.)

1 Like
#42

The implementation is indeed a working GitHub sync adaptor. It works well with the caveat that wikis in which a large number of tiddlers are created or modified in a short period of time could exceed the GitHub API limits. This is due to the fact that the core sync mechanism processes each tiddler change atomically, which translate to a separate git commit for each tiddler modification/creation.

The solution would be a rewrite in which we batch multiple changes within a minute or two as a single commit, potentially storing interim changes in IndexedDB to guard against data loss, much as @joshuafontany suggests.

#43

I’d like to hear more about this implementation. Is it a custom saver? Does it use tm-http-request?