TiddlyPWA: Secure offline storage and cross-device synchronization solution

[Newbie]

Thanks! That made me confident enough to give it a try . I’m still hestitant of having the admin page on a internet-front-facing for personal notes, but for keeping the Grok TiddlyWiki TakeAways and the Tutorial-built wiki seems great.

Here’s the steps I followed to migrate both wikis (Grok and tutorial-built wiki) I had on TiddlyDesktop into TiddlyPWA:

0. Export the wiki you want to import into TiddlyPWA as .json
1. Login to glitch.com
2. Go to ServerHosting/Glitch TiddlyPWA’s tiddler
3. Write the password you want to use to access the wikis admin panel
4. Click on the remix button, the glitch.com for the site will open with the readme
5. Click on “Preview > New Window” to open the admin panel
6. Use the password from Step3, the admin menu should unlock
7. Click “Create new wiki” button and write the name to identify the wiki information
8. Go to “TiddlyPWA — TiddlyWiki Storage & Sync Solution” and paste the URL from the admin page and the hash for the wiki you just created (Listed on the wiki list at the admin page) and click “Upload”
9. Go to the given URL by “~UploadApp”, copy the token used on UploadApp and set a new password. Only works on non-private tabs due to using the browser memory database.
10. (Optional) If the wiki has extra plugins, drag and drop them from “More > Plugins > Plugins” on the original wiki tab into the TiddlyPWA tab. Will require using the “Save” button so the new tiddlywikiapp.html is saved on Glitch
11. Click on “Import” from the “Wiki More options > import” and select the exported JSON from the original wiki
12. Click on “Import” button from the list of tiddlers shown to be imported
13. Reload the website and login if you did not save the password on the browsers
14. Sadly, the wiki configuration is not exported into the JSON, so just try to mimic the original wiki options, navigating from all the places on $:/ControlPanel and setting everything to the same values as the original wiki

Here’s the final result, looks like the images are exported as base64 on the JSON. It would be awesome to be able to also export configuration options on Wikis that rely heavily on it. Also I can not see the “Create your own” and “Help” buttons on for TakeAways but maybe is just a setting I missed setting up .
I did not see the usual “Install” button that other PWAs have, but I’m unsure if this is expected for all PWAs or not, from my complete ignorance on the subject:

Currently I’m using the clever save and export into Gitlab Pages from @boris for personal notes, which allows leveraging the private gitlab pages feature to put a barrier to the Wiki from the wide internet. It has it’s drawbacks, like having to reload the site between browsers restarts to ensure you are not overwritting your notes, or using loads of space if you do not clear the commits from the repository (I think this is due to the whole HTML being uploaded and it being encripted, currently is using almost 1GB and the wiki only has an small photo tiddler ).

Having the tiddlers auto-update and refresh is almost magical, and looks almost like the dream I have of tiddlywiki commiting just the tiddlers changes/creations/deletions into the chosen SCM for data backup from the hosted static site (Maybe someone already did this and I’m totally unaware ).

Hopefully this helps others to give TiddlyPWA a try as well

[Christoph13]

I was about to say that I got TiddlyPWA to work on my Mac and iOS devices which now sync instantly. Impressive!
Thanks @Newbie for your detailed description!

1. Is the procedure you (Newbie) describe above necessary? In principle one should be able to drag and drop the content tiddlers and the tiddlers needed for the particular wiki, e.g. for Groktiddlywiki there are

• GrokTiddlyWiki
• TakeAway
• TiddlyRemember

as per Grok TiddlyWiki — Build a deep, lasting understanding of TiddlyWiki

However, in practice I ran into problems even when deselecting tiddlers that were already in existence: a banner obstructed the top menu bar of the wiki and some more issues.
So, should the drag and drop work at least in principle and if not, I would like to better understand why not.

2. How does the syncing work? It appears way too fast to be exchanging an entire HTML file each save. What is the granularity of the saving/syncing mechanism and what is lurking in the shadows (if anything)? So far the experience is very good indeed but I have not tested large wikis with complicated content!

3. On my Mac, what needs to be done to use TiddlyPWA with Tiddly Desktop or should I access the wiki through Safari?

4. Where does the wiki handled by TiddlyPWA reside in the file system?

Thanks everyone and especially @valpackett, of course!!!

[Newbie]

@Christoph13

Is the procedure you (Newbie) describe above necessary?

To be honest, I do not think is necessary but convenient for wikis with a large set of tiddlers. I only used it as it was the recommended method by @valpackett on this post

The plugin drag and drop step exists because the plugins are not exported into the .json.

@Christoph13

How does the syncing work? It appears way too fast to be exchanging an entire HTML file each save.

TiddlyPWA was born specially to be an alternative that didn’t submit a whole .html each time (among other things), as the original blogpost mentions. For what I can understand from Synchronization and Encryption tiddlers and from @valpackett posts, it creates two categories for tiddlers, one with the “app” information (plugins, core, etc), and another as “data”, and then syncs the detected changes from “data” tiddlers into the browser’s IndexedDB (A kind of Cookie Jar on “steroids” that exists on modern browsers) and then upload the changes into the server (anyone with more expertise feel free to correct me ).

@Christoph13

On my Mac, what needs to be done to use TiddlyPWA with Tiddly Desktop or should I access the wiki through Safari?

What you suggest sounds like a good experiment. Having the wiki on TiddlyDesktop would make it easier to detach the wiki from other browser cache, although I’m not sure how well would play. Maybe we could just drag and drop the “TiddlyPWA” and “Web App Manifest” plugins into a TiddlyDesktop generated wiki and check how well it plays after setting up the sync server(I will try as soon as I have a chance ).

@Christoph13

Where does the wiki handled by TiddlyPWA reside in the file system?

If you refer on what you are currently seeing when the wiki is open then on the browser’s cache.
If you refer to where the TiddlyPWA Sync server is saving the “data” tiddlers, it is doing so into the sqlite3 file inside the hidden .data folder on the project’s root folder.

Last but not least, take all I wrote on this answer with a pinch of salt, as I’m learning how it works myself and I’m highly prone to errors and missunderstadings

[TW_Tones]

@valpackett I have long argued for a PWA implementation and I am excited that one is presented here, I would love to jump on the bandwagon, however I am currently time limited and in attempting to follow the instructions here, there is a lot of assumed knowledge or hit Brick walls for example I try “Try It Out – Right Here, Right Now” I get “Could not decode the salt”.

Perhaps we could consolidate, simply or extend the guidance to allow a larger audience to adopt it?

I would like to see this migrated into an easy to use solution that we can use to market the value of TiddlyWiki to a much broader audience, but for now, even with my technical knowledge I am not getting any where. Let alone to get a holistic understanding so I can compare it with other implementations.

[valpackett]

@Christoph13

On my Mac, what needs to be done to use TiddlyPWA with Tiddly Desktop or should I access the wiki through Safari?

Through Safari (or Firefox etc.) Trying to mix other solutions might lead to everything exploding

@Newbie

inside the hidden .data folder on the project’s root folder

Well, the default (Glitch dot com oriented) is .data/tiddly.db relative to where the server is started at, but the path can be configured via the --db flag or the DB_PATH environment variable.

[valpackett]

@TW_Tones

I try “Try It Out – Right Here, Right Now” I get “Could not decode the salt”

That… sounds strange. Ummm are you using password manager autofill? It might have autofilled something into the salt field under the “If you are going to sync a pre-existing wiki into this one, click here” dropdown >_< Yeah, the message is definitely confusing if the dropdown is actually closed. I’ll change it and try to add some anti-unexpected-autofill measures as well.

[Christoph13]

Silly question perhaps but does TiddlyPWA have access to the file system? I am on macOS and Safari atm, if that is of importance.
If not, how would I out-reference image files that I do not want included in the wiki itself?

[valpackett]

Nope! That’s not currently possible.

With current filesystem APIs I think it would be possible to sync external files without copying them to IndexedDB, but implementing that sounds like a bit of a nightmare. Starting with UX type decisions even. If we have add one external file on one device, what happens on other synced devices? TiddlyPWA is not Syncthing.

If you just want local references that don’t sync the files themselves… That should be a separate plugin actually (:

[ludwa6]

Like @TW_Tones , i’m both really keen on this most promising solution to what seems quite a popular problem, but am also finding its implementation a bit overly-challenging for a non-developer-type to adopt. Specifics of my problem were posted here (rather off-topic; sorry!), and answered by @valpackett here (quickly, too: much appreciated!).

Still, i’m not yet seeing how this works at the most fundamental level -for example saving. I am logged-in to the same test wiki i set up in 2 different browsers (Chrome instances, both); i’ve made an edit in one, it’s been saved and shows up in Recent Changes in that instance, but not in the other. I haven’t (yet) got access to a cloud server (still don’t know how to do that), and neither has this TW instance been downloaded to my desktop (as TW does by default w/o some cloud storage), so… Where does this instance with my 1st Test Tiddler reside?

Perhaps this software is not for the likes of a non-developer user like me, but i’m keen enough, i’m willing to invest some effort in this; i just have to ask: should i stay away until there’s a more n00b-friendly UI/UX about it? Or is it OK to bumble along thusly in this channel? Or is there a better channel for new-user-supportive conversation?

[Ste_W]

@ludwa6

should i stay away until there’s a more n00b-friendly UI/UX about it? Or is it OK to bumble along thusly in this channel? Or is there a better channel for new-user-supportive conversation?

Activly Bumble!

They are a friendly bunch and generally quite helpful.

[Christoph13]

Thanks @valpackett, but I was not thinking about a syncing solution (which TiddlyPWA is not) but merely the ability to

• link to files that exist locally on the computer’s filesystem
• maybe even embed images present in the filesystem without having to import the image files themselves to the wiki

I am clear about having to sync files myself.

[LoryMoney]

If your wiki is powered by nodejs there is a feature that does exactly what you are looking for.

Automatically Serve directory structure under Nodejs

In case someone is walking the same path, I got to semi-automatic solution that creates tiddlers mapping files (*). Following my previous example just by having a dog.png under directory /files/animals/ you can type {{/files/animals/dog.png}} wich renders the dog image out of the box. This automatically generated tiddler will have a parent field that is the directory in wich is located. Following the dog.png example, parent = “files/animals”. So far so god you create the missing tiddler “file…

[pasajeros]

Thanks @valpackett , I really like TiddlyPWA. I first tried it on glitch, now trying to setup in a docker container.
When I try to start the container - all env variables set and correct deno command, I think - I get:

....
Download https://deno.land/x/sqlite@v3.8/src/function.ts
Download https://deno.land/x/sqlite@v3.8/build/vfs.js
Download https://deno.land/x/lz4@v0.1.2/mod.ts
Download https://deno.land/std@0.159.0/encoding/ascii85.ts
Download https://deno.land/x/lz4@v0.1.2/wasm.js
error: Uncaught (in promise) Error: Run Deno with --unstable!
	if (!('BroadcastChannel' in globalThis)) throw new Error('Run Deno with --unstable!');
	                                               ^
    at listen (https://codeberg.org/valpackett/tiddlypwa/raw/branch/release/server/run.ts:9:49)
    at https://codeberg.org/valpackett/tiddlypwa/raw/branch/release/server/run.ts:32:29

Any idea?

In docker I’m using Deno 2.0.0-rc.5 and I think --unstable is deprecated, and for instance Deno.seekSync (function) as well. Maybe TiddlyPWA needs Deno 1.x.y?

[valpackett]

Yep, the current flag is --unstable-broadcast-channel. Updated the docs and released an update that upgrades the sqlite dependency to a new version that works on Deno 2 (and includes a previous decorator fix that didn’t get into the release branch), so now it should work well!

[pasajeros]

Hi @valpackett , I tested, it works ! Thanks a lot.

[Lamnatos]

@valpackett thank you, thank you for your great work! I’ve provisionally transferred three of my TW to TWPWA to see how it works for me.

Is there a way to change the “app” icon? All three of them have the same one and since I’m using the text-less option in iOS, I need to remember which one is which. It be even nicer if it could pick each wiki’s own defined icon (for example like regular bookmarks show the bookmarked TiddlyWiki’s .ico resource).

Here’s what I’m seeing right now:

[vilc]

You can set the favicon by saving an image as $:/favicon.ico tiddler, more info here: https://tiddlywiki.com/#Setting%20a%20favicon. This is what TiddlyPWA does,
Don’t be confused by the “.ico” in the tiddler title, it’s just a title. The image doesn’t have to be in ICO format. PNG is usually the best choice for compatibility with various browsers.
After changing the favicon, you have to remove and add the PWA desktop shortcut again for it to reflect the change.

[Lamnatos]

Huh, I knew about the $:/favicon.ico mechanism and I was already using it in the original wiki that I transferred into the PWA version. When I did not see my favicon in the new version I thought it was there but it wasn’t used.

I guess favicons are part of the things that are not exported by the “Export All” Tool. I also had the same issue with my custom ViewTemplate ordering, I had to recreate it by hand.

I wish TiddlyPWA had a function similar to TiddlyHost’s “Upload file into this wiki” that would get everything from the source wiki.

[Lamnatos]

After a bit of experimenting, I realized my best bet to carry everything over was to start from scratch. Instead of using the Export Tool, I dragged the whole HTML file into the PWA version and everything carried over properly. Tiddlers, plugins, settings, the whole thing.

[valpackett]

$:/favicon.ico is used by default, but you can also further customize the icons part of the app manifest by adding tiddlers with the $:/tags/ManifestIcon tag. type/sizes/purpose fields will be passed through to the manifest.