[tw5] Major updates to Chromium native file system saver plugin

Google Group (Read Only)
#1

This is a plugin I’ve been working on for a while that allows you to save your wiki to the file system without downloading a new file each time and without requiring an external program besides the browser. It currently only works in Chromium (desktop) browsers such as Edge/Chrome since they are the only browsers which support the necessary APIs (though others may support it once the APIs are fully standardized).

You can see File System Access API | Can I use… Support tables for HTML5, CSS3, etc for browsers which will probably work. I have not tested it on Safari so if you try it out there, let me know how it goes.

I’ve implemented several new major features for slaymaker1907/TW5-browser-nativesaver (github.com) for those who are interested.

  • By default, the plugin now checks for file consistency issues (such as having the same wiki open in multiple tabs).
  • I’ve softened a bit on the security implications of saving file handles to IndexedDB when the wiki is served from file://. You can now avoid selecting the file each time via a setting.
  • In addition to the default saver style, you can also have a save trail of versions under a backups folder. It’s a bit clunky though so I would still recommend using a separate backup system for your wiki.
  • The plugin shows a settings modal at load time. The plugin needs an interaction event (such as a click) to use parts of the file system API and I found it takes me out of the flow to have the file picker show up after I’ve edited a tiddler.
    The new API for triggering modals from JS is pretty nice, but let me know if there is a way to trigger modals in a way that works with older wikis. I’m using “$tw.modal.display(tiddlerTitle)”, but this seems to be new to 5.2.1.
3 Likes
#2

I tried it and it worked for me. Nice job. From both a file: url and a localhost url I was able to save and reload. I was also able to save the file handle in the indexdb. I didn’t try any of the other new features.

But even when I had the file handle saved in indexdb, the settings modal still comes every reload. I’m not sure I understand the intended workflow, but it seems like it would be less confusing if the modal would only come when the indexdb file handle is missing. Is that possible?

Also, I notice you implemented the reset-file-saver message handler as a TW widget. In other code (i.e. https://github.com/Jermolene/TiddlyWiki5/blob/master/plugins/tiddlywiki/browser-storage/startup.js) I’ve seen it implemented via a startup module. Any thoughts on that?

#3

Dyllon,

I’m unable to get the hash comparison to work. Every time I save, I see a log message about there being no known hash:

$:/plugins/slaymaker1907/browser-nativesaver/saver.js:464 Saving with method ‘autosave’
$:/plugins/slaymaker1907/browser-nativesaver/saver.js:77 Input saver style [] is not recognized, using default of SingleFile.
$:/core/modules/utils/logger.js:52 saver-handler: Saving wiki with method autosave through saver slaymaker1907/browser-nativesaver
$:/plugins/slaymaker1907/browser-nativesaver/saver.js:379 No previous known hash to compare against
$:/plugins/slaymaker1907/browser-nativesaver/saver.js:413 Creating writable…
$:/plugins/slaymaker1907/browser-nativesaver/saver.js:415 Created writable
$:/plugins/slaymaker1907/browser-nativesaver/saver.js:418 Wrote data successfully
$:/plugins/slaymaker1907/browser-nativesaver/saver.js:422 Closed writable
$:/plugins/slaymaker1907/browser-nativesaver/saver.js:485 Successfully saved to file system

I never see the “Computing hash of text to save…” message in the log.

Also, when I click the “Reset file save location” button, nothing in the indexdb gets deleted. The button doesn’t seem to have any effect in spite of the log messages:

$:/plugins/slaymaker1907/browser-nativesaver/saver.js:495 Reseting file saver…
$:/plugins/slaymaker1907/browser-nativesaver/saver.js:506 Reset file save location.

Brian

#4

The modal is required even if using IndexedDB. We need a button click both to show the choose file/folder dialogue as well as getting write/read permission if using IndexedDB. If you have a wiki mostly for reading, you can click the “close” modal button to exit without setting up the file save location/file permissions.

Thank you very much for finding the bug with the consistency check! I rewrote a chunk of that recently when adding the save trail. I have fixed this and pushed the fix to GitHub and the example wiki (plugin version 0.6.2). I tested it out by opening the example wiki in two tabs and it was able to detect when the file had changed unexpectedly.

#5

This seems like it would be a real game changer. About 75% of the browser usage today is Chromium based. But it needs a bit of documentation from the user perspective.

What are consequences of using or not using indexDB? What are the consequences of using or not using the consistency check?

Thanks!

1 Like
#6

Dyllon,

The modal is required even if using IndexedDB. We need a button click both to show the choose file/folder dialogue as well as getting write/read permission if using IndexedDB. If you have a wiki mostly for reading, you can click the “close” modal button to exit without setting up the file save location/file permissions.

I commented out the call to $tw.modal.display and the saving flow is still working for me. If I clear the indexdb (via developer tools since the reset button isn’t working for me), then the sequence looks like this:

  1. Reload the TW page.

  2. Click the + button to create a new tiddler

  3. Click the checkmark to save the tiddler

  4. A dialog box asks me to choose a file to save to.

  5. I choose the same file I already have open and I’m asked if I want to replace the file and I say yes

  6. The file saves
    Now with the indexdb entry re-populated, the sequence looks like this:

  7. Reload the TW page

  8. Click the + button to create a new tiddler

  9. Click the checkmark to save the tiddler

  10. A dialog box asks me if I want to let the site edit the file. I click the “edit file” button

  11. The file saves
    So it is working for me even without the settings modal. Do you see this same behavior?

Thank you very much for finding the bug with the consistency check! I rewrote a chunk of that recently when adding the save trail. I have fixed this and pushed the fix to GitHub and the example wiki (plugin version 0.6.2). I tested it out by opening the example wiki in two tabs and it was able to detect when the file had changed unexpectedly.

Great, thanks for fixing it.

[…]

Also, when I click the “Reset file save location” button, nothing in the indexdb gets deleted. The button doesn’t seem to have any effect in spite of the log messages:

$:/plugins/slaymaker1907/browser-nativesaver/saver.js:495 Reseting file saver…
$:/plugins/slaymaker1907/browser-nativesaver/saver.js:506 Reset file save location.

Does the reset button work for you? The only way I’ve been able to get the save location to reset is to use developer tools to clear out the indexdb.

Brian

#7

[…]

Also, when I click the “Reset file save location” button, nothing in the indexdb gets deleted. The button doesn’t seem to have any effect in spite of the log messages:

$:/plugins/slaymaker1907/browser-nativesaver/saver.js:495 Reseting file saver…
$:/plugins/slaymaker1907/browser-nativesaver/saver.js:506 Reset file save location.

Does the reset button work for you? The only way I’ve been able to get the save location to reset is to use developer tools to clear out the indexdb.

I changed this line:

window.indexedDB.deleteDatabase(UNIQUE_PLUGIN_ID);

to this:

window.indexedDB.deleteDatabase(DB_NAME);

and now the reset button is working for me.

#8

Good catch, reset should be working again (version 0.6.3). (sorry for the spam Brian, I kept replying to you instead of the whole chain).

#9

…

Now with the indexdb entry re-populated, the sequence looks like this:

  1. Reload the TW page
  2. Click the + button to create a new tiddler
  3. Click the checkmark to save the tiddler
  4. A dialog box asks me if I want to let the site edit the file. I click the “edit file” button
  5. The file saves
    So it is working for me even without the settings modal. Do you see this same behavior?

I also think the modal isn’t needed. The API requires user interaction. … But I think the described behaviour is good. The permission is requested, when the first save happens. Since this save is a user interaction it should be good enough.

-mario

#10

It does work, though I think it is disruptive asking as soon as something is done which triggers autosave. However, I have put in an option with version 0.7.1 to disable the modal for those who don’t like it.

#11

Hi,
I do like that option.

-mario

1 Like
#12

Folks,

I believe this is already available on tiddlywiki.com at https://tiddlywiki.com/#Saving%20on%20Browser%20with%20File%20System%20Access%20API

It is not yet comprehensively documented and it is hard for me to determine what level of functionality and customisation is available to us. As a solution only on Chromium browsers it is not yet global in application, so understanding its value is even harder to determine.

I also ask myself “We require a click event to start the save dialogue” if this could not be placed in a save button “lookalike” or another way to make it user friendly. Ie just in time, not startup, Although in this thread others suggest they do not need it.

Can someone write a user designer perspective and/or comparison with existing methods?

My concerns;

  • How to design online tiddlywikis with a non-intrusive saving mechanism users can understand.
  • Dealing with the contention possible with two parties editing the same site.
    I would appreciate it is someone can spell this out a little more for us who need it, and can’t easily understand this from the jargon and reading between the lines in this discussion.

Thanks

Tones

#13

As inspiration, this seems to be a decent implementation of native file storage API: https://bangle.io

#14

As inspiration, this seems to be a decent implementation of native file storage API: https://bangle.io

This does look pretty good. Another example is https://app.diagrams.net/.

These two apps have a webpage at a well-known url which holds the javascript functionality. The data files are loaded in separately. Compare this to tiddlywiki in which the functionality and the data is combined together into a single file.

I think to implement the same workflow in tiddlywiki, you’d have to have a “launcher” instance of tiddlywiki which would read a tiddlywiki instance from disk as the “data file”.

I seem to recall https://tiddlywiki.fission.app/ implements such a launcher, but currently that page has an endlessly spinning “Authorizing with fission” message and the console has an “Uncaught (in promise) Error: Improperly formatted header value: skeleton” in webnative.js, so I couldn’t confirm my memory.

I think the workflow implemented by the above two apps is “safer” than what I saw in the TW chromium native file saver. With the TW native saver the workflow looks like this:

  1. Load my native saver enabled TW using some url (possibly a file:// url)
  2. Click the Save button in HTML Native File System Saver modal
  3. From the file dialog select the same file I’m already editing
  4. Dialog box “a file with this name already exists, do you want to replace it?”
  5. Start sweating a little bit…if I’ve chosen the wrong file here, I might be overwriting something important
  6. Sweat a little bit more especially if I’ve loaded it from a web url where it isn’t as easy to tell that I’ve selected the matching file or not
  7. Cross my fingers, click the “replace” button and hope for the best
    The bangle and diagrams.net applications don’t have the same room for user error since you are prompted for what file to read and then it automatically saves back to that same file. I find that workflow to be less nerve-wracking.

Maybe with tiddlywiki’s unique structure there is an even better workflow to be had, I don’t know. And maybe the TW nativesaver can already be used with a better workflow and I just missed it.

1 Like
Tiddlystow a(nother) native file saver
#15

Hi Brian,

I seem to recall https://tiddlywiki.fission.app/ implements such a launcher,

That’s right. TiddlyDesktop also has a similar architecture. The challenge with Fission is that images stored in ones Fission drive are not accessible via a URL; there’s just a JS API to retrieve specific files. I have experimented with using a service worker to map local URLs to the JS API, which I think might be a promising generic technique for working with non-URL-based storage.

but currently that page has an endlessly spinning “Authorizing with fission” message and the console has an “Uncaught (in promise) Error: Improperly formatted header value: skeleton” in webnative.js, so I couldn’t confirm my memory.

I hope to be able to work on TiddlyWiki on Fission soon, and will make an update to the latest version of the SDK.

Best wishes

Jeremy

#16

I think this approach might work within the Local Storage solution that CJ has developed. Unfortunately I am supporting a couple of “life and health” issues just now, so I am not able to test that assumption personally…

Hans

#17

Aside: CJ, being my other handle.

For reference, if of use, my recent posts related to TiddlyWiki and local storage:

#19

For contention, the current version actually does provide some integrity measures. One of the reasons for introducing the save button at startup is so the plugin can quickly read the original file and compute its file hash as soon as possible.

One the first save (i.e. when you click the button on the launch modal), the hash of the serialized wiki is recorded. On subsequent saves, the plugin compares the hash of the new version to the hash of the old version to see if the file has changed unexpectedly (i.e. changes were made in a different browser pane or it is on a network drive and someone else changed it).

I’m hesitant to rely too much on Tiddlywiki internals to try and maintain forward compatibility, but it would be nice if this check were possible without triggering an actual save. The other reason is to avoid unexpected prompts when autosaving things obviously.

I will also admit that as a software developer, I’m not the best person at making people understand the saving mechanism without overwhelming them with technical details.

As for using a more intuitive save button, I’m welcome to suggestions and/or pull requests so long as things scale for different browser/wiki layouts and it doesn’t pull in any large images (an SVG or something would probably be ok). I agree the button used right now is kind of ugly. Another thing to keep in mind with suggestions is that people might be using different color schemes which I’d like the plugin to try and respect.

Thanks for your feedback!

#20

I actually implemented something similar in an experimental branch to the launcher idea a while back but abandoned it because it felt very clunky to me slaymaker1907/TW5-browser-nativesaver at sync-adaptor (github.com).

The biggest problem was that sync adaptors don’t work as nicely as regular savers with falling back to another saver in case of errors. If people were interested in that approach, it would probably be better done as a separate plugin and just duplicate code since savers work very differently compared to sync modules. It’s also a lot more work since there is much less documentation on sync adaptors compared to vanilla saver modules.

One thing I still like about the sync adaptor/folder wiki idea is that versioning can be done much more cheaply (at the cost of recovery complexity if you want to go back to an old version). Instead of just saving the whole wiki each time, the plugin could just save the old version of the changed tiddlers.

If the API supported it, I’d love to just save the old versions in a compressed file so it takes up less space. However, that isn’t really practical since JSZip is a huge dependency for one plugin and because the file system API can’t do changes in place. You have to completely write the whole file each time.

Thinking about that issue though, I could possibly implement gzip compression for the individual versioned files. Plain gzip at level 5 gives a compression ratio of ~17% for the example wiki (so compressed it is 17% the size of the original) which is quite a bit. I don’t want to do some sort of diff or patch algorithm since I don’t know if I trust myself enough to implement such a thing correctly plus you can’t just use 7zip in that case to recover the data. isomorphic-git · A pure JavaScript implementation of git for node and browsers! might also work so it’s just a git repo in that case.

In terms of stressing if you picked the right file, assuming you used the modal button, you don’t need to stress out too much since the consistency check should catch that.

#21

Jeremy, is there an easy way to serialize the wiki to a string without triggering a save? That would allow me to give a better integrity check if people don’t save immediately. I can’t just “await fetch(”./example-wiki.html")" because of dumb browser restrictions with file://. That was what I initially tried to do.

Thanks,
Dyllon