[tw5] Major updates to Chromium native file system saver plugin

Now with the indexdb entry re-populated, the sequence looks like this:

  1. Reload the TW page
  2. Click the + button to create a new tiddler
  3. Click the checkmark to save the tiddler
  4. A dialog box asks me if I want to let the site edit the file. I click the “edit file” button
  5. The file saves
    So it is working for me even without the settings modal. Do you see this same behavior?

I also think the modal isn’t needed. The API requires user interaction. … But I think the described behaviour is good. The permission is requested, when the first save happens. Since this save is a user interaction it should be good enough.

-mario

It does work, though I think it is disruptive asking as soon as something is done which triggers autosave. However, I have put in an option with version 0.7.1 to disable the modal for those who don’t like it.

Hi,
I do like that option.

-mario

1 Like

Folks,

I believe this is already available on tiddlywiki.com at https://tiddlywiki.com/#Saving%20on%20Browser%20with%20File%20System%20Access%20API

It is not yet comprehensively documented and it is hard for me to determine what level of functionality and customisation is available to us. As a solution only on Chromium browsers it is not yet global in application, so understanding its value is even harder to determine.

I also ask myself “We require a click event to start the save dialogue” if this could not be placed in a save button “lookalike” or another way to make it user friendly. Ie just in time, not startup, Although in this thread others suggest they do not need it.

Can someone write a user designer perspective and/or comparison with existing methods?

My concerns;

  • How to design online tiddlywikis with a non-intrusive saving mechanism users can understand.
  • Dealing with the contention possible with two parties editing the same site.
    I would appreciate it is someone can spell this out a little more for us who need it, and can’t easily understand this from the jargon and reading between the lines in this discussion.

Thanks

Tones

As inspiration, this seems to be a decent implementation of native file storage API: https://bangle.io

As inspiration, this seems to be a decent implementation of native file storage API: https://bangle.io

This does look pretty good. Another example is https://app.diagrams.net/.

These two apps have a webpage at a well-known url which holds the javascript functionality. The data files are loaded in separately. Compare this to tiddlywiki in which the functionality and the data is combined together into a single file.

I think to implement the same workflow in tiddlywiki, you’d have to have a “launcher” instance of tiddlywiki which would read a tiddlywiki instance from disk as the “data file”.

I seem to recall https://tiddlywiki.fission.app/ implements such a launcher, but currently that page has an endlessly spinning “Authorizing with fission” message and the console has an “Uncaught (in promise) Error: Improperly formatted header value: skeleton” in webnative.js, so I couldn’t confirm my memory.

I think the workflow implemented by the above two apps is “safer” than what I saw in the TW chromium native file saver. With the TW native saver the workflow looks like this:

  1. Load my native saver enabled TW using some url (possibly a file:// url)
  2. Click the Save button in HTML Native File System Saver modal
  3. From the file dialog select the same file I’m already editing
  4. Dialog box “a file with this name already exists, do you want to replace it?”
  5. Start sweating a little bit…if I’ve chosen the wrong file here, I might be overwriting something important
  6. Sweat a little bit more especially if I’ve loaded it from a web url where it isn’t as easy to tell that I’ve selected the matching file or not
  7. Cross my fingers, click the “replace” button and hope for the best
    The bangle and diagrams.net applications don’t have the same room for user error since you are prompted for what file to read and then it automatically saves back to that same file. I find that workflow to be less nerve-wracking.

Maybe with tiddlywiki’s unique structure there is an even better workflow to be had, I don’t know. And maybe the TW nativesaver can already be used with a better workflow and I just missed it.

1 Like

Hi Brian,

I seem to recall https://tiddlywiki.fission.app/ implements such a launcher,

That’s right. TiddlyDesktop also has a similar architecture. The challenge with Fission is that images stored in ones Fission drive are not accessible via a URL; there’s just a JS API to retrieve specific files. I have experimented with using a service worker to map local URLs to the JS API, which I think might be a promising generic technique for working with non-URL-based storage.

but currently that page has an endlessly spinning “Authorizing with fission” message and the console has an “Uncaught (in promise) Error: Improperly formatted header value: skeleton” in webnative.js, so I couldn’t confirm my memory.

I hope to be able to work on TiddlyWiki on Fission soon, and will make an update to the latest version of the SDK.

Best wishes

Jeremy

I think this approach might work within the Local Storage solution that CJ has developed. Unfortunately I am supporting a couple of “life and health” issues just now, so I am not able to test that assumption personally…

Hans

Aside: CJ, being my other handle.

For reference, if of use, my recent posts related to TiddlyWiki and local storage:

For contention, the current version actually does provide some integrity measures. One of the reasons for introducing the save button at startup is so the plugin can quickly read the original file and compute its file hash as soon as possible.

One the first save (i.e. when you click the button on the launch modal), the hash of the serialized wiki is recorded. On subsequent saves, the plugin compares the hash of the new version to the hash of the old version to see if the file has changed unexpectedly (i.e. changes were made in a different browser pane or it is on a network drive and someone else changed it).

I’m hesitant to rely too much on Tiddlywiki internals to try and maintain forward compatibility, but it would be nice if this check were possible without triggering an actual save. The other reason is to avoid unexpected prompts when autosaving things obviously.

I will also admit that as a software developer, I’m not the best person at making people understand the saving mechanism without overwhelming them with technical details.

As for using a more intuitive save button, I’m welcome to suggestions and/or pull requests so long as things scale for different browser/wiki layouts and it doesn’t pull in any large images (an SVG or something would probably be ok). I agree the button used right now is kind of ugly. Another thing to keep in mind with suggestions is that people might be using different color schemes which I’d like the plugin to try and respect.

Thanks for your feedback!

I actually implemented something similar in an experimental branch to the launcher idea a while back but abandoned it because it felt very clunky to me slaymaker1907/TW5-browser-nativesaver at sync-adaptor (github.com).

The biggest problem was that sync adaptors don’t work as nicely as regular savers with falling back to another saver in case of errors. If people were interested in that approach, it would probably be better done as a separate plugin and just duplicate code since savers work very differently compared to sync modules. It’s also a lot more work since there is much less documentation on sync adaptors compared to vanilla saver modules.

One thing I still like about the sync adaptor/folder wiki idea is that versioning can be done much more cheaply (at the cost of recovery complexity if you want to go back to an old version). Instead of just saving the whole wiki each time, the plugin could just save the old version of the changed tiddlers.

If the API supported it, I’d love to just save the old versions in a compressed file so it takes up less space. However, that isn’t really practical since JSZip is a huge dependency for one plugin and because the file system API can’t do changes in place. You have to completely write the whole file each time.

Thinking about that issue though, I could possibly implement gzip compression for the individual versioned files. Plain gzip at level 5 gives a compression ratio of ~17% for the example wiki (so compressed it is 17% the size of the original) which is quite a bit. I don’t want to do some sort of diff or patch algorithm since I don’t know if I trust myself enough to implement such a thing correctly plus you can’t just use 7zip in that case to recover the data. isomorphic-git · A pure JavaScript implementation of git for node and browsers! might also work so it’s just a git repo in that case.

In terms of stressing if you picked the right file, assuming you used the modal button, you don’t need to stress out too much since the consistency check should catch that.

Jeremy, is there an easy way to serialize the wiki to a string without triggering a save? That would allow me to give a better integrity check if people don’t save immediately. I can’t just “await fetch(”./example-wiki.html")" because of dumb browser restrictions with file://. That was what I initially tried to do.

Thanks,
Dyllon

Dylon,

One of the reasons for introducing the save button at startup is so the plugin can quickly read the original file and compute its file hash as soon as possible.

In some of my own experiments, in a startup action if the wiki is available for editing I immediately save and reload it to indicate this session “owns it”. Similarly when finished the final save needs to release it.

  • With a splash screen advising its loading this may be ok because in need only be done once per session (that has the wiki for edit/save) even on larger wikis.

I have come across two issues;

  • The browser session needs a finger print to be saved in local storage, so this can be compared with the loaded copy of the wiki to check ownership
    • I have found an almost unique fingerprint
  • If the wiki is on file:// I have not found how to get a seperate finger print for each tab, so it could be opened in more than one tab and save will only save the current tabs changes.

Have you found a way to open a file:// wiki in two tabs and not confuse the session?

It’s not really a locking system like you describe. Instead, it basically works by comparing the contents of the data that was saved last time to what it reads off of the disk. If it doesn’t match what it expects, an error is reported. It uses hashing to make this take less memory, but that doesn’t make that much of a difference besides reducing memory consumption.

My suggested route for recovery if an error is reported there is to attempt to save again. It will fall back to the download saver and you will have to merge the two versions manually using tiddler import/export.

I’ve uploaded a YouTube video demonstrating how the check works. Note that since the plugin only cares about contents, the wiki needs to be non-identical (usually happens immediately due to modification timestamps).

It would definitely be possible to store a session fingerprint in local storage or IndexedDB to check for issues locally. However, that doesn’t prevent issues with concurrent modifications from different browsers or on different computers via network drives.

1 Like

Dyllon,

[…]

In terms of stressing if you picked the right file, assuming you used the modal button, you don’t need to stress out too much since the consistency check should catch that.

The stress I describe in steps 1-7 below is only about using the modal button. During the first save, the consistency check is not done, right? Therefore, that first save during each session is a potentially damaging operation. That’s the source of the stress.

Folks,

I am watching this with “bated breath” ( in a nervous and excited state anticipating what will happen) however as I do not understand the technical part of the saver I am finding it hard to contribute.

It seems to me this is approaching the “Nirvana” of savable wikis without any additional installation of servers, savers or Browser addons. This will transform how people can acquire, use and distribute wikis to naive users. It just needs a little more work.

What follows is some ‘innovative’ solutions I have developed that may be able to smooth the path to a solution with the least “resistance” of not almost transparent.

  • You may also find these tips and tricks powerful in their own right.
  • ‘Innovative because’ “I see few if anyone using them”

I believe I can add a lot here but there remains a slight technical barrier for me. What I want to see is to minimize user interaction including between reloads, and when forced interaction is necessary, improve and guide the user for minimum complexity. Here are a few techniques that may help overcome the barriers.

  • Copy path to clipboard: If we want to save a tiddlywiki to an existing location, as may be forced when reloading the wiki or browser, we already have the full path and filename available in the browser, it just needs to be available to the save as dialogue.

    • I have created a save button with additional actions that copies the contents of $:/info/url/full to the clipboard, where ctrl-V pastes the full path and filename and I just click save.
    • This would save the need to remember the full path and filename and reduce the error of selecting the wrong file
    • Remember also
      • the full path tells you if the wiki is at a file location which is a useful piece of information
      • If we have a modal pop up as mentioned below in “Bait and switch”, this can be explained for the user, perhaps a variation for the detected OS.
  • Custom Target/tab: If we want to protect against opening the same wiki in a tab, one trick is after the first save (knowing the full path and filename) we can construct a html link with an appropriate target attribute, then navigate to that (if possible closing the original tab), thus the wiki is now running in a tab with a named target.

    • If we bookmark or pin the tab this remains the target. If clicked in future it will open the very same tab, never another new tab.

      • Warns if unsaved changes exist in the tab.
    • This should help given the global file:// issues

  • Bookmarklet: Perhaps extending the value of Custom Target/tab and because there is value using it, we can consider the use of a bookmarklet both with a target and a payload if necessary. An example may be so if the user returned to the source site, published on the internet the bookmarklette can recall you already have a local copy and with a click. When saving from the source URL for the first time we could prompt the user to use their saved bookmark if they want to access their local copy, to not create a new one, or override and existing one.

  • Bait and switch: Using the existing save button or a customized copy which does more work than the default save button, on this occasion opening a modal with an explanation rather than adding another tiddler and set of instructions would be wise. One could also allow the full path top be copied to the clipbord etc…

    • However this modal would not be displayed if it can go ahead and save automatically.
  • Local storage: As long as one does not tax the local storage much, it can act as a fairly safe space to store changes, and if when closing the tab the dirty indicator triggered the Native file save, then all tiddlers are saved to file and the local storage vacated.

    • The point here is automated save is not necessary but can be optional. The user should be able to choose when they invoke the file saving.
  • File Associations: Most browsers on opening a tiddlywiki file, what ever its name, detect its html and opens it as expected. This allows us to save tiddlywikis as for example a empty.tw file. Helping distinguish it from all other html files. Then we can provide (yes install) a file association to force such files to be opened in a particular browser, in this case the one with local storage for that wiki such as the one containing the IndexDB. Perhaps a .twc (tiddlywiki chrome) or .twf firefox or .twe tiddlywiki edge would be useful.

    • Browsers also enable handlers for given extension/mime types to be set, which may be another avenue to improve user interaction more eg .tw*
  • Note: For me, if my default browser, or all browsers have the Timimi extension installed it does not matter in which browser I open the wiki it can save to the local filename.

    • Currently testing this solution I am using Edge as it is my only browser without Timimi installed. Timimi once installed is almost transparent at saving, its only problem is it requires both a local install and an addon for each browser.

    • There would be value if it were possible to choose or avoid the Native saver on file based wikis in browsers running Timimi.

[…]

I believe I can add a lot here but there remains a slight technical barrier for me.

What I want to see is to minimize user interaction including between reloads, and when forced interaction is necessary, improve and guide the user for minimum complexity. Here are a few techniques that may help overcome the barriers.

  • Copy path to clipboard: If we want to save a tiddlywiki to an existing location, as may be forced when reloading the wiki or browser, we already have the full path and filename available in the browser, it just needs to be available to the save as dialogue.

Does this one have any technical barrier for you? Sounds like you have it figured out. Sounds like a useful approach as long as the tiddlywiki is loaded from a file url.
[…]

  • Custom Target/tab: If we want to protect against opening the same wiki in a tab, one trick is after the first save (knowing the full path and filename) we can construct a html link with an appropriate target attribute, then navigate to that (if possible closing the original tab), thus the wiki is now running in a tab with a named target.

What needs to know the full path and filename here? The user will know and the browser will know, but the functionality running in the web page (i.e. the tiddlywiki instance) will not know. It only has access to the filename minus the directory location. The directory location is hidden from it.

The user has the information required to construct the html link you mention, but the tiddlywiki page does not.
[…]

    • This should help given the global file:// issues

I’m not sure what you mean here.

  • Bookmarklet: Perhaps extending the value of Custom Target/tab and because there is value using it, we can consider the use of a bookmarklet both with a target and a payload if necessary.

What do you mean by “payload”? Could you give an example?

  • An example may be so if the user returned to the source site, published on the internet the bookmarklette can recall you already have a local copy and with a click. When saving from the source URL for the first time we could prompt the user to use their saved bookmark if they want to access their local copy, to not create a new one, or override and existing one.

I’m not sure how the bookmarklet would know you already have a local copy
[…]

Brian;

  • Copy path to clipboard: If we want to save a tiddlywiki to an existing location, as may be forced when reloading the wiki or browser, we already have the full path and filename available in the browser, it just needs to be available to the save as dialogue.

Does this one have any technical barrier for you? Sounds like you have it figured out. Sounds like a useful approach as long as the tiddlywiki is loaded from a file url.

The only issue is the user needs to be told and/or reminded they can use crtl-v in the file dialogue. Its a trick and not commonly used, although easy to use. This is why if we can rely on local storage for small changes (no auto-save) and trigger a modal when saving we can present a modal to guide the user and address the need for the button click.

[…]

  • Custom Target/tab: If we want to protect against opening the same wiki in a tab, one trick is after the first save (knowing the full path and filename) we can construct a html link with an appropriate target attribute, then navigate to that (if possible closing the original tab), thus the wiki is now running in a tab with a named target.

What needs to know the full path and filename here? The user will know and the browser will know, but the functionality running in the web page (i.e. the tiddlywiki instance) will not know. It only has access to the filename minus the directory location. The directory location is hidden from it.

The full path and directory is known by a tiddlywiki loaded from file. If a user stores the full path and filename in a wiki it also known to that wiki (eg store in a cookie)

The user has the information required to construct the html link you mention, but the tiddlywiki page does not.
[…]

Unless it is given and stores the full path and filename, however if we have a bookmarklet it can store the the full path and filename including a target tab. Clicking the same bookmark will always open in the same tab avoiding the opening of multiple tabs and the global file:// issues(see next notes)

    • This should help given the global file:// issues

I’m not sure what you mean here.

Just the “problem” with all file tiddlywikis having access to the same file:// domain. However it could be used for communication between wikis if we wanted. However as a rule we want a file:// tiddly wiki opened only once in a browser (or any browser) so there is not “contention”

  • Bookmarklet: Perhaps extending the value of Custom Target/tab and because there is value using it, we can consider the use of a bookmarklet both with a target and a payload if necessary.

What do you mean by “payload”? Could you give an example?

  • An example may be so if the user returned to the source site, published on the internet the bookmarklette can recall you already have a local copy and with a click. When saving from the source URL for the first time we could prompt the user to use their saved bookmark if they want to access their local copy, to not create a new one, or override and existing one.

I’m not sure how the bookmarklet would know you already have a local copy

The bookmarklet only exists if you created it from a local copy, it contains the link and target but can include a payload ie a tiddler containing details about the local copy. Thus the bookmarklete knows you have a local copy.

A trick that can be used is on arriving at an internet site we want to save a local copy we use the existing mechanism to save the local copy, then we can open that copy (ideally closing the internet tab #1). Once you have opened the local copy, you have access to the full path. Now so we do not loose the link to the local copy we can provide a link (we build it and can add a payload) for users to drag into their favorites and bookmarks.

#1 - before closing the internet tab

  • Save a cookie (or more) for that site indicating the existence of a local copy.

  • When our internet wiki loads it can test for these cookies, and advise the user a local copy exists.

    • It can then ask ONCE the bookmark to be clicked, dropping the “payload tiddler”, then the internet site can also save the full path into the cookie(s)
  • Subsequent visits to the internet site in the same browser can list all local copies of itself and provide links to its target tab OR allow new copies to be saved locally.
    I know my replies are somewhat convoluted but this is a somewhat complex problem with substantial features if we can achieve it. I believe we just need a little more tweaking and a strong guided workflow.

Yes, though assuming you use this plugin in another writer, it should be detected on the other side. The only way to really run into problems is with very rapid race conditions. Something like:

  1. Window 1: Initiate a save, confirm the original value to be X.
  2. Window 1 stops executing due to OS/browser scheduler.
  3. Window 2: Initiate save, confirm original value to be X.
  4. Window 1: Finishes write Y.
  5. Window 2: Finishes write Z.
    While unlikely, this could happen and any changes in Y would be overwritten with Z.

I did some investigation, and it looks like it is possible to serialize the wiki for comparison before the first save. I’m investigating that, but there are some weird scenarios where the first save is slightly different from subsequent saves.

1 Like
  • Copy path to clipboard: That’s a good idea and should be possible when loaded as file://. UI design is a consideration here to make it clear to users when copying the path.

  • Custom Target/tab: Do you mean something like Open Links in a New Tab, Or Re-Use Already Existing Tab (usefulangle.com)? I think this only works when clicking on links from a common page.

  • Local storage: Not really all that safe since we can’t guarantee that changes will be flushed to disk before the browser closes. If going with that kind of approach, I would rather just use a JS lock which can work with tabs in the same browser window.

  • File Associations: Might work, though that requires some platform specific code such as an installer to setup the associations.

For Timimi, there is obviously the solution of disabling the saver via a setting, but that obviously sucks if you have multiple people using the wiki. It’d be nice if there was a way to specify a priority for savers, but I don’t think there is one right now. I don’t want to add a bunch of custom logic to this saver for other specific savers. I could add in some special field containing JS that can be modified for determining whether to enable this saver or not, but that seems like a lot of complexity for users.

1 Like