Can I pull data from browser storage?

#1

I have been experimenting with browser storage. I was wondering if it is possible to pull data stored from a first wiki into data storage into a second wiki. For example, I have a Tiddlywiki we will call “writings,” and I would like to pull certain data I have placed into browser storage using the Browser Storage plugin for use in a second Tiddlywiki I will call “Processor.”

In case it is important, I work with Windows 10 using Chromium-based browsers, such as Chrome, Edge, Oprah, and a little known browser called Avast.

Merv

#2

Welcome @Mervin.

In the “Filter” tab of the Advanced Search, you can enter this filter:

[[$:/temp/BrowserStorage/Log]get[text]enlist-input[]]

This will display all the tiddlers which were loaded from browser storage at startup. You can click the “Export Tiddlers” button to the right and select “JSON file”. Then drag the downloaded JSON file into your “Processor” tiddlywiki. This will import the tiddlers from the JSON file.

It might be better to instead use the following filter. Its output should have the tiddlers which were saved to browser storage since startup in addition to those already present at startup:

[[$:/temp/BrowserStorage/Log]get[text]enlist-input[]] [haschanged[]subfilter{$:/config/BrowserStorage/SaveFilter}] :and[unique[]]

Beware the Browser Storage plugin in TiddlyWiki 5.2.2 and below has a bad data loss bug fixed in this commit: Allow browser storage plugin to delete existing tiddlers by btheado · Pull Request #6625 · Jermolene/TiddlyWiki5 · GitHub. The 5.2.3 pre-release has the fix. I think the release of 5.2.3 is expected within a few days.

#3

Yes, Welcome @Mervin,

Someone asking questions like this, the first time you post to talk.tiddlywiki suggests to me you may have more to share with the community, so I hope we see you here more and if you want to have more casual discussions have a look at the Cafe Topic.

I would have answered similarly to @btheado, but thought I would add a little more. What @btheado’s solution shows you is, you do not need to address local storage directly and that more often than not, as long as you can find the list of relevant the tiddlers you can just export, import, flag, save package, zip, drag and drop tiddlers as part of tiddlywikis native features.

However, If you open the same wiki (file or URL) in a separate tab it may have access to the same local storage, this can cause problems, but may also be a mechanism to achieve what is implied in your question.

However it may be helpful if you share the “bigger picture” you are trying to achieve because there is most likely multiple paths to the outcomes you desire.

If however, you open different wiki’s with a file address, in separate tabs, file wikis have access to each others local storage, but there are no tiddlywiki features to make use of this yet. As a result I started a conversation Inter wiki communications in browser and other features

  • An example may be to provide a way to package a group of tiddlers in a json tiddler and share this tiddler in local storage/cookie etc… from which any file based wiki can import.
  • Recently I started thinking about a review process where Alice edits one or more tiddlers and packages and sends the changed tiddlers to bob, bob imports and review the proposed tiddler changes and imports or rejects the changes.
  • I have started collecting a small set of JavaScript macros packaged for tiddlywiki of my own, and others but there is plenty of room for additional and innovative developments in this area.
#4

Browsers only allow you to use the same local storage, if it comes from the same origin.
See: Same-origin policy - Web security | MDN

#5

And to make this clearer;

Two URLs have the same origin if the protocol, port (if specified), and host are the same for both.

How the file:// protocol is handled is not so clear to me, but the browser themselves provide various types of storage local storage, Session Storage, IndexedDB, WebSQL and more. Chrome devtools show these. As far as I can see a file based wiki may not have access to any file or folder except those below its own folder (at most) however the browser provides resources that can be shared between browser tabs or other instances of tiddlywiki.

#6

Right. With that comes “fragility”? In the sense you can only ensure it works on one browser in one situation on one platform.

If that browser fails you loose your work?

I think my point is: why do it?

I can see cases where limited browser storage makes some sense; but I think they are limited.

Why are we advancing this rather than other methods?

Just a query, TT

#7

The file:// protocol is 1 origin, but we had to adjust the local-storage code, so different file-wikis can use “logical-storage” and don’t interfere with each other.

1 Like
#8

I thought it would be handy when working with TiddlyHost, so I didn’t have to save as often. Alas, I found it was very easy to go over the browser cache limit. Even though the actual tiddlers (including an image tiddler) would have been under 1 meg physically, it appears the cache was counting it differently, or expanding it somehow.

#9

Exactly. It all gets too complicated and unreliable unless you are uber-techie on this kind of thing. Which I am not.

Overall I think it fragile.

For short term use, say you want a student to post a report, it looks okayish. But beyond singular quick use I don’t trust it.

Just my paranoia, TT

#10

This is true but we often rely in this fact perhaps even without realising it. Until a save is committed to a server or disk this is true even when not using local storage.

  • In fact the addition of local storage can sometimes add an additional layer of security as long as you reduce the risk of loss by still saving and backing up regularly.
  • This is especially the case when you can disconnect from the server, and reconnect later.

Yes, an IndexDB solution mentioned somewhere recently would help overcome many of the local storage limitations as I understand it.

  • IndexDB could be a way to introduce other features where the wikis ability to save to server or file is limited. eg log changes, backup and file wiki communications

[Edited] A quick test shows that even an unsaved draft tiddler is retained in local storage if the tab is forced shut, or browser closes.

1 Like
#11

@Mervin this method of @btheado’s works on the tiddlers loaded from local storage, you need to reload, to see all that was loaded from local storage, but if you set the local storage “Browser Storage Save Filter” to [all[]], perhaps even [all[shadows+tiddlers]]you can be confident all changed, but not saved tiddlers, are in local storage.

  • if you want to determine what has change and is in local storage right now without a reload, a good proxy for this is the [haschanged[]] filter, when you combine this list of tiddlers with the list in $:/temp/BrowserStorage/Log then you basically know what is presently in local storage.
  • If you regularly save to file/server then the volume retained in local storage is minimal, only uncommitted changes, so you are not using much local storage and all the feared lost of data is actually quite low.
  • For your project @Mervin the local storage plugin will not necessarily contain the tiddlers you may want another (file://) wiki to access

I know this is not answering your question directly but I think it is important information for your project.

  • I will see if I can learn more and share back.
#12

Why not use the 2nd filter I shared in my post which uses haschanged and picks exactly the changed tiddlers which match the browser storage filter:

Maybe I misunderstand what you are trying to say.

#13

Sorry, I did not noticed the haschanged[] within -

I like the [haschanged[]subfilter{$:/config/BrowserStorage/SaveFilter}] :and[unique[]] filter, nice stuff.