Concept: plugin for non-hierarchical filesystem integration

[damscal]

Hello TW community,
this is a concept I’ve been wondering about for some time.

There are use cases where it would be nice to be able to explore/edit our node.js wikis with external applications relying on the filesystem. For example, we may want to do a complex metadata query of our audio tiddlers and open them as a playlist (something that wouldn’t be possible just searching for filenames), or to edit our text tiddlers from our favourite editor, all directly by navigating the filesystem from our applications. Some times one can also wish to be able to establish relationships between tiddlers as easily as moving/coping files around in file managers.

Traditional filesystems are strictly hierarchical, therefore not a good option for our use case. However, virtual filesystems such as FUSE can handle non-hierarchical structures very well. Tagsistant and to a lesser extent TMSU are good examples of this. These programs couple a virtual fs with a database, translating filesystem navigation to database queries. Tagsistant also translates some filesystem actions (copy, move, rename, delete) to propagate the changes into the database.

Depending on the capabilities of the TW CLI, it could be that the same could be done with TiddlyWiki on node.js, perhaps even not needing a database.
This approach would make it possible to interact with tiddlers through virtually any filesystem-aware application or platform (the virtual fs can be shared over a network), skyrocketing the number of possible TW use cases and making many common/repetitive operations (such as tagging) much quicker and more intuitive.

For example, in Tagsistant, moving a collection of files from a folder ./tag1/tag2/@/ to ./tag2/tag3/@/ removes the tag tag1 and adds tag3 for all those files, and this can be done from any file manager, even over a network. The tags order in the path address doesn’t matter, ./tag1/tag2/@/ and ./tag2/tag1/@/ always show the same files, which are all files tagged both tag1 AND tag2. Other logical operators are available and quite complex queries can be quickly performed simply navigating the virtual filesystem.

This would not replace TW frontend, but it would open up several possibilities.

Below is a diagram of how an hypothetical plugin “vnodeFS” could operate:

So this thread is to discuss the possibilities, see if it’s something the community would want, or to point to alternative solutions.

Dam

[smily]

You’re crazy, but I love the idea. You could either add it to the nodejs edition as a native addon or have a standalone thing that calls the REST API.

Recent Windows versions also seem to have FUSE-like APIs now: Windows Projected File System (ProjFS) or Cloud Sync Engines. The plot thickens.

[TW_Tones]

Interesting for sure, but could we not start with a “Virtual - Virtual File system” to browse tiddlers that runs inside tiddlywiki file or folder? This would provide a prototype for your proposal but a tool in itself.

In some ways it makes sense ensuring interaction inside tiddlywiki reflects interaction with tiddlywiki outside (File folder) or virtual file system.

[pmario]

Hi Dam,

TLDR; I think most of the things you describe can already be done with a wiki-folders. The important thing is, that we need to be able to work with Windows and Unix system.

FUSE file systems don’t work well with Windows. Unix directory mounts are complicated and Windows “Junctions” are insane.

The TW way as described below, is a bit tricky but preserves our sanity

---

The nodejs version of tiddlywiki already contains several mechanisms, that allow us to load external files into a wiki and make them look like tiddlers in 1 wiki.

The default structure of a tw wiki-folder looks like this:

myWiki/tiddlers/
  |    |
  |    + New Tiddler.tid
  |    + test.txt
  |    + test.txt.meta
  + tiddlywiki.info

Where the tiddlywiki.info file contains global info about the wiki folder

In the tiddlywiki.info link you can see a parameter named: includeWikis, which allows us to include the content from 1 wiki into an other wiki.

I personally use that a lot to allow my eg: bundler-server wiki to include content from the bundler wiki folder. So the -server can load additional plugins, that I need for development, but the bundler wiki only contains content to be published. …

The structure of my bundler dev environment looks like this:

bundler/tiddlers/
  |     |
  |     + New Tiddler.tid
  |     + test.txt
  |     + test.txt.meta
  + tiddlywiki.info

bundler-server/tiddlers/
  |            |
  |            + config/
  |            |    |
  |            |    + myConfig.tid
  |            + tests-in-server.tid
  + tiddlywiki.info
      {
       "description": "Server configuration of the wikilabs edition",
       "plugins": [
            "tiddlywiki/tiddlyweb",
            "tiddlywiki/filesystem"
        ]
        "includeWikis": [
             "../bundler"
        ]
       }

As you can see. The server tiddlywiki.info file includes the content of the bundler wiki and it includes 2 default TW plugins that are needed for the nodejs backend.

Those 2 plugins are not needed for the “bundler single file wiki”. So they are only used in the -server. … TiddlyWiki knows, where to find those 2 plugins because the environment variable TIDDLYWIKI_PLUGIN_PATH is set accordingly.

myConfig.tid and tests-in-server.tid are only needed for development, but shouldn’t be publshed with the “bundler” wiki. …

---

A second mechanism is tiddlywiki.files Files … As written in the docs:

A tiddlywiki.files JSON file in a sub-folder within a TiddlyWiki folder overrides the usual logic for recursively scanning the folder for tiddler files. Instead, the tiddlywiki.files file specifies instructions for loading tiddlers from specific files and folders.

We use the tiddlywiki.files mechanism to include 3rd party libraries, like the codemirror.js library, without the need to modify the library itself. …

The tiddlywiki.files mechanism can be used to implement something that you describe, in a relatively straight forward way.

I’ll try to “create” a structure similar to what you describe.

external/tag1/tag2/
  |       |    |
  |       |    + bbb.txt
  |       + aaa.txt

myWiki/tiddlers/
  |     |
  |     + files/
  |     |  |
  |     |  + tiddlywiki.files
  |     + ccc.tid
myWiki/editable/
  |     |
  |     + test-edit.txt
  + tiddlywiki.info

With the above configuration, you can move *.txt files between the tag1 and tag2 directory, and they will be automatically tagged in the wiki, without manipulating the files. … Important: The server has to be restarted after the files are moved

It’s only necessary, that you define the right parameters in the tiddlywiki.files file. The editable directory needs to be inside the “myWiki” folder for security reasons. So those files can be directly overwritten with the wiki.

If the file aaa.txt or bbb.txt are changed in the wiki, they will be created inside the myWiki/tiddlers folder. That’s intended.

If those tiddlers shouldn’t be edited, the ViewTemplate has to be adjusted, depending on the eg: tags.

tiddlywiki.files looks like this:

{
    "directories": [
        {
            "path": "../../../tag1",
            "filesRegExp": "^.*\\.txt$",
            "isTiddlerFile": false,
            "fields": {
                "title": {"source": "basename-uri-decoded"},
                "created": {"source": "created"},
                "modified": {"source": "modified"},
                "type": "text/plain",
                "tags": ["tag1"]
            }
        },
        {
            "path": "../../../tag1/tag2",
            "filesRegExp": "^.*\\.txt$",
            "isTiddlerFile": false,
            "fields": {
                "title": {"source": "basename-uri-decoded"},
                "created": {"source": "created"},
                "modified": {"source": "modified"},
                "type": "text/plain",
                "tags": ["tag1", "tag2"]
            }
        },
        {
            "path": "../../editable",
            "filesRegExp": "^.*\\.txt$",
            "isTiddlerFile": false,
			"isEditableFile": true,
            "fields": {
                "title": {"source": "filename"},
                "created": {"source": "created"},
                "modified": {"source": "modified"},
                "type": "text/plain",
                "tags": ["editable"]
            }
        }
    ]
}

There are some more mechanisms that can be used to dynamically move new tiddlers to different directories. See: https://tiddlywiki.com/#Customising%20Tiddler%20File%20Naming

---

The above code snippets are copied over from a working proof of concept wiki.

[pmario]

@smily

Recent Windows versions also seem to have FUSE-like APIs now: Windows Projected File System (ProjFS)

I didn’t know about ProjFS. Thaks for the link

[damscal]

Thank you all!

Recent Windows versions also seem to have FUSE-like APIs now: Windows Projected File System (ProjFS) or Cloud Sync Engines . The plot thickens.

@smily that’s interesting! Good to know that Windows is now implementing virtual filesystems. That should allow the proposed tool to be cross-platform.

Interesting for sure, but could we not start with a “Virtual - Virtual File system” to browse tiddlers that runs inside tiddlywiki file or folder? This would provide a prototype for your proposal but a tool in itself. In some ways it makes sense ensuring interaction inside tiddlywiki reflects interaction with tiddlywiki outside (File folder) or virtual file system.

@TW_Tones good point, it sounds like a good idea to start with. Eventually @joshuafontany 's Yjs project could achieve consistent real time two-ways syncing between node.js server and the local store.

@pmario Thank you Mario for your insights. That is an interesting setup, yet the point of this discussion is to let external applications browsing/querying our wikis. Example: you maintain your photos collection in your wiki, where you have metadata for each file. Now, you’d like to open and edit say every photo occurred during summer vacations in the last 5 years, but only if either your daughters Jane or Emily are among the subjects. You can manually export those image tiddlers, edit the files, restart the server and import the edited files back again, yet this is not very convenient if you are required to perform many different queries and frequent edits. This is just one example, but there can be many others.

A thing we would need for this to work is to define a syntax to translate path strings to TW filter queries.
In Tagsistant, navigating the virtual filesystem with a path like ./{/Deep Space 9/Voyager/}/{/video/sound/}/-/{/length:/minute/gt/5/date:/year/lt/2015/} is equivalent to the logical query ("Deep Space 9" in tags OR "Voyager" in tags) AND ("video" in tags OR "sound" in tags) AND NOT (length > 5 minutes OR date < year 2015). It’s quite straightforward and intuitive. Given the rich ecosystem of TW filters, a broader syntax could be implemented.
In this particular case Tagsistant syntax seems very compact compared to current TW filters syntax (my understanding is that one would have to save at least 3 diffetent filter expressions into 3 variables and then combine them in one last expression, is that correct?
If something like [tag[Deep Space 9] tag[Voyager]] :intersecate[tag[video] tag[sound]] was a valid syntax then perhaps the result could be achieved with just one expression, maybe an idea for core development? @jeremyruston ).

In my opinion Tagsistant’s syntax would be a good starting point,
in virtual filesystems there shouldn’t be strict limitations for allowed characters. However / (and \ in Windows) are probably going to be an issue.

[jeremyruston]

Hi @damscal I’m keen on the idea of exposing the tiddler store via interfaces other than the web. The one that I keep thinking about is actually IMAP, the attraction being that mobile devices have decent native clients for IMAP. One would create a new tiddler by creating a message, and perhaps tagging via dragging messages into different folders.

@damscal

maybe an idea for core development?

We are steadily improving TW’s filter language, and certainly recognise the limitations at the moment. One very tentative idea is a way of creating local variables within a filter.

I think it may be too late to adopt Tagsistant’s syntax, it seems very different.

[pmario]

@damscal

Thank you Mario for your insights. That is an interesting setup, yet the point of this discussion is to let external applications browsing/querying our wikis.

In your OP you wanted to move files from 1 directory structure to an other directory structure and the Wiki should change the tags according to that structure.

For example, in Tagsistant, moving a collection of files from a folder ./tag1/tag2/@/ to ./tag2/tag3/@/ removes the tag tag1 and adds tag3 for all those files, and this can be done from any file manager, even over a network.

That’s exactly, what my setup can do. You can move your files on the filesystem. Restart the server … done. Tags are changed in the wiki.

My setting also showed the possibility to change a file from the wiki and save it back to a directory that an external program can see.

The main limitation here is security. … A server can’t be allowed to write files back “outside” of it’s wiki-directory. … Otherwise it would be possible to “go up the directory tree” … up to the users home and read or write data from there. …

[pmario]

@damscal

Example: you maintain your photos collection in your wiki, where you have metadata for each file. Now, you’d like to open and edit say every photo occurred during summer vacations in the last 5 years, but only if either your daughters Jane or Emily are among the subjects.

That’s a new example, that also should be solvable, with the existing functions we have at our disposal already. … We would need to decide, how the directory structure should look like to achieve this. …

In your drawing you defined the following communication path:

A vnodeFS plugin would need to communicate with the node.js server and the server would need to communicate back …

From my point of view nothing new has to be invented here. We have a very reliable mechanism to communicate structures. … The file-system itself.

You defined in your OP what the tag1/tag2 structure means. Files in the tag1 directory should be tagged “tag1” … Files in tag2 directory should be tagged: “tag1” and “tag2”

The same can be done with “family/daughters/emily” and “family/daughters/jane” … We only need to establish this as a convention. It’s like an API definition on a “directory structure” basis instead of an REST API with URL paths. … There isn’t even a big difference, how they look like.

@damscal

You can manually export those image tiddlers, edit the files, restart the server and import the edited files back again, yet this is not very convenient if you are required to perform many different queries and frequent edits. This is just one example, but there can be many others.

I think, it’s not needed to manually export / import something … As I showed in my example, it’s possible, that the wiki can read files from outside and show them with tags depending on the configuration.

So eg:

• Every image in the “family” tree can be shown with a tag “family”, without the need to modify the file.
  • No extra “meta date” needed, since the directory structure already contains this info. … per “convention”
• We can start from there.
• At the start all images can be in the “family” directory.
  • Images from mother, father, jane, emily and the dog can be mixed in 1 directory.
• The server can read them and show them with the tag them “family”
• The server can read “created” and “modified” date from the file timestamp
• So some “simple filters” can already be created in the wiki
  • Eg only show images from 2020 …
• Now the user can have a look at the images in the wiki, and add more tags.
  • eg: Emily for images which only show Emily. … and so on

There is a relatively new mechanism in TW that I didn’t mention yet. … It’s called filter cascades

Such a filter cascade can be used to create new tiddlers in other directories than the default directory.

The documentation about this mechanism is: Customising Tiddler File Naming

So a filter that can “create/manipulate Emily’s metadata” could look like this:

[tag[family]tag[Emily]then[family/daughters/emily/]]

So the wiki would create 2 new directories like “family/daughters/emily/” and save the file there.

Using that mechanism and defining more of the cascade filters as above would allow us to create meta data in the wiki, which in turn will result in a new structure in the file system.
eg:

[tag[family]tag[Emily]then[family/daughters/emily/]]
[tag[family]tag[Jane]then[family/daughters/jane/]]
[tag[my lifeblood]then[family/together/]]
[[family/]]

and so on. … There is 1 speciality with “filter cascades” … The first filter, that creates an output will be used to define the “file structure”. … So the order of the elements matters. AND the last filter should contain a filter that always returns something. In my example above that’s [[family/]]

Important: I didn’t try the above configurations, so I’m not 100% sure if there are no hickups, but the principle if already part of TiddlyWiki.

That’s it for the moment.
Have fun!

PS: I do have about 24,000 images on my harddrive and it would definitely be worth a try, if TiddlyWiki is able to manage that, in a way we discuss here.

PPS: The “file saver” filter cascade was developed lately, to be able to improve the TW documentation file structure.

[TW_Tones]

@pmario this sounds interesting but it it wise to use tags?, perhaps an alt-tags eg “folder-tags” so it is easy to separate the functionality and possibly automate it more. I have some wikis where a lot of tags are in use, it may be hard to read which are driving folders and which are not.

[damscal]

Hello,

@jeremyruston That’s great to know . Even in this scenario, I’d still be thinking about a virtual filesystem as the outer layer for external applications interaction. The reasons are:

• A virtual filesystem can be designed to appear as a regular one to applications, so pretty much any fs aware app can access it. It can then be remotely shared with all kinds of protocols, not being forced to use one in particular.
• A virtual fs can be non-hierarchical, so that it can be designed to navigate wikis in many different ways, non mutually exclusive.
  For example users could navigate to a folder ./tiddler1/$backlinks/ to find all backlinks to tiddler1. Then they can continue navigating to ./tiddler1/$backlinks/tiddler2/$tags/ to find all tags of tiddler2, which links to tiddler1. This would be a network navigation. An advantage is that users are not requested to remember exactly a large number of navigation keywords, because the filesystem shows them the options they have (in the form of folders) while they are navigating.
  Another example is the kind of navigation implemented by Tagsistant (explained above), where a filesystem path corresponds to a logical query.
  Another option could be allowing users to associate specific dirnames with filter expressions. So that for example navigating to ./$filter/saved-filter/ shows the results of the filter expression associated with the dirname saved-filter.
  All of these navigation features are not necessarily mutually exclusive, they could be combined to work together. For example, ./tiddler1/$backlinks/$filter/saved-filter/ can be designed to show the results fo saved-filter, while receiving the tiddler titles listed in ./tiddler1/$backlinks/ as inputs. There should be very few to no limitations, I’d guess almost every navigation/query feature that TW can offer can be translated to work on a virtual fs.
• A virtual fs can be designed to translate all kinds of actions. For example users could navigate to a ./$commands/$new-tiddler folder where creating a new folder automatically creates a new tiddler. Then they could navigate directly to this new folder e.g. ./$commands/$new-tiddler/my new tiddler/ and find the .tid file there. Creating/editing text files in ./$commands/$new-tiddler/my new tiddler/$fields/ could update the tiddler fields… and so on.

@pmario your work and posts are impressive, and your setup can be certainly be of good use to many users. However, I can be sure it wouldn’t work for me… the frustration coming from being forced to organize my files in rigid hierarchies is what led me to TiddlyWiki (besides its powerful transclusion features). I know I would be spending countless hours writing an endless tiddlywiki.files, I would constantly be changing things and in the end never be satisfied with the result, because my brain (as most human brains) doesn’t work in a hierarchical way, but rather in a relational way. So I think that if I implemented a setup as yours it would only be for backup purposes in the unlikely event of having to switch from TW to something else.

When dealing with files I’d be looking forwards to the following things:

1. When I look for files, I want to be able to follow my line of thought instead of getting lost in large hierarchical structures (hierarchies are only functional when they are small). I want to input the most important keywords that come to my mind, get a list of related keywords, select those that are relevant, and repeat, incrementally narrowing my search until I’m satisfied with the results. Also consider that we don’t remember most of our files. Many times we don’t perform a search with a particular target file in our mind, we want to see what’s in there that is related to what we have in mind. Our brains work in terms of attributes, relationships and actions, not in terms of categories and boxes.
2. When I search for files (in a virtual fs) I really only want to get symlinks. I want my real files to stay in a static archive, each of them having a URI path that never changes, so that when I’m configuring applications I’ll provide those static permalinks and never have to re-configure them again if I change something. I change stuff (names, metadata, ecc.) a lot.

These are not problems within TW, but they are if I need to work with external applications.

Best wishes,
dam

[jeremyruston]

Hi @damscal indeed, you have a very interesting vision. I’m not so sold on the idea of encoding complex queries into paths, but the flexibility required to do so is certainly interesting.

[pmario]

That’s a very interesting discussion and I did start to write a reply, but I saw, I’ll need at least an hour or more to write it, to make sense. … I don’t have the time at the moment. I need to update all my TW plugin-editions to 5.2.2 first.

… to be continued

[TW_Tones]

I too cant follow the full implications of what you propose without more thought but i am very interested.

However I question your reasons and motivations in a purely logical way because the problems you voice can already be solved in the wiki ui with additions and tools.

Of course it would take time to address each but an example we can already build multiple alternate even overlapping and interconected hierarchies even fuzzy lists search and search suggestions all of which can help meet your apparent requirements.

Part of your motivation to build a new thing may be because you have not realised the existing potentials to meet your requirements. I do not want to discorage your efforts, please continue but I would encorage you to look closer at what is already possible. Feel free to ask. Perhaps in a different thread.

[damscal]

Hello,

of course, I’m very open to whatever solution that minimises developing time and maximises usability.
I recognise TW UI’s potentials, yet there are times where users need to have seamless read/write access to tiddlers from external applications as well.
It would be nice to have a good-for-everything UI, but in the philosophy of not having to reinvent (and maintain) the wheel each time, sometimes it’s more cost-efficient to take advantage of external apps instead.

Now this concept is unconventional and it’s not high priority (esp as long as filter language is in steady development and real-time two-ways sync with the local store isn’t available yet), so it’s good for it to stay on hold and mature in the back of our heads.

Mario’s solution can already work for simple hierarchies (thank you a lot @pmario ), I would use it for the time being.

Best wishes,
dam