WebDAV wikifarm - further work

[saqimtiaz]

On the off chance that anyone in the community is already actively using my WebDAV based wiki farm setup (as described here), if you have any suggestions or requests let me know and I will take them under consideration.

I have started a second round of work on this that focuses on the following:

Implemented:

• support for IIS as a WebDAV backend

Work in progress:

• exploring syncing changes in a wiki across tabs, browser and devices without needing to setup a server by using a private Git repo instead; replaces:
  • optional session locks, and experimentation around cross browser tab syncing using the Broadcast Channel API to prevent data loss
• browsing and restoring wiki backups from the File Explorer UI
  • this may require breaking changes to how backup paths are configured
  • a rotating backups system that limits the total number of backups created
• an upgrade mechanism for upgrading wikis entirely from within the UI
  • updating plugins from third party plugin libraries as part of the wiki upgrade
• support for deleting and renaming files
• allow a server-root configuration setting for when the root of the WebDAV server is a subdirectory.

[TiddlyTweeter]

“Wiki Farm”? Is that like the beginning of https://youtu.be/UTDQnOqUe-Q?t=182

What does it do?

Who might be interested?

Just asking
TT

[saqimtiaz]

Considering that it will be at least another four weeks before I am able to further my work on this, I have uploaded a prerelease of the new version of the plugin here in case any existing users are interested in testing it.

Changes since the last stable release:

• support for IIS as a WebDAV backend, including with gzip compression enabled.
• a new backups policy to limit the number of backups created, which combines a Tower of Hanoi based rotating backups system (as implemented by pmario), with a milestone backup system inspired by work by Daniel and Simon Baird in the TiddlyWiki classic days (LessBackupsPlugin).
• the ability to browse and preview the backups for a wiki via the $:/Backups tiddler. The next step here will be to implement the ability to restore backups. The UX is very much a work in progress, especially in terms of integration with the file explorer and error handling when the backups directory does not exist.

@xcazin as the only member of this forum that I know of to be actively using this, I thought you might appreciate the heads up.

[pmario]

Hi Saq,
Thanks for the new update. I do have a problem – I’m using IIS and have wiki at: localhost/test/index.html

If I do request localhost/test, there is rewrite rule, which does a permanent redirect to localhost/test/index.html otherwise I cannot save the wiki. I alsways got a 409 (conflict) with a PUT to localhost/test/

also localhost is redirected to localhost/index.html → I do not want to allow a directory listing at root

But the test subdir localhost/test/index.html should allow it. ATM the “refresh directory” button requests localhost/, which always returns index.html and nothing else. If index.html is clicked, it opens localhost/index.html, which is the wrong one.

So I think WebDAV utilities need to recognise the “subdirectory” and request PROPFIND there and not on root.

But I’m not a 100%, what’s going on at: $:/plugins/sq/webdav-utils/startup-webdavls.js yet

[saqimtiaz]

Hi @pmario, I am not completely confident that I follow everything you are describing and cannot investigate until I return home in some weeks. However, the limitation that the plugin expects that the WebDAV server is served at the root rather than subdirectory is a known one that I intend to address by allowing a server-root configuration. The reminder is appreciated as I have not thought about this recently, I am adding it to the list of issues to address in the next release.

@pmario

If I do request localhost/test, there is rewrite rule, which does a permanent redirect to localhost/test/index.html otherwise I cannot save the wiki. I alsways got a 409 (conflict) with a PUT to localhost/test/

You should be able to save without that redirect with this plugin installed.

[pmario]

@saqimtiaz

However, the limitation that the plugin expects that the WebDAV server is served at the root rather than subdirectory is a known one that I intend to address by allowing a server-root configuration.

The server actually does contain a wiki at root, but there should be index.html only. Nothing else should be shown. Creating new files at root should be forbidden.

It has a subdir at localhost/test/ atm, which allows a save with your plugin. But it cannot list the directory as it seems, because it requests PROPFIND from localhost/ which has to be rewritten as of my requirements.

Just a reminder: If I do create a new wiki, it will also be created at localhost and not a localhost/test

I’ll check the code, if I have time and give feedback.

Thanks for the great work!

[saqimtiaz]

Most of what you describe just needs for the plugin to set and respect a server-root path and only treat everything under that path as WebDAV. This is something that I plan to implement.

In terms of things being forbidden, the best you can do on the client is hide functionality that you do not want. If you truly want to forbid creation of new files in the server root, that would need to be configured at the server end.

Lastly, I am not sure about the redirects and how you are using them. They should not be necessary with this plugin at all. If you are redirecting all requests from / or test/ then that will be problematic and not something that the client should be expected to work around. On the other hand if you just redirect GET requests, then that should work just fine.

@pmario

It has a subdir at localhost/test/ atm, which allows a save with your plugin. But it cannot list the directory as it seems, because it requests PROPFIND from localhost/ which has to be rewritten as of my requirements.

I feel there are two distinct issues here. Firstly, the known one that the plugin expects that the server root is server under WebDAV and tries to show the contents of /. This will be addressed. The second is that it sounds like / is served under WebDAV but is showing the wrong contents?

The PROPFIND to localhost/ does not return both the index.html and the test subdirectory? Check the debugger and the raw XML response returned for the request. If the wikis at /index.html and the /test directory are served under WebDAV by the same server, I would expect the response to return both, and for both to be displayed in the $:/FileExplorer tiddler. OR are you rewriting all requests to / and not just GET requests? The plugin needs to be able to make PROPFIND and MKCOL requests to directory paths.

Apologies if any of this is unclear, I am struggling to wrap my head around your server configuration and what it is exactly that you are experiencing and I suspect that is due to the redirects.

[pmario]

@saqimtiaz

Apologies if any of this is unclear, I am struggling to wrap my head around your server configuration and what it is exactly that you are experiencing and I suspect that is due to the redirects.

Thanks for your feedback. I think your thoughts are on the point.

I do suspect the same thing. Even if I’m using the IIS WebDav server for a long time now, I did always use it with my own webdav-lm plugin, which does need the redirects. Probably a software problem on the plugin side.

I did install the WebDav wikifarm and the file-upload plugin today for further testing. So it is very likely, that the server side redirects are the culprit here.

I think it’s very useful, to go through this trouble, because it will give us information about problems, that other users may also have. So we may be able to avoid them in the future

I think, I’ll need to read more IIS documentation and do some more experiments.

[btheado]

@saqimtiaz

Considering that it will be at least another four weeks before I am able to further my work on this, I have uploaded a prerelease of the new version of the plugin here in case any existing users are interested in testing it.

Have you pushed this to github? I don’t see it at GitHub - saqimtiaz/tw5-plugins-sandbox: Sandbox for experimental TW5 plugins with no support.

I ask because a few weeks ago I was looking at factoring out the UI explorer so it can be used by more than just webdav.

I can just get the lastest code from your prelease, so it isn’t a big deal if you don’t push the code any time soon. I’m not sure when I’ll get back to this again anyhow.

For example, here is some code which uses a refactored version of your explorer UI to browse the system tiddler “folder hierarchy”:

title: $:/SystemTiddlerExplorer

\define getDataTiddler() not-used
\define getDirectories() [all[shadows+tiddlers]removeprefix[$:]removeprefix<currentPath>splitbefore[/]suffix[/]addprefix<currentPath>sort[]]
\define getFiles() [all[shadows+tiddlers]removeprefix[$:]] :filter[removeprefix<currentPath>splitbefore[/]!suffix[/]] :and[sort[]]

\define navigate-actions()
<$action-setfield $tiddler=<<treeState>> text=<<currentPath>>/>
\end navigate-actions

\define file-click-actions()
<$action-sendmessage $message="tm-open-window" $param={{{[<currentFile>addprefix[$:]]}}}/>
\end file-click-actions

{{$:/plugins/sq/webdav-utils/ui/file-explorer/base}}

It is very incomplete, but the basic browsing functionality works.

This isn’t my usecase for factoring out the explorer UI, just an easy to test proof-of-concept. If/when I do a more thorough job, I was thinking of making a pull request. Hopefully it wouldn’t be a large burden for you to maintain and having it in your repository would make it easy for another developer to copy the relevant tiddler/tiddlers and use the explorer UI in their own project. Otherwise it can just live as a fork.

[saqimtiaz]

Hi @btheado,

This version of the plugin is from a development branch that isn’t currently on GitHub, though I don’t think the File Explorer code has changed at all from the last stable release that is in the repository. I wont be back at my desk for another 3-4 weeks so cannot push this branch until then.

I would highly welcome refactoring of the Explorer UI to be reusable. This version of the File Explorer was created in collaboration with @telumire and making it a reusable component was always the long term goal that we discussed but just never got around to. I was imagining that it would accept filters/functions which determine what to show, as well as what to exclude.

Please do feel free to make a PR against the main branch when time permits, and I’ll resolve any merge issues that might arise when I merge the dev branch.

Ultimately I would like to get to a point where there is an SQUI plugin with sub-plugins for components such as the Explorer to make them easier to reuse, for myself and for others.

[saqimtiaz]

@pmario

I do suspect the same thing. Even if I’m using the IIS WebDav server for a long time now, I did always use it with my own webdav-lm plugin, which does need the redirects. Probably a software problem on the plugin side.

The WebDAV utilities plugin tries to work around this issue by assuming that if the wiki is loaded from a location that does not include a filename, then the filename is index.html and this is appended internally to the URL. This allows wikis loaded from URLs like localhost/testing to be saveable. In the future I could imagine making the default filename configurable in case someone is using a different file extension.