Announcing the MultiWikiServer Plugin

#63

Have a closer look at GitHub PR – Usage section.

#64

Thanks, I got it up and running, but couldn’t find a way to create an empty instance with only the core and the necessary plugins. The edition/multiwikiserver is pre-populated with docs, dev-docs and tour

#65

Hi @Anjar as @pmario says, there are some instructions in the readme. However, I do need to update the docs to clarify how MWS relates to editions and wiki folders:

To use MWS requires a wiki folder that specifies the $:/plugins/tiddlywiki/multiwikiserver plugin in its tiddlywiki.info file. The wiki containing the MWS plugin is used for administration and configuration, while user data is stored within the /store folder inside the wiki folder as a SQLite database (and a folder for attachments).

At the moment, MWS automatically creates the “docs” and “dev-docs” wikis and populates them with the content from the respective editions. In the future it will be possible to configure which editions are made available as wikis.

You should be able to ignore them in use, though, you can just create whatever recipes and bags you need.

2 Likes
#66

Thanks @jeremyruston - this answered all my questions. Great work as always, this is a very exciting path forward for tiddlywiki!

#67

I have a branch which brings up MWS with no recipes (it does have the test bags) - is current with Jeremy as of right now 2024-06-26.

git clone https://github.com/PotOfCoffee2Go/TiddlyWiki5.git --branch multi-wiki-startup
cd ./TiddlyWiki5
npm install

In your package.json reference it by ‘file’ with the path to where you installed it.

"dependencies": {
        . . . 
    "tiddlywiki": "file:../TiddlyWiki5"
  },

Any changes to ‘TiddlyWiki5’ take effect upon restart.

Changed plugins/tiddlywiki/multiwikiserver/modules/startup.js to skip loading TW recipes and commented out console log of saveBagTiddler debug messages .

* 0360fa9dd (HEAD -> multi-wiki-startup, origin/multi-wiki-startup) Skip loading TW recipes
* 4c8da2788 Comment out startup debug messages
* 0ff3875b8 (origin/multi-wiki-support, multi-wiki-support) Update readme.tid
*   38e1ea8f9 Merge branch 'master' into multi-wiki-support
|\  
| *   0031a95df Merge branch 'tiddlywiki-com'
| |\  
| | * eecd40723 Avoid accented characters in filenames
1 Like
#68

Thank you @poc2go, that’s brilliant!

#69

I’m currently testing with the test recipes/bags. For a totally empty SQLite database also comment out lines 164-175 in plugins/tiddlywiki/multiwikiserver/modules/startup.js

#70

Ran into an interesting side effect of storing image tiddlers on MWS database.

I have a single file client wiki loaded from a server which stores/retrieves a subset of tiddlers to a MWS SQLite database, The wiki has a JS macro which connects to the server which in turn interfaces to the database. Another user far-far away, running the same single file wiki from the server can exchange the tiddlers stored on the database with others.

One of the use cases was to draw on an image; lines, arrows, circles - store on the database which could be requested by other clients who could add their drawing markup.

The user on the left edits the image with important scribbles and sends to the server. The user on the right receives the image w/scribbles - all is good.

edit-image
edit-image1735×639 162 KB

But when the user on the right wants to edit the image and add their all important scribbles, this happpens:

edit-denied

Editing is denied since the database has converted the image to an attachment.

Oh bother.

1 Like
#71

Prompted by your modifications I’ve pushed a refactoring that introduces a number of new MWS commands:

  • --mws-create-bag to create bags
  • --mws-create-recipe to create recipes
  • --mws-load-plugin-bags to load the core plugins into bags
  • --mws-load-wiki-folder to load a wiki folder (including processing the plugins in tiddlywiki.info)
  • --mws-save-tiddler-text to set the text field of a tiddler

I’ve removed the demo code from “startup.js”, replacing it with a “–build load-mws-demo-data” command that does the same thing using the above commands.

1 Like
#72

I have a plan to fix this, but I am uncertain whether it will be practical.

The idea is that when you click “edit” on a _canonical_uri image tiddler then in the background the draft tiddler will be synced to the server. When the server receives the draft via the sync process, it can inspect it and see that it is a _canonical_uri tiddler and then, if possible, it can replace it with the embedded version of the image tiddler.

The effect would be that after clicking the “edit” button on a _canonical_uri image tiddler there would be a brief delay where the _canonical_uri would be shown, and then it would be replaced by the full image for editing.

1 Like
#73

I think that should be a new toolbar button. I may want to rename, tag or add fields to the tiddler without editing the image.

2 Likes
#74

I kinda figuired you would be moving the loading process to the tiddlywiki.info file at some point. Thanks! can now use MWS out-of-the-box, no code changes required! Will set that up - got about 10 ‘server’ edition wikis that are loaded into the database for testing.

The mischief I have been up to:

The database is shared between MWS and my server by having my server boot up a tiddlywiki multiwikiserver edition which returns a reference to $tw. Thus both systems are using the same database via $tw.mws.store. So techically - there is only a single process accessing the database. Attempts to have two separate processes use the database did not go well.

Two of the TiddlyWikis are multi-user classics - a chat system and a dice game. My server handles the login, bearer token, and session controls (interface to clients via socket.io ). It manages ‘room’ tiddlers for chat and current state tiddlers for each user playing the dice game. Most of which is stored on the database in recipes/bags - got a few tiddlers left to move to it.

Concurrently, MWS is used on a different port to update the client side wikis during development. Was using ‘server’ edition but had to reload the wiki for the changes to take effect. With MWS the changes are live and immediate.

Setting up a reverse proxy so can finally have a demo facing the web. Turns out that is not as simple as it sounds.

Something to the effect of ‘tiddlywiki --mws-load ./mws/store --savewikifolder ./mywikifolder’ would be super handy. Which I am sure is in the works.

MWS is an awesome design and excellent system - works great and is not even finished yet! Having a lot of fun with it.

#75

Attachments are a new feature and I already have found that some type of special handling is needed. Some way for the author to specify a tiddler is to be stored as an attachment regardless of size or type; or specify the tiddler is not to be an attachment.

A checkbox on import should be able to handle that. A default can be automated, but the author is really the one to make the decision. For large media files indicate that due to size it will be saved as an attachment. The attachment checkbox would only be available on the MWS edition (or editions that use MWS).

A similar scenario is how images are handled by server edition. If I wish the image to be in the wiki (which is usually based on if I am going to distribute the wiki as a single file) I drop the image and import it - and the image is in the wiki. If the wiki is going to always be a Node.js wiki, I place the image in the ‘files’ sub-directory and use _canonical_uri. The important thing being - it is my choice.

Attachments should have their own field. _canonical_uri can exist anywhere and the author enters them, messes with them; exist on your machine, on the web, they can change if moved to a different host, etc. _attachment only exists on MWS, the uri is generated by and uniquely handled by MWS, they are special. For example, the author would almost never specify the uri. _attachment should for the most part be totally automatic. Best advise to an author would be to leave it alone - the only thing you can do is break it.

Attachments are a new and substantial addition to TiddlyWiki. The SQLite database has definite advantages but for the most part is transparent - is another tiddler store. But the attachments will be the unsung heroes of the MWS release (well, unless you got a song about them - if so, put it in an attachment). Would not be surprised to see attachments added as a ‘server’ edition feature at some point.

Now is the time to give some thought on how and where to integrate attachments into the TiddlyWiki Architecture; before the release.

Just thinking out loud.

#77

From a recent 20 year anniversary meetup Jeremy presented an overview of MWS, during which I possibly had a epiphany,

That the MWS server can host multiple-wikis where a wiki is effectively a recipe, and a recipe contains bags, that the server must have access to all recipes and bags.

Can I ask if someone who understands the direction of MWS can confirm or deny what I am saying and/or consider the necessary features to permit this approach?

  • I understand it is possible for example for the Sever to “share a set of tiddlers” to all wikis within it, allowing a server distributed set of config settings or content, to all wikis. We may call this a one (server) to many (wikis) relationship.
  • That each wiki can maintain its own bags, it is possible for a given wiki to share that bag with others or the server in a one to one relationship, or one to many relationship.

What I realized is if in a given (sub)wiki, access is limited to a single user, then it may as well be a private single user wiki, yet because the server can access it, it can define a wiki consisting of multiple single user wikis, the server wiki is in effect able to be a multi-user and multi-wiki tiddlywiki.

  • Sure each participating wiki needs the logic and tools to deal with what it receives, what it owns and what it shares, but ultimately it can represent a single authorized user of a somewhat “federated wiki”, or multi-user wiki

Thus if we have a multi-wiki server where its wikies are allocated to specific users then we have a multi-user wiki.

  • The designer will need to design this so the individual users of individual wikis are able to understand what they own and the scope of their influence on the greater wiki. Public, private and shared bags?

Thanks in advance
Tony

#79

See in the Announcements Categorie ans specifically Coming Up: Livestream 20th September 2024

#80

It is a correct that a wiki is made from a recipe, and that a recipe consists of a list of bags.

I don’t understand what you mean by “the server must have access to all recipes and bags”; recipes and bags are part of the server and so by definition the server has access to the recipes and bags that it contains.

The terminology isn’t quite right here. The server doesn’t exactly “share” anything; it would be more correct to say that recipes and bags can be setup so that a set of standard config settings or content can be shared between multiple wikis.

Wikis do not necessarily maintain their own bags. Bags are independent entities. A bag may be included in more than one recipe.

The idea at the moment is that access controls will apply to bags, not to wikis themselves. I don’t understand the rest of this section. It doesn’t make sense to say “the server can access it” because of the point above: bags and recipes are contained within servers, and so by definition the server can access all the bags and recipes that it contains.

There’s quite a bit more that I don’t fully understand: for example, the idea of a wiki being “allocated” to specific users, and the idea of a wiki “receiving” something. These are new terms without an established meaning and so might need to be explained to give context to others.

I appreciate that the terminology can be difficult. The PR at Introduce "Multi Wiki Server" Plugin by Jermolene · Pull Request #7915 · TiddlyWiki/TiddlyWiki5 · GitHub tries to establish a clear understanding of what the various new terms mean.

#81

Thanks for your feedback @jeremyruston

This is the way I am conceptualising it, for the argument I am putting. So yes.

Yes that is the challenge for me to understanding MWS

Or only one recipe that maps to a particular user or wiki?

Yes, I have read this multiple times, and in fact I suppose why am asking is a question of terminology.

  • Clearly the discussion so far is about multiple wikis, one could even say a server perspective.
  • I am trying to look at it from the perspective of one of the multiple wikis and or a wiki for a given user.

I am trying to answer two things;

  • Can we effectively build multiuser wikis where each user can only see or edit particular content and we can design which content is visible or editable elsewhere with appropriate permissions.
  • The method and MWS terminology we may use to describe this kind of implementation.

Background

It has being a long standing interest of mine to prepare multi-user wikis so teams can work together on tiddlywiki without risks of overwriting or contention, and some rudimentary permissions.

  • It appears this may be possible with MWS
  • TiddlyBob went a long way to doing this however is not reliabily maintained or supported, and I dont expect it to be, MWS supported by the core team may prove to be the best solution.
#82

I would [humbly] add on the general topic of MWS that single-user, local solutions which do not require knowing an IP URL have an immediate benefit as well.

There might be some lack of clarity on the plans for such a scenario.

So to be specific: much like running a local TiddlyWiki empty.html file, could an MWS wiki have an emptyMWS.html URL to be used locally by the 1-generation “child” wikis?

(I would assume that this is also greatly possible with a corresponding local sqlite file.)

file:///dir1/dir2/dir3/Downloads/emptyMWS_05.html
file:///dir1/dir2/dir3/Downloads/emptyMWS_05.sqlite

(If this is possible right now, then please let me know how to do this.)

And then a very related point of clarity: Bags can be drag-and-dropped to another (independent) wiki in a single act of importing a group of Tiddlers. When performing this identical action subsequently, does this mean that the bag still has enough definition in the receiving wiki, as to replace all old/altered locations of Tiddlers in the bag? I guess treating a bag like a plugin?

This would (desirably) mean that Tiddlers’ paths that are changed in the “upstream” bag - and Tiddlers that are deleted in the “upstream” bag - are also changed in the receiving local wiki after a new import of the bag. Note that this is different than a typical Tiddler export/import.

In this way, users could choose between a typical import and a bag import technique. Sometimes users want to import deletions and absent Tiddlers, but other times not.

#83

MWS is a not primarily a file server. If you see a file:// URL in a browser, it means that the browser did load the file directly from your local drive. There is no intermediate server with the configuration you describe.

As soon as there is a client / server configuration you will have an http(s):// URL protocol and a localhost domain name.

So an empty MWS wiki would look like this: http://localhost:8080/empty.html

Loading a file:// TWs is possible right now without MWS.

You will need to define your **Download Settings**, if you want to overwrite your file

I’m using Edge for the screenshots to show the browser default save mechanism.

  • Browser Settings → Downloads → Ask me what to do with each download and
  • Browser Settings → Downloads → Show downloads menu when a download starts

image
image998×451 22.7 KB

  1. This will allow you to select how and where to download the file.
  2. Select the filename or overwrite the existing file.

image
image1920×431 64.8 KB

  1. Select the “old” file name to overwrite the existing file – or
  2. Create a new file

image
image1081×501 34.3 KB

  • No – Bags can not be dragged and dropp imported into other wikis. A bag is a configuration file.
  • Bags are no plugins. Plugins are tiddlers with a type:application/json, that contain several other tiddlers, which are activated at wiki-startup.

A bag is an abstract configuration element, that contains meta data that defines which tiddlers should be requested from the database. This can be content tiddlers and could also be plugin tiddlers.

A recipe is a second configuration, that defines, which bags should be used to create a wiki. eg:

  • recipe - myPersonalWiki

    • bag: system … could contain the TW core, themes, palettes
    • bag: sharedPlugins … could contain plugins that I want to use with several wikis
    • bag: personalContent … Would contain some personal but public tiddlers

  • recipe - myWorkWiki

    • bag: system … could contain the TW core, themes, palettes
    • bag: sharedPlugins … could contain plugins that I want to use with several wikis
    • bag: workRelatedPlugins … could contain other plugins only needed fro work related stuff
    • bag: myWorkContent … Work related content

If the recipe myPersonalWiki is opened all the tiddlers in the bags, which are listed in the recipe will be loaded. The principle is: The last one wins

So tiddlers in the personalContent bag are loaded last and will overwrite everything from bags which are listed prior to personalContent.

eg: sharedPlugins may contain plugin-A – If you import a newer version of plugin-A into your personalContent bag, those plugins will be activated. Overwriting the plugin from sharedPlugins.

Everything in workRelatedPlugins will overwrite tiddlers from system and sharedPlugins – If and only if, there are tiddlers with the same title.

So the user is in full control of their content

Changes in every bag should be dynamically updated. But if plugins which contain javascript code are updated a browser-tab reload will probably be needed.

I do not know, how this is resolved in MWS at the moment. So @jeremyruston will need to jump in here.

hope that helps
-mario

1 Like
#84

What does MWS Plugin have to do with a “plugin” at all?