Node-Red Based Low-Code Server to Dynamically Create TiddlyWiki's

[jeremyruston]

@poc2go

@jeremyruston by naming the package ‘TW5-Node-RED’ and the using variation of these logos am I looking at jail time in my future?

Hi @poc2go that is not a problem at all. We should have clearer guidelines about this stuff, but in brief, TiddlyWiki itself, the docs, the Motovun Jack mascot etc are all published under a BSD license which means that anyone can use and modify them for any purpose. We request an acknowledgement, but it is not required.

So, from a jail time perspective you’re automatically in the clear, and from a good citizen perspective, you’re in the clear by acknowledging TiddlyWiki.

Anyhow, I’m delighted you’re getting close to publication. These are very powerful capabilities that will enable users to explore new applications for TiddlyWiki.

[pmario]

@poc2go

• The Mustache delimiters are <% / %> since {{ / }} are used everywhere in WikiText. See Node-Red Mustache.

hmmm, This may cause a problem with the new “conditional shortcut” syntax that comes up with TW v5.3.2

[Scott_Sauyet]

@poc2go

The system uses single file TiddlyWikis stored on and delivered by the server. The purpose is to add server functionality to TiddlyWiki single page application where tiddlers are dynamically added, changed, and removed on the client by the server - based on user interaction.

Does this imply that it could also do multi-user TW? I’ll be looking sometime in the next few months at how TW could work for me multi-user. I’ve been planning on looking at how to extend the Node.js version to support multi-user, and am not looking forward to it. I’m curious if this might help fill that gap. Bob.exe may one-day fit my needs here, but at the moment,

• under consideration Security and authentication to limit access and editing

access control is only “under consideration”.

Also, is the database of tiddlers in-memory or stored to the filesystem or to some DBMS? If it’s stored, is it updated on every change, or only on stop/start, or on some other cadence? Part of my reason for preferring Node implementation is that I like frequent git backups of my data; this is easy and inexpensive if I’m only backing up a few (or a few dozen) tiddlers at a time, not duplicating entire wikis on every change. If this is saved in a text format (as Node stores tiddlers as text .tid files), then this would fit together well with that workflow.

Even if it doesn’t fit these needs, I’m very excited to see this coming together!

[poc2go]

Hi @Scott_Sauyet

The server is designed from the get go to be multi-user, so the short answer is yes. I will be posting more on TW Talk about the design of the server with some teasers about what it can do and how it does it. Putting together a demo.

TW5-Node-RED has two standard Node-RED installed storage systems - a memory store and a file store. The file store has options for buffering, retain in memory after write, throttling, etc. The default is physical write every 30 seconds which is fine for me. Can set it to whatever you want. The writes are not usually expensive as explained below.

Can use JSON, but almost all of the tiddlers are stored in .tid format. There is a twist - instead of having a separate file for each .tid - the servers file reader / writer takes a bunch of tiddlers and separates them with a line ‘+======+’ (exactly) between them. I have some files with 1000’s of .tid formatted tiddlers from csv data downloaded from the web in a single file.

For one - can just edit the file to fix a tiddler. Plus can just append to it to add more tiddlers - appends are fast cheap writes so easy on disk i/o. Another advantage is that if two or more tiddlers have the same title - the last one (ie: most current) wins. So have a history of all the updates to that tiddler in the file. Also simplifies resolving collisions that are bound to occur in the multi-user universe. So have both of the tiddlers that are in conflict - can be merged or whatever. The files are pruned daily by importing into TiddlyWiki (which handles duplicate tiddler titles as discussed - last wins) and export back out. And finally don’t have 1000’s upon 1000’s of files clogging and messing up my precious directories.

The server does have git version control built in. There is a mechanism to store tiddlers in Node-Red flow files which are under version control - just need to turn it on - give it a git repo, credentials, etc. I haven’t used it that much because the whole project directory which includes all the stuff is already under version control.

It might be a fit - The main issue is probably Node-RED. Flows need to be designed and wired up, move tiddlers here, store it there, publish to multiple clients - do the task you need done. There are nodes in Node-RED that will do that - and much more. There a learning curve to figure out Node-Red. You already know the TiddlyWiki side, the server needs to know what to do as well.

[poc2go]

@pmario

hmmm, This may cause a problem with the new “conditional shortcut” syntax that comes up with TW v5.3.2

Seems this is going to be a continuing issue. Thanks for the heads up.

I brought it up half a year ago hoping could find a sequence of two delimiting characters that are not used by
WikiText. Is a challenge.

I’m going to use section sign ‘§’ and broken bar ‘¦’ which are valid HTMLCharset characters - one at 166 and the other at 167 - well above the normal characters on a keyboard.

What I’m thinking, the tiddlers are undoubtably going to be created/edited in either TiddlyWiki or Node-RED. Both being web editors - is fairly easy to add formatting buttons like the [[]] button that TiddlyWiki has and will work the same. I’ll make a shortcut key - should work fairly smooth

So would look something like this - where a view template tiddler is changed based on the ‘mdUrl’ field:

title: $:/poc2go/tw5-node-red/redrum/toc/template/view

§¦#mdUrl¦§
<p>'§¦bookName¦§' has been exported as Markdown to '§¦filename¦§'.</p>
§¦/mdUrl¦§

§¦^mdUrl¦§
<$transclude mode="block" tiddler="$:/core/ui/ViewTemplate/body"/>
§¦/mdUrl¦§

I think would be fairly safe to say not going to be using those characters in WikiText any time soon.

Open to other ideas.

[jeremyruston]

@poc2go

I brought it up half a year ago hoping could find a sequence of two delimiting characters that are not used by
WikiText. Is a challenge.

Perhaps it would be worth us choosing and documenting an official “escape syntax” that will never be used by the core so that it’s guaranteed to be clear for extensions/addons to use.

We could use {% foobar %}, which was one of the earlier options we considered for the conditional shortcut syntax.

[TW_Tones]

@poc2go this looks like a revolutionary extension of tiddlywiki server configurations. Thanks very much for sharing this work, which no doubt has consumed many hours. I would like to adopt it enthuiasticaly.

If you can at some point give instructions for a “dumb user” to install and do some basic setup, it would be appreciated. Personaly I have intentionaly taken the role of a superuser of tiddlywiki, avoiding straying into Developer teritory, although I do occasionaly, and I am sure I can contribute on testing, documentation and feature and usage development,

• This maintining a "superuser only rol"e is I belive a useful to test tiddlywikis native capabilities because unlike many developers I don’t, or can’t revert to javascript code solutions when I find a gap in tiddlywiki, thus I tend to champion the filling of gaps in such a way a broader audience can make use of these. Did I ever tell you this @jeremyruston ?
• I can contribute a lot more but only once I have a working server/wiki platform in my hands and at least a rudimentaty understanding.
• The truth is once I get my hands on this I expect to drive it as far as I can.
  • There are substantial stratigic benifits to this model that could solve many problems we face making tiddlywiki solutions more broadly available.

Of course if we could safely put it on the internet it would be another revolution, however it has become clear now days that a lot of home users and remote workers use a VPN to connect into a trusted Network, and in that private network we have application servers, that is Intranet installs, behind a VPN can be used safely and effectivly. It sounds like this is easy with TW5-Node-RED.

• This would open up many of the organisation, team or business solutions if not yet public internet solutions.

Perhaps later I could elaborate on how we may make use of this in the real world.

[Scott_Sauyet]

@poc2go

The server is designed from the get go to be multi-user, so the short answer is yes.

Great!

@poc2go

I will be posting more on TW Talk about the design of the server with some teasers about what it can do and how it does it. Putting together a demo.

I’m very excited to see this!

@poc2go

The file store has options for buffering, retain in memory after write, throttling, etc. The default is physical write every 30 seconds which is fine for me.

That would be plenty often for me. If I lose a few minutes work, it’s not the end of the world. A few hours is scary, and a few days is terrifying! 30 seconds would be fine.

@poc2go

There is a twist - instead of having a separate file for each .tid - the servers file reader / writer takes a bunch of tiddlers and separates them with a line ‘+======+’ (exactly) between them.

Is this a whole wiki or some subset created by internal means that are either configurable or entirely irrelevant to the user? Also, what happens to binary tiddlers such as PNGs or PDFs? Do they simply get their own separate files as done in the Node version?

@poc2go

It might be a fit

It sounds very much like it could be, which is very exciting to me. I was not looking forward to trying many of the things I would need for this.

@poc2go

The main issue is probably Node-RED. Flows need to be designed and wired up, move tiddlers here, store it there, publish to multiple clients - do the task you need done.

I’m a programmer by profession. That part doesn’t scare me at all.

One more question: Is there authentication/authorization built in? I would love to publish wikis in read-only mode, but allow authorized users to access edit mode and be able to save changes.

I’m thrilled to hear this is coming, and anxious to see where you go with it.

[poc2go]

@TW_Tones

install and do some basic setup

I don’t have it out on GitHub yet, one of the reasons is have not decided on the license - Apache License 2.0. is being used by most of the packages. What I’ll do is zip it up and put on one of my servers. Can download, unzip, ‘cd tw5-node-red’, ‘npm install’, ‘npm start’ - go to ‘http://localhost:1880/uibtw/hello.html’. It has buttons to do things - open the server look at the flows. I have a writeup.

Thats it. Or it’s broke. Give me a few days and will post a link…

I don’t have a Windows system - died a few years back and never got around to get another one. So - has never been installed on Windows. Should work.

I got some documentation - markdown documents running on docsify. ‘http:/localhost:1880/docs’ - some old, some missing, some in TiddlyWiki. Will dig around and see what I can find.

@TW_Tones

if we could safely put it on the internet

It runs on Express server so about as good as can get. I put my servers on a hosted site, so if someone breaks into it and crashes them - their problem not mine. IBM uses it. Your right - set it up right and will be fine

@TW_Tones

a revolutionary extension of tiddlywiki server configurations

The server becomes an extension of your TiddlyWiki. I mostly use three tiddlywikis, ‘empty’, ‘WikiLabs’ by pmario and Tix by telmiger. When they load there is hardly anything in them. The server sends a tiddler to the story that has buttons to get stuff from the server. It doesn’t matter if I use ‘empty’, ‘WikiLabs’, or ‘Tix’ the server always sends that tiddler so I can press buttons to get stuff. Has 8 years of financial records - 300+ meg.

Server side it needs to know how to get stuff. That is done by drag-n-dropping nodes (they are like little black boxes with a wire going in, a wire coming out) that you can open up and change parameters. These boxes are wired together called a ‘flow’ to perform tasks.

So who is involved in making this stuff. To make a new node that does a task requires a JavaScript developer. Just like making a new filter operator in TiddlyWiki requires a JavaScript developer. So who wires the flows - an application developer, would be someone that knows how these black boxes interact with each other. In TiddlyWiki would be like someone that makes view or editor templates - got know how nested widgets <$let><$wikify><$view> interact with each other.

On the TiddlyWiki cient is a button ‘get subtitle’. This is a flow on the server. The first ‘green’ colored node is where messages come in from the client TiddlyWikis → the yellow ‘get subtitle’ node is a command → the node named $:/SiteSubtitle has a tiddler in it → the ‘blue?ish’ node sends the message back to the client TiddlyWiki. This flow changes the wiki subtitle. Double click the $:/SiteSubtitle node and would see a tiddler in it. Change the text, close it and save - press the button again and subtitle would change to new text.

Although trivial, is pretty simple to do. My question would be ‘How many tiddlers can I put in that node?’ As many as you want. Send stylesheets, macros, templates, datatiddlers, image tiddlers, HTML, WikiText, Markdown. And with just a few nodes. Haven’t even scratched the surface.

So your not dumb - now your an expert

[poc2go]

@Scott_Sauyet

Is this a whole wiki or some subset created by internal means

On startup TiddlyWiki has a boot function
it knows it is runnning on node.js (not the browser) so it handles some things differently
for example, $tw.Commander - the commandline processor (the browser can’t use that)

• loads utilities module - $tw.utils
• constructors for tiddler $tw.Tiddler and wiki $tw.Wiki
• creates a main wiki (active tiddler store) $tw.wiki = new $tw.Wiki
• into $tw.wiki loads JavaScript modules, system and shadow tiddlers, buncha stuff, all stuff

The active tiddler store is where all the TW JavaScript module code lives and executes.
Indirectly being used whenever a TiddlyWiki function is called.

I stay away from it. - but “Fools rush in where angels fear to tread” - Alexander Pope (1688-1744)
So, yeah, I do sometimes - to overload a shadow tiddler, or get tiddlers from it.

Here is a little bit of JS code - but just skim over it.

So to store a collection of tiddlers a $tw.Wiki instance needs to be created.
To keep the little bit of sanity I got left, lets call it a - ‘twiki’ - an empty tiny wiki

const twiki = new $tw.Wiki

Now an array of objects containing fields shows up

var fromSomewhere = [ { title: 'hello', text: 'surprise!' }, { title: 'Im lost', tag: 'help!' } ];

Could been read from a file, from another twiki, Node-Red flow, or a client TiddlyWiki (network)
who cares, we got put them somewhere before they get lost

twiki.addTiddlers(fromSomewhere);

Now need to put it where everybody can get to it

RED.global.set('Lost and Found', twiki);

‘RED’ is a global variable created before the express server is even created - seen by all

Meanwhile - Deep in the shadows of code forgotten a long time ago:

var hiLittleBuddy = RED.global.get('Lost and Found');

hiLittleBuddy.addTiddler({ title: 'got ya', text: 'milk and cookies!' });

Use TW filter to get all lost and found tiddlers

var lostAndFound = hiLittleBuddy.getTiddlersAsJson('[all[]]');
console.log(JSON.parse(lostAndFound));

Displays:

[
  { title: 'got ya', text: 'milk and cookies!' },
  { title: 'hello', text: 'surprise!' },
  { title: 'Im lost', tag: 'help!', text:'' }
]

---

---

Ut-oh, forgot all about no JavaScript @TW_Tones can’t do that! Would do it the Node-Red way

All of this could be done with two nodes, well the display, so three. The point is can get down to a pretty low level using nodes.

So is a full blown TiddlyWiki system.

[Scott_Sauyet]

@poc2go

@Scott_Sauyet

@poc2go

the servers file reader / writer takes a bunch of tiddlers and separates them with a line ‘+======+’ (exactly) between them.

Is this a whole wiki or some subset created by internal means …?

On startup TiddlyWiki has a boot function…

Thank you. I’ve been spending some time seeing how Node TW functions so parts of this are familiar.

I was asking a much more simple, specific question. When you have a file full of ‘+======+’-separated tiddlers, does it include all the tiddlers for a given wiki, some specific subset of them, or multiple wikis in one? Is this behavior configurable?

[poc2go]

@Scott_Sauyet

does it include all the tiddlers for a given wiki, some specific subset of them, or multiple wikis in one? Is this behavior configurable?

Depends on the filter you use when selecting from the given twiki.

lets say we have a given twiki ‘givenTwiki’ loaded up with image tiddlers. Now in givenTwiki the objects are not native JavaScript - they are all instances of $tw.Tiddler - they are their own animal.

The images are at URL `http://localhost:1880/my/images’ on the server.

So a bunch of png image tiddlers that look similar to this - different titles/_canonical_uri’s
you got 10 of them

title: bluesky
tags: images
type: image/png
_canonical_uri: /my/images/bluesky.png

and a bunch jpeg like this - with different titles and _canonical_uri’s
you got 5 of them

title: dirtydog
tags: images
type: image/jpeg
_canonical_uri: /my/images/dirtydog.jpeg

Get them out of the givenTwiki - use a TW filter - while convertng to native JS objects.
(the JSON parse part is habit - I always want a real copy - none of this implicit referencing stuff)

var all = JSON.parse(givenTwiki.getTiddlersAsJson(’[all[]]’));
var allAgain = JSON.parse(givenTwiki.getTiddlersAsJson(’[tag[images]]’));
var onlypngs = JSON.parse(givenTwiki.getTiddlersAsJson(’[type[images/png]]’));
var onlyjpegs = JSON.parse(givenTwiki.getTiddlersAsJson(’[type[images/jpeg]]’));
var bluesky = JSON.parse(givenTwiki.getTiddlersAsJson(’[[bluesky]]’));
var emptyArray = JSON.parse(givenTwiki.getTiddlersAsJson(’[type[text/plain]]’));

all - will have 15 tiddlers - if you write them to disk .tid or JSON will get 15 tiddlers in the file
same - with allagain 15
onlypngs - will have 10 - if you write them to disk .tid or JSON will get 10 tiddlers in the file
onlyjpeg - will have 5
bluesky - will have 1
and empty - will have none - will get JSON will be ‘[]’ 2 bytes empty array - .tid 0 bytes empty file

There is another twiki ‘aDifferentGivenTwiki’ with 22 regular tiddlers

var fromDiffTwiki = JSON.parse(aDifferentGivenTwiki.getTiddlersAsJson(’[all[]]’));

var combined = onlypngs.concat(fromDiffTwiki);

combined - will have 32 tiddlers, 10 png tiddlers plus the 22 regular tiddlers

If written .tid or JSON will get 32 tiddlers

Hopefully, answers your question?

[Scott_Sauyet]

@poc2go

Hopefully, answers your question?

Yes, thank you very much.

I can’t wait to see this!

[poc2go]

@jeremyruston

documenting an official “escape syntax” that will never be used by the core so that it’s guaranteed to be clear for extensions/addons to use.

We could use {% foobar %}

That would be a long term solution.

To describe a use case:

Polling data from RSS feeds of world news BBC, Guardian, CNN, etc

First pass

• News article data is substituted into same looking tiddlers
  • So now have 200 tiddler or so

Second pass

• Update the tag with the article’s parent tiddler title
• Update the parent tiddler list field with children tiddlers (articles)
  • need in date order - most current first

Probably guessed - being set up for the TOC macro

Then broadcast all tiddlers to TiddlyWiki clients

Mustache templates are doing the whole thing - every 15 minutes

Those tiddlers have WikiText so need delimiters to do my substition.

BTW - going commerical with it - gotta pay for the feeds

[poc2go]

Wanted to give an update on this project.

Expected to release TW5-Node-RED by year end '23 - was between contracts, holiday time, so figured could get finished. But - such a cherry job presented itself - had to take it - so TW5-Node-RED sent to the back seat … again.

That being said, the coding is now complete (tho needs a serious refactoring). Currently collecting and organizing the notes that I have accumulated over the years - one stick drive at a time. Some are out of date, some not existing, disorganized, and being worked on - but wanted to deliver a preview of TW5-Node-RED.

TW5-Node-Red — Node-Red Connected 01/13/2024 09:01 is an offline wiki of the notes that will be used to create docsify documentation (wait for it - takes a bit to load). Many of the links will not work as expected as the wiki needs to be connected to the TW5-Node-RED server to work properly. That being said - the content should give an idea of the system.

Comments about the documentation notes would be helpful - a tech writer; I am not

[TW_Tones]

FYI: SystemTag: $:/tags/RawMarkupWikified/TopHead can be used to put <script src="/svr/page/js/tw5-node-red.js"></script> in the document <head>

[poc2go]

@TW_Tones

SystemTag: $:/tags/RawMarkupWikified/TopHead

For TiddlyWiki wikis the $:/plugins/potofcoffee2go/tw5-node-red/network.js macro is all that is needed.

The ‘srv/page/js/tw5-node-red.js’ script is used for standard non-TiddlyWiki HTML pages to access tiddler content from the TW5-Node-RED server. Say a web page layout using Bootstrap.

In the the Bootstrap web page head would add the script

<script src="/svr/page/js/tw5-node-red.js"></script>

In the <body> somewhere would have something like

<div id="tag-line"></div>

allowing the div to be accessed using CSS selector ‘#tag-line’.

Then, usually at the end of the <body>, tell the server to execute the ‘home-page’ flow when the page loads

<script>
  nodered.onLoad = async () => {
    nodered.topic('home-page');
  }
</script>

In TW5-Node-RED Flow Editor create a flow ‘home-page’

with a tiddler that has the field ‘selector’ with value ‘#tag-line’ - this is the CSS selector the script will use to place the tiddler text into the web page. The text of the tiddler - <h2>This is our tag line!</h2>

When the page loads, the server ‘home-page’ flow is called and the above div will be populated with ‘This is our tag line!’ as a level 2 header.

---

Using the sync server can create a bunch of tiddlers on the TW5-Node-RED server. From the TW5-Node-RED project directory:

tiddlywiki ./public/app/home --init server
tiddlywiki ./public/app/home --listen

then go to http://localhost:8080

Create tiddlers that are the content of the the web page by providing a ‘selector’ field containing #id or .class of where in the Bootstrap layout the tiddler content is to go. To send those tiddlers to the web page, in the ‘to story’ (set-network) node use the From ‘Path’ property.

This allows people to change the content of the site using TiddlyWiki ‘server’ edition sync server. Note that changes take effect immediately on the site when a tiddler is saved - in this case since the tiddlers are only sent onLoad - when the web page is reloaded.

I have an example of this in the beta release - however mostly cover using TiddlyWiki wikis as clients as opposed to regular HTML pages as clients.

[pmario]

@poc2go

TW5-Node-Red — Node-Red Connected 01/13/2024 09:01 is an offline wiki of the notes that will be used to create docsify documentation (wait for it - takes a bit to load). Many of the links will not work as expected as the wiki needs to be connected to the TW5-Node-RED server to work properly. That being said - the content should give an idea of the system.

@poc2go … Sorry for my ignorance, but I cannot find any GH repo to install it. There is quite some documentation, which looks interesting, so I wanted to test it – even if it’s beta and there will be breaking changes.

You mentioned, that the code could be improved, but it works pretty stable for you. So it would be great to have something to “contribute” to the docs – From a node-red newbies point of view

I did read about 1/3 of the docs and it seems I do get a hang of the concept (open the internal links ;). It would be nice to play with the stuff, while reading it. I’ll pretty sure I’ll finish it soon. But without some stuff to play with, it’s hard to give feedback.

Well done – kind regards
-mario

[poc2go]

@pmario

I cannot find any GH repo to install it

I haven’t put it out on GH yet - putting it out there next weekend regardless if ready or not. Am obsessing on the neatness, accuracy, and completeness of the project - but is time for my baby go into the wild - warts and all.

Will post the GH link here when it is out there.

[pmario]

@poc2go

Am obsessing on the neatness, accuracy, and completeness of the project […]

Yea, that’s always a problem. The finishing touches (last 20%) can take a lot of time

@poc2go

Will post the GH link here when it is out there.

Looking forward to have a closer look.

You should probably create a completely new thread
-m