Node-Red Based Low-Code Server to Dynamically Create TiddlyWiki's

poc2go · October 31, 2023, 8:15pm

On startup TiddlyWiki has a boot function
it knows it is runnning on node.js (not the browser) so it handles some things differently
for example, $tw.Commander - the commandline processor (the browser can’t use that)

loads utilities module - $tw.utils
constructors for tiddler $tw.Tiddler and wiki $tw.Wiki
creates a main wiki (active tiddler store) $tw.wiki = new $tw.Wiki
into $tw.wiki loads JavaScript modules, system and shadow tiddlers, buncha stuff, all stuff

The active tiddler store is where all the TW JavaScript module code lives and executes.
Indirectly being used whenever a TiddlyWiki function is called.

I stay away from it. - but “Fools rush in where angels fear to tread” - Alexander Pope (1688-1744)
So, yeah, I do sometimes - to overload a shadow tiddler, or get tiddlers from it.

Here is a little bit of JS code - but just skim over it.

So to store a collection of tiddlers a $tw.Wiki instance needs to be created.
To keep the little bit of sanity I got left, lets call it a - ‘twiki’ - an empty tiny wiki

const twiki = new $tw.Wiki

Now an array of objects containing fields shows up

var fromSomewhere = [ { title: 'hello', text: 'surprise!' }, { title: 'Im lost', tag: 'help!' } ];

Could been read from a file, from another twiki, Node-Red flow, or a client TiddlyWiki (network)
who cares, we got put them somewhere before they get lost

twiki.addTiddlers(fromSomewhere);

Now need to put it where everybody can get to it

RED.global.set('Lost and Found', twiki);

‘RED’ is a global variable created before the express server is even created - seen by all

Meanwhile - Deep in the shadows of code forgotten a long time ago:

var hiLittleBuddy = RED.global.get('Lost and Found');

hiLittleBuddy.addTiddler({ title: 'got ya', text: 'milk and cookies!' });

Use TW filter to get all lost and found tiddlers

var lostAndFound = hiLittleBuddy.getTiddlersAsJson('[all[]]');
console.log(JSON.parse(lostAndFound));

Displays:

[
  { title: 'got ya', text: 'milk and cookies!' },
  { title: 'hello', text: 'surprise!' },
  { title: 'Im lost', tag: 'help!', text:'' }
]

Ut-oh, forgot all about no JavaScript @TW_Tones can’t do that! Would do it the Node-Red way

All of this could be done with two nodes, well the display, so three. The point is can get down to a pretty low level using nodes.

So is a full blown TiddlyWiki system.

Scott_Sauyet · October 31, 2023, 11:11pm

Thank you. I’ve been spending some time seeing how Node TW functions so parts of this are familiar.

I was asking a much more simple, specific question. When you have a file full of ‘+======+’-separated tiddlers, does it include all the tiddlers for a given wiki, some specific subset of them, or multiple wikis in one? Is this behavior configurable?

poc2go · November 1, 2023, 1:00am

Depends on the filter you use when selecting from the given twiki.

lets say we have a given twiki ‘givenTwiki’ loaded up with image tiddlers. Now in givenTwiki the objects are not native JavaScript - they are all instances of $tw.Tiddler - they are their own animal.

The images are at URL `http://localhost:1880/my/images’ on the server.

So a bunch of png image tiddlers that look similar to this - different titles/_canonical_uri’s
you got 10 of them

title: bluesky
tags: images
type: image/png
_canonical_uri: /my/images/bluesky.png

and a bunch jpeg like this - with different titles and _canonical_uri’s
you got 5 of them

title: dirtydog
tags: images
type: image/jpeg
_canonical_uri: /my/images/dirtydog.jpeg

Get them out of the givenTwiki - use a TW filter - while convertng to native JS objects.
(the JSON parse part is habit - I always want a real copy - none of this implicit referencing stuff)

var all = JSON.parse(givenTwiki.getTiddlersAsJson(’[all[]]’));
var allAgain = JSON.parse(givenTwiki.getTiddlersAsJson(’[tag[images]]’));
var onlypngs = JSON.parse(givenTwiki.getTiddlersAsJson(’[type[images/png]]’));
var onlyjpegs = JSON.parse(givenTwiki.getTiddlersAsJson(’[type[images/jpeg]]’));
var bluesky = JSON.parse(givenTwiki.getTiddlersAsJson(’[[bluesky]]’));
var emptyArray = JSON.parse(givenTwiki.getTiddlersAsJson(’[type[text/plain]]’));

all - will have 15 tiddlers - if you write them to disk .tid or JSON will get 15 tiddlers in the file
same - with allagain 15
onlypngs - will have 10 - if you write them to disk .tid or JSON will get 10 tiddlers in the file
onlyjpeg - will have 5
bluesky - will have 1
and empty - will have none - will get JSON will be ‘[]’ 2 bytes empty array - .tid 0 bytes empty file

There is another twiki ‘aDifferentGivenTwiki’ with 22 regular tiddlers

var fromDiffTwiki = JSON.parse(aDifferentGivenTwiki.getTiddlersAsJson(’[all[]]’));

var combined = onlypngs.concat(fromDiffTwiki);

combined - will have 32 tiddlers, 10 png tiddlers plus the 22 regular tiddlers

If written .tid or JSON will get 32 tiddlers

Hopefully, answers your question?

Scott_Sauyet · November 1, 2023, 1:24am

Yes, thank you very much.

I can’t wait to see this!

poc2go · November 1, 2023, 12:16pm

That would be a long term solution.

To describe a use case:

Polling data from RSS feeds of world news BBC, Guardian, CNN, etc

First pass

News article data is substituted into same looking tiddlers
- So now have 200 tiddler or so

Second pass

Update the tag with the article’s parent tiddler title
Update the parent tiddler list field with children tiddlers (articles)
- need in date order - most current first

Probably guessed - being set up for the TOC macro

Then broadcast all tiddlers to TiddlyWiki clients

Mustache templates are doing the whole thing - every 15 minutes

Those tiddlers have WikiText so need delimiters to do my substition.

BTW - going commerical with it - gotta pay for the feeds

poc2go · January 13, 2024, 3:18pm

Wanted to give an update on this project.

Expected to release TW5-Node-RED by year end '23 - was between contracts, holiday time, so figured could get finished. But - such a cherry job presented itself - had to take it - so TW5-Node-RED sent to the back seat … again.

That being said, the coding is now complete (tho needs a serious refactoring). Currently collecting and organizing the notes that I have accumulated over the years - one stick drive at a time. Some are out of date, some not existing, disorganized, and being worked on - but wanted to deliver a preview of TW5-Node-RED.

TW5-Node-Red — Node-Red Connected 01/13/2024 09:01 is an offline wiki of the notes that will be used to create docsify documentation (wait for it - takes a bit to load). Many of the links will not work as expected as the wiki needs to be connected to the TW5-Node-RED server to work properly. That being said - the content should give an idea of the system.

Comments about the documentation notes would be helpful - a tech writer; I am not

TW_Tones · January 17, 2024, 10:57am

FYI: SystemTag: $:/tags/RawMarkupWikified/TopHead can be used to put <script src="/svr/page/js/tw5-node-red.js"></script> in the document <head>

poc2go · January 21, 2024, 5:29am

For TiddlyWiki wikis the $:/plugins/potofcoffee2go/tw5-node-red/network.js macro is all that is needed.

The ‘srv/page/js/tw5-node-red.js’ script is used for standard non-TiddlyWiki HTML pages to access tiddler content from the TW5-Node-RED server. Say a web page layout using Bootstrap.

In the the Bootstrap web page head would add the script

<script src="/svr/page/js/tw5-node-red.js"></script>

In the <body> somewhere would have something like

<div id="tag-line"></div>

allowing the div to be accessed using CSS selector ‘#tag-line’.

Then, usually at the end of the <body>, tell the server to execute the ‘home-page’ flow when the page loads

<script>
  nodered.onLoad = async () => {
    nodered.topic('home-page');
  }
</script>

In TW5-Node-RED Flow Editor create a flow ‘home-page’

with a tiddler that has the field ‘selector’ with value ‘#tag-line’ - this is the CSS selector the script will use to place the tiddler text into the web page. The text of the tiddler - <h2>This is our tag line!</h2>
home-page-editor

When the page loads, the server ‘home-page’ flow is called and the above div will be populated with ‘This is our tag line!’ as a level 2 header.

Using the sync server can create a bunch of tiddlers on the TW5-Node-RED server. From the TW5-Node-RED project directory:

tiddlywiki ./public/app/home --init server
tiddlywiki ./public/app/home --listen

then go to http://localhost:8080

Create tiddlers that are the content of the the web page by providing a ‘selector’ field containing #id or .class of where in the Bootstrap layout the tiddler content is to go. To send those tiddlers to the web page, in the ‘to story’ (set-network) node use the From ‘Path’ property.

This allows people to change the content of the site using TiddlyWiki ‘server’ edition sync server. Note that changes take effect immediately on the site when a tiddler is saved - in this case since the tiddlers are only sent onLoad - when the web page is reloaded.

I have an example of this in the beta release - however mostly cover using TiddlyWiki wikis as clients as opposed to regular HTML pages as clients.

pmario · January 21, 2024, 10:05am

@poc2go … Sorry for my ignorance, but I cannot find any GH repo to install it. There is quite some documentation, which looks interesting, so I wanted to test it – even if it’s beta and there will be breaking changes.

You mentioned, that the code could be improved, but it works pretty stable for you. So it would be great to have something to “contribute” to the docs – From a node-red newbies point of view

I did read about 1/3 of the docs and it seems I do get a hang of the concept (open the internal links ;). It would be nice to play with the stuff, while reading it. I’ll pretty sure I’ll finish it soon. But without some stuff to play with, it’s hard to give feedback.

Well done – kind regards
-mario

poc2go · January 21, 2024, 10:27am

I haven’t put it out on GH yet - putting it out there next weekend regardless if ready or not. Am obsessing on the neatness, accuracy, and completeness of the project - but is time for my baby go into the wild - warts and all.

Will post the GH link here when it is out there.

pmario · January 21, 2024, 10:41am

Yea, that’s always a problem. The finishing touches (last 20%) can take a lot of time

Looking forward to have a closer look.

You should probably create a completely new thread
-m

pmario · January 21, 2024, 11:04am

It would be nice, if you do set the themes “sidebarlayout” setting to fluid-fixed as it is at tw-com.

poc2go · January 21, 2024, 1:15pm

Done - also set the animation back to 400ms

TW_Tones · January 22, 2024, 12:22am

The fist 80% takes 80% of the time, the last 20% also takes 80% of the time. Should we be surprised if things are at least 60% longer than we expect?

poc2go · January 27, 2024, 6:22am

The demo site secured and running at TW5-Node-RED demo https://poc2go.ngrok.io/svr/wiki/nodered.html?app=welcome so unlike the previous offline version the buttons should work and can now interact with the server.

Use the flow editor by pressing the ‘Flow Editor’ button which opens the editor in a tiddler. Or go to TW5-Node-RED https://poc2go.ngrok.io/red to open in a browser tab. Obviously there are limitations such as pressing ‘inject’ nodes is blocked and can not allow any changes you make to be deployed - but can get a feel for how the editor works - open nodes to see the properties and such.

May be a little laggy depending on the load. Set up is on a shoe-string budget - running on a Raspberry Pi. Hope is usable - let me know if the response times are acceptable.

Plan on having the project on GitHub within the next couple days. Will keep informed.

pmario · January 27, 2024, 3:44pm

Very nice!

I’m looking forward to visit the GitHub repo and start a server myself.

As a little present, I did create a new Twilight-Node-Red palette (based on your changes), that should fix the “mouse hover problem” in your tabbed-TOC

I also tried to fix some other problems. As a side-effect the palette looks a bit different than your settings, but I think the whole thing is more consistent now. I did create the new palette with my Palette-Manager. The previews clearly show, where the problems are. (Sadly, some of them are in core vanilla base )

have fun!
mario

$ _palettes_Twilight-Node-Red.json (4.8 KB)

poc2go · January 27, 2024, 6:35pm

Is a definite improvement to the TOC and is more consistent.

I forgot all about Palette-Manager. There are just to many TiddlyWiki toys to keep track of !

Thanks for the improvements!

poc2go · February 3, 2024, 10:58am

TW5-Node-Red is designed to face the web - so security is a concern.

JWT token

The first line of defense is a JWT bearer token which is generated by the server upon client connection and sent to the the TiddlyWiki client. The token is server signed (HMAC-SHA256) and saved in a client side tiddler which is transmitted for all server requests - (with the exception of application level heartbeats which contain no useful information). The token is not encrypted for performance reasons and to allow client side to easily verify the the server signature if so desired.

Most vulnerabilities of JWT bearer tokens revolve around its storage. If in a cookie is vulnerable to a whole variety of attacks - if in session or local storage is vulnerable to cross-site scripting (XSS) attacks. To circumvent these attacks the token is stored client side in a tiddler which is sent with every request. The server uses that tiddler to verify each message.

There is a unique token generated for each connection. Thus tokens do not expire as are only useful for that specific connection (browser tab). This does require the user to sign in when establishing a connection to the server.

User authentication and authorization

All users are required to create an account with the server. The account contains a unique ‘User ID’ , ‘Password’, ‘User name’ (shown to other users on the server) and optional ‘Email’ address which is stored server-side in a TW database. The ‘authentication’ node can be modified to perform any action required by the application developer - (the default is to accept any request for an account).

The password is encryted using the bcrypt hash() function prior to storage in the TW ‘Users’ database thus the actual password typed by the user is not known - anywhere. When regestering a new account the password is displayed in the account created confirmation tiddler to the user - once - is last time the actual ‘typed’ password is known.

When a user logs in, the database is queried which contains the password hash - bcrypt compared() the typed password with the stored hash - and verified. An in memory flag is set indicating the user is authorized.

Application developer

All aspects of the Security System can be modified via Node-RED.

telumire · March 8, 2025, 3:22pm

Hi @poc2go, I’m very interested in your project. Up until recently, I was using my RP5 to run TiddlyWiki PWA (link), but now I plan to use Home Assistant OS. I haven’t found a way to run a Deno or Node.js server on HA, but since there’s a Node-RED add-on available, I was wondering if it’s possible to use it to run a TiddlyWiki server. Is your package public?

poc2go · March 12, 2025, 12:24pm

Hi @telumire

The TW5-Node-RED project topic has the directions to install a beta version.

After installing go to http://localhost:1880/svr/wiki to see draft docs. Will ask for User: demo Password: demo. The Node-RED Flow Editor uses the Node-RED default User: admin Password: password.

However - the beta is a Proof-Of-Concept which is loaded with test stuff, things that work, things that don’t. The dependencies have not been updated so has a lot of vulnerabilities - is running the old v3 release of Node-RED, and uses a modified version of TiddlyWiki v5.3.5. (with a very early implementation of MWS ).

But it proved the Node-RED / TiddlyWiki interface is viable and the Node-RED TiddlyWiki comm and nodes are complete. The beta also highlighted areas that needed to be addressed for a clean (and less complicated) implementation.

To address issues discovered during the beta; am actively working on another repo - tw5-pocket-io which contains components that will be merged with the Node-RED components from the beta. Expect around June or July should have practicable system.