TiddlyWiki Integration / Automation

[nemo]

@jeremyruston

covering Bash

A little bit of quick and dirty testing, and this has worked for me from the linux commandline:

$ curl 'http://127.0.0.1:8080/recipes/default/tiddlers/A%20wild%20tid-from-bash' -X PUT -H 'Content-type: application/json' -H 'X-Requested-With: TiddlyWiki' -T "tid.json"

with tid.json looking like so (I put some extra custom fields in for demonstration purposes

{
  "title": "A wild testing tid",
  "color": "blue",
  "created": "20250715114555840",
  "modified": "20250715115147940",
  "my field name": "my field value",
  "tags": "scratch",
  "type": "text/vnd.tiddlywiki",
  "text": "Like magic, a tid arrives in your wiki from the ~~wilderness~~ land of shell",
  "revision": "0",
  "bag": "default"
}

Two things of note:

• The json file format is not quite compatible with the JSON obtained by export tiddler > JSON file menu option - the saved version has surrounding square brackets ( […] ). Remove those from the saved file, and then it’s compatible for upload this way.
• The title field in the JSON is ignored, with the one provided in the URL being respected, both for filename, and title value within the saved file (note the URL and title disagree in my example)

[Scott_Sauyet]

@jeremyruston

I find that pushing data into a Node.js TiddlyWiki instance via the HTTP API is the most convenient way to operate. There is no need to restart the server, and the changes will automatically synchronise to any connected clients.

That makes me wonder, though, if the node version could get an option to watch for file changes in its known directories, performing as it does with the HTTP API when a file changes. I mostly do my edits on my Node wiki tiddlers from within the UI. But if they’re JS-heavy, I sometimes switch to vscode. It would be useful to not have to remember to restart in these cases.

[saqimtiaz]

You can use the --watch-path argument in newer versions of node.js and set that up as a dev script: Command-line API | Node.js v24.4.0 Documentation

[Scott_Sauyet]

Yes, watching the files is clear enough. But that method would simply restart the Node server. I was looking for something a little more subtle. I’ve never used the TW API except as called by the UI. But I believe Jeremy was saying that when it is used, all connected users get updates without a need for a server restart. I was hoping to get the same behavior when Node sees a file changed. Of course, if this involves JS tiddlers, perhaps a reload would still be required, but I would hope the same notification that happens of load of JS tiddlers would still apply.

[jeremyruston]

@Scott_Sauyet

I was hoping to get the same behavior when Node sees a file changed

Node.js offers APIs for watching file directories, but they are full of caveats and platform specific oddities (see the docs for fs.watch). I had expected that these issues would be resolved over time, but that hasn’t happened.

One possibility might be for the server to watch what used to be called a “dropbox” directory, and now perhaps better called an “inbox” directory. Any tiddler files deposited there would be dynamically imported into the wiki. Perhaps the file would be deleted once it has been imported. Something simple like that would open the door to a bunch of useful scenarios, and the semantics should be a good deal simpler.

[stobot]

That would be VERY helpful for my integrations!

[Scott_Sauyet]

@jeremyruston

Node.js offers APIs for watching file directories, but they are full of caveats and platform specific oddities (see the docs for fs.watch). I had expected that these issues would be resolved over time, but that hasn’t happened.

Yes, I’ve seen that, but I’ve built several tools that successfully used watch. However, they were all single-platform (Linux), although I often developed and tested them on either Windows or Mac. I don’t know if I only had simple requirements that didn’t trigger any of these complexities, if the support teams simply chalked off any errors to unknown, non-repeatable glitches, or if I merely got lucky.

@jeremyruston

One possibility might be for the server to watch what used to be called a “dropbox” directory, and now perhaps better called an “inbox” directory. Any tiddler files deposited there would be dynamically imported into the wiki. Perhaps the file would be deleted once it has been imported.

That’s an interesting approach. I can see the advantages. But it wouldn’t help with the workflow I’ve developed, though: I do most of my editing inside the TW UI. When I have complex JS changes, where I really want a full-featured editor (brace matching, syntax highlighting, error-reporting, jump-to-declaration, etc.) I open the tiddler file in vscode, edit it there, restart the server and refresh the page. If file watchers worked as I would like, then I could skip the restart step, and I would (I hope) be notified in the UI of the need to reload.

But an inbox would definitely be useful.

[TW_Tones]

@stobot

@TW_Tones

This is why I wonder if you are best importing into your interactive wiki, or wonder why you are not because then the server will do all the work for you creating files etc…

@TW_Tones: I know this was directed to nemo, but I currently am doing everything through drag and drop and that’s what I’m trying to improve upon.

• See how powerful the interactive environment is, import and html get is also equivalant to drag and drop.

@nemo

• per-file based (for import ease, and also git history saving. Stretch goal was that it used git internally to handle history)
• relatively easy to setup / fun to use

Realising that the node implementation of TW covered the third point is the only reason I’m here. (noting that there were options that covered the first three points including stretch git goal, but failed the fourth. I figured from TW classic history that TW would still score very high on “fun to use”)

I just want to clarify that I have built large systems with per file/folder access both personal and professional and I understand where one would come from to desire this.

• However I am a convert to the logic, structure, meta data, automation and parsing available within an intereractive wiki, turn it into a tiddler and process it.

It is importiant to understand that logicaly we have the tiddler, wether it is an independant file or an “object” within the interactive tiddlywiki. Keep in mind in an interactive wiki;

• We have drag and drop
• Import file or files
• Http actions
• Javascript access to the wiki store eg via bookmarklets
• Command line access to actions against a wiki

Now as soon as data is imported be it a single or multiple tiddler we gain access to the wikis interactive environment, relavant examples include JSON Mangler, custom Parsers;

• It is here we can build simple or sophisticated tools to process incoming data, and if built on top of a node wiki logical tiddlers become independant files.

@nemo

Relevant background though: I’m a linux admin and the old unix philosophy that “everything is a file” is pretty core to how I approach many IT problems, and when I went looking (about 18 months ago) for a better note taking system, my basic essential criteria was

• Consider a tiddler a logical file with metadata, and on node it is also a physical file.

I will also add with TiddlyWiki its trivial for you to use a customised instance of TiddlyWiki to import and process data, to create native tiddler formats to be added to the wiki using that data.

• If the data import process is once off or upfront task, a custom wiki allows you document the process, test and review and clone into multiple batch imports allows others to see how you did it.

[nemo]

@TW_Tones

@stobot

@TW_Tones: I know this was directed to nemo, but I currently am doing everything through drag and drop and that’s what I’m trying to improve upon.

• See how powerful the interactive environment is, import and html get is also equivalant to drag and drop.

You describe it as powerful, but I interpreted stobot’s desire for an improvement as an indication that for this use case, drag and drop is tedious.

@TW_Tones

Consider a tiddler a logical file with metadata, and on node it is also a physical file.

Exactly. And physical files make it super easy for me to bulk import them via file creation outside the TW software itself.

I’m performing one-off conversion of files that are in a range of different formats, into either TW wikitext, or md+meta, or txt+meta. Doing that outside TW for a once-off import is relatively trivial. Doing it inside TW for a once-off import sounds like the worst thing to me, or at least, for me and my current skills. (if that’s not what you’re advocating, then I’m lost as to what you are advocating).

@TW_Tones

If the data import process is once off or upfront task, a custom wiki allows you document the process, test and review and clone into multiple batch imports allows others to see how you did it.

I plan on writing up my process and sharing my code. It’s just linux shell code, not wikitext code.

[jacng]

nemo offers a very interesting approach here using command line CURL to add a tiddler to TW in node.js

In the context of “Tiddlywiki Integration / Automation” with the external environment that was discussed in another thread, I wonder how feasible / easy is it to do the reverse : Send “tm-http-request” from TW to a helper app in the external environment to perform some actions and return a response to TW ? I imagine the helper app to be some sort of minimal web server (Python server ?), preferably available on multiple OS, or maybe even command line tool that can receive the http-request, invoke a local script to perform the action, and return a response ? Has anybody done it already ?

I presume TW on node.js already has some native mechanism for TW client to talk to the node.js server to access the local environment. Even so, a universal approach to access the external environment using tm-http-request that can potential work for all variant of TWs would still be useful ?

[saqimtiaz]

@nemo

A little bit of quick and dirty testing, and this has worked for me from the linux commandline:

I have been using a somewhat similar setup in one of my wikis for quite a few years now and it has worked pretty well.

[stobot]

@jeremyruston

It is such a useful and common pattern that it would be helpful to have some examples in the docs, perhaps covering Bash and JavaScript. As to R, there appears to be libraries call httr and httr2 that can make outgoing HTTP requests. I got ChatGPT to write some sample R code but I’m unable to verify that it works.

Just to follow-up, eventually got this to work - thanks everyone!

# Readme: This sends tiddler content into a running node.js wiki

# Libraries
library(httr)
library(jsonlite)

# Host details (edit as needed)
tiddlywiki_host = 'localhost:8080'

# Tiddler title (edit as needed)
tiddler_title = 'Tiddler Title'
encoded_title = URLencode(tiddler_title)
api_url = paste0(tiddlywiki_host,'/recipes/default/tiddlers/',encoded_title)

# Tiddler content (edit as needed)
tiddler = list(
  title = tiddler_title,
  text = 'This is content for the text field',
  created = format(now(),format='%Y%m%d%H%M%S00000'),
  modified = format(now(),format='%Y%m%d%H%M%S00000')
)
json_body = toJSON(tiddler,auto_unbox=TRUE)

# Send the content to the wiki (leave this)
response = PUT(
  url = api_url,
  add_headers('Content-Type' = 'application/json','x-requested-with' = 'TiddlyWiki'),
  body = json_body
)

# Response (leave this)
if (status_code(response) >= 200 && status_code(response) < 300) {
  print("Tiddler imported successfully!")
} else {
  print("Error importing tiddler:")
  print(content(response, "text"))
}

[TW_Tones]

@nemo

You describe it as powerful, but I interpreted stobot’s desire for an improvement as an indication that for this use case, drag and drop is tedious.

Although I did not state it specificaly if drag and drop is tedious you can take steps to reduce this;

• Drag and drop multiple files at a time
• Drag and drop a single monolything file and then split it within tiddlywiki
• Import or get one or MORE files at a time (use multiple on browse widget)
• Sometimes when you have tiddlywiki in one tab and another, or a website with the source data, drag and drop is simpler because you never have to conciously interact with the file system or command line.

@nemo

TW for a once-off import sounds like the worst thing to me, or at least, for me and my current skills.

I took intergration and Automation as meaning reoccuring, not once off.

• For me I have a custom wiki I use for this, so although the actual activity may be once off the methods and tools are not. So using a wiki with JSON Mangler for example to do the import and processing I quickly import and manage and package the data and drop this on the target wiki.
• Yes, we do need expierence with these tools, but if you are going todo many one off things, or lots of steps for something once off it is worth the investment.

More than a decade ago I decided TiddlyWiki was my platform of choice so I have invested in building the tools I need for most activities I do. This results in ongoing and multiplying advantages to myself.

Along with JSON Mangler consider also TiddlyXML — XML features for Tiddlywiki

[stobot]

While the R code / HTTP API solution will save me some work in that maybe I can tee up a lot of the importing I’m doing into daily scripts, that’s more “batch” and not “automation”.

I’m still trying to solve for automation in where triggers that are not done by me at all get information into my wiki in near real time. I use plenty of automation outside of TiddlyWiki today linking mostly all parts of the Microsoft stack. PowerAutomate (formerly Flow) is great for this and I have flows like employees requesting vacation time in a SharePoint list getting approved and added into various Outlook Calendars. I have people updating status reports in Word that get aggregated into overall status reports with notifications in Outlook and Teams.

The only way I can think to extend PowerAutomate to do the same input/output in TiddlyWiki is via file creation. Because all of my Project Management is in TiddlyWiki, there’s a lot of back and forth with Outlook, Teams, Loop, SharePoint, Excel, Access, etc. PowerAutomate can already automatically trigger based on file creation, I’m just missing the last piece of the puzzle.

It doesn’t have to be PowerAutomate, for example if there was the functionality that Jeremy described where a file created into an “Import Folder” could get auto-imported into TiddlyWiki, that’d solve it. I can already do the reverse and have a file created by TiddlyWiki kick off other automations.

I hope that adds clarity, this has been a very interesting thread!

[TW_Tones]

Quite involved then and thus a lot of details

Like you I played with similar uses of tiddlywiki. Using a wiki with local access rights, I have manually triggered external batch files to generate files which are already visible from within TiddlyWiki and whilst I could not interrogate this content to do further processing (I think the HTML get may allow this) it was very helpful.

• My key usecase was to run a sophisticated LAN access test using ping to identify if variouse devices making up the local network and internet were reachable and visible in DNS. Obviuously I only triggered this when I needed to, so it was batch on demand.

There are a few non-standard triggers out there that can occur on a timer etc… and if an external batch is repeated and retains the results sometimes the startup actions can trigger the ingestion process.

• I suppose it depends on the circumstances but ingestion only when you look is a valid model.
• There are tricks but browsers are about browsing so they assume interaction, so this needs to be taken into account, such as if the browser makes tabs go to sleep etc…

It seems there are so many ways and needs in automation, that can influence when processing needs to take place. TiddlyWiki fits nicely into the visulisation and interaction space, or on demand dash board.

• I did a lot of this kind of stuff in my career from downloading and uploading to / from Aircraft when they arrive at the airport to logging load and passenger records, and much more.
• I found the greater challenges when a lot of analysis was needed. I think tiddlywiki is good for this part of automation, but we can still leverage external tools, rather than trying to jam too much into tiddlywiki.

[saqimtiaz]

@stobot

The only way I can think to extend PowerAutomate to do the same input/output in TiddlyWiki is via file creation

The way I went about this was to watch for the creation/modification of files and using that as the trigger for a http request to the TW API. I used fswatch (node.js) to detect file system changes but you might want to have a look to see if you can find an equivalent solution that works with your stack.

[pmario]

@jeremyruston

One possibility might be for the server to watch what used to be called a “dropbox” directory, and now perhaps better called an “inbox” directory. Any tiddler files deposited there would be dynamically imported into the wiki. Perhaps the file would be deleted once it has been imported. Something simple like that would open the door to a bunch of useful scenarios, and the semantics should be a good deal simpler.

I would create a workflow like this.

• The server constantly watches for if a single file exists eg: load.inbox
• If the file exists, that’s the trigger to load all files from the inbox in one go
• using $tw.loadTiddlersFromPath(), which gives us the same functionality as the TW –load command.
• So we can import several files in “mixed” formats: .html, .tiddler, .tid, .json and even tiddlywiki.files
• When a file is imported 2 things can happen:
  1. Log some file info into a log-file → Delete file
  2. Log file info, Move file to an “archive” directory instead of deleting the file
• When everything is imported → Delete the load.inbox file

With this workflow it’s easy to test and prepare the files in the inbox manually.
Also an external batch file can check for the load.inbox file. As long as it exists, no new files should be moved to the inbox, or an error can be raised for manual intervention.

Just some thoughts.

[stobot]

Thanks for the reminder, I’m testing the watch-fs now and it somewhat works in TiddlyWiki App

What I’m testing:
Watch FS — Modern TiddlyWiki plugin developing framework
Releases · tiddly-gittly/watch-fs

With this installed

1. I can edit tiddlers with an external application (such as notepad), save, and wiki will update automatically.
2. I can copy the .tid out of the folder, edit it, and copy it back in and overwrite it and things also go well.
3. When I copy a .tid that wasn’t already in the wiki, it behaves strangely.
   a. It automatically adds a second file with a _1 suffix
   b. It shows in the wiki, but further in-wiki editing only changes the _1 file strangely, even though the _1 is not shown anywhere in the wiki.
4. When I delete some tiddlers in the folder (not in the wiki), I get a Javascript error at sync.

So, while this is a bit finicky, if I can continue to use watch-fs safely within TiddlyWiki App and use my automation to only replace existing tiddlers, then I can have automated refreshes which is what I’m going for.

If anyone who knows more sees a problem with this methodology, please let me know!

[nemo]

@saqimtiaz

somewhat similar setup

That’s a good writeup, thankyou! I’ve not played with nodemon before, but will be keeping it in mind, and watch-fs looks even more useful.

@TW_Tones

I took intergration and Automation as meaning reoccuring, not once off.

I think we’ve been talking a little cross purposes here. I was advocating for node TW to see and pickup changes to the filesystem (and refuting claim that would necessarily be expensive) as useful, and my thinking at the time was primarily my own usecase so far, which is one-off imports.

In general, I think I’d categorise “automation” as being reoccurring, but “integration” could be either reccuring, or once-off/adhoc.

@TW_Tones

Yes, we do need expierence with these tools, but if you are going todo many one off things, or lots of steps for something once off it is worth the investment.

I doubt I’ll ever be importing from tomboy notes ever again. Or from tw classic files. Or from cctiddly, etc. More relevant for me (and only mentioned above as an aside), it’s important to me that I not only have the data in TW, but in the git repo that is storing the history of the data. That is, I’m importing the history into git. The final version is also visible to TW. But afaik, TW doesn’t manage local git repos, certainly not to the esoteric git usage I do. So even if I do end up running similar imports of similar data, I would do the same. If the history-to-git wasn’t a consideration, even then I still think “convert to (relatively simple) .tid format and write to a file” is simpler than “convert to valid json format, and send via API call”.

However circling back to automation of regular tasks, I’ve been thinking of some of the things I’ve got automated to other adhoc solutions and what could be put into TW instead, and I think the watch-fs will suit that well (though I also need to delve into the plotting thread: Does Tiddlywiki have a preferred Plotting/Graphics js tool? - my current adhoc setup is writing data in csv format and visualising it using dygraph, but updating the CSV within TW, having TW pickup the new data, and then visualise that within TW I think will be a much nicer workflow)

@stobot

If anyone who knows more sees a problem with this methodology, please let me know!

Those cases where it works well look ideal! Thank you for testing. Natve watch-fs for live edits to existing files, and nodeTW API for adding / deleting, will make for the ideal mix that never requires node to be restarted?

[TW_Tones]

@pmario

I would create a workflow like this.

I think this workflow sounds good however as is often the case it is common to leave out a way to submit a delete or merge (tiddler) through the inbox. Such methods should be implemented from the begining rather than added later.

Similarly the idea that something more complex can be submitted such that once injested into the tiddlywiki it may trigger actions or renders inside the interactive wiki is well worth thinking about when architecting such a solution.

• I know that you @pmario are security aware, so allowing a key to be provided that must match one in the wiki etc… can be considered, however such automation is usualy taking place in a trusted environment.

[Edited]

See Proposal: WikiShell / TiddlyCLI - A Command Console for TiddlyWiki - #38 by TW_Tones and my references to using bookmarklets. This introduces another mechanisium for Intergration and Automation by triggering JS Functions.