TiddlyWiki Integration / Automation

[jacng]

nemo offers a very interesting approach here using command line CURL to add a tiddler to TW in node.js

In the context of “Tiddlywiki Integration / Automation” with the external environment that was discussed in another thread, I wonder how feasible / easy is it to do the reverse : Send “tm-http-request” from TW to a helper app in the external environment to perform some actions and return a response to TW ? I imagine the helper app to be some sort of minimal web server (Python server ?), preferably available on multiple OS, or maybe even command line tool that can receive the http-request, invoke a local script to perform the action, and return a response ? Has anybody done it already ?

I presume TW on node.js already has some native mechanism for TW client to talk to the node.js server to access the local environment. Even so, a universal approach to access the external environment using tm-http-request that can potential work for all variant of TWs would still be useful ?

[saqimtiaz]

@nemo

A little bit of quick and dirty testing, and this has worked for me from the linux commandline:

I have been using a somewhat similar setup in one of my wikis for quite a few years now and it has worked pretty well.

[stobot]

@jeremyruston

It is such a useful and common pattern that it would be helpful to have some examples in the docs, perhaps covering Bash and JavaScript. As to R, there appears to be libraries call httr and httr2 that can make outgoing HTTP requests. I got ChatGPT to write some sample R code but I’m unable to verify that it works.

Just to follow-up, eventually got this to work - thanks everyone!

# Readme: This sends tiddler content into a running node.js wiki

# Libraries
library(httr)
library(jsonlite)

# Host details (edit as needed)
tiddlywiki_host = 'localhost:8080'

# Tiddler title (edit as needed)
tiddler_title = 'Tiddler Title'
encoded_title = URLencode(tiddler_title)
api_url = paste0(tiddlywiki_host,'/recipes/default/tiddlers/',encoded_title)

# Tiddler content (edit as needed)
tiddler = list(
  title = tiddler_title,
  text = 'This is content for the text field',
  created = format(now(),format='%Y%m%d%H%M%S00000'),
  modified = format(now(),format='%Y%m%d%H%M%S00000')
)
json_body = toJSON(tiddler,auto_unbox=TRUE)

# Send the content to the wiki (leave this)
response = PUT(
  url = api_url,
  add_headers('Content-Type' = 'application/json','x-requested-with' = 'TiddlyWiki'),
  body = json_body
)

# Response (leave this)
if (status_code(response) >= 200 && status_code(response) < 300) {
  print("Tiddler imported successfully!")
} else {
  print("Error importing tiddler:")
  print(content(response, "text"))
}

[TW_Tones]

@nemo

You describe it as powerful, but I interpreted stobot’s desire for an improvement as an indication that for this use case, drag and drop is tedious.

Although I did not state it specificaly if drag and drop is tedious you can take steps to reduce this;

• Drag and drop multiple files at a time
• Drag and drop a single monolything file and then split it within tiddlywiki
• Import or get one or MORE files at a time (use multiple on browse widget)
• Sometimes when you have tiddlywiki in one tab and another, or a website with the source data, drag and drop is simpler because you never have to conciously interact with the file system or command line.

@nemo

TW for a once-off import sounds like the worst thing to me, or at least, for me and my current skills.

I took intergration and Automation as meaning reoccuring, not once off.

• For me I have a custom wiki I use for this, so although the actual activity may be once off the methods and tools are not. So using a wiki with JSON Mangler for example to do the import and processing I quickly import and manage and package the data and drop this on the target wiki.
• Yes, we do need expierence with these tools, but if you are going todo many one off things, or lots of steps for something once off it is worth the investment.

More than a decade ago I decided TiddlyWiki was my platform of choice so I have invested in building the tools I need for most activities I do. This results in ongoing and multiplying advantages to myself.

Along with JSON Mangler consider also TiddlyXML — XML features for Tiddlywiki

[stobot]

While the R code / HTTP API solution will save me some work in that maybe I can tee up a lot of the importing I’m doing into daily scripts, that’s more “batch” and not “automation”.

I’m still trying to solve for automation in where triggers that are not done by me at all get information into my wiki in near real time. I use plenty of automation outside of TiddlyWiki today linking mostly all parts of the Microsoft stack. PowerAutomate (formerly Flow) is great for this and I have flows like employees requesting vacation time in a SharePoint list getting approved and added into various Outlook Calendars. I have people updating status reports in Word that get aggregated into overall status reports with notifications in Outlook and Teams.

The only way I can think to extend PowerAutomate to do the same input/output in TiddlyWiki is via file creation. Because all of my Project Management is in TiddlyWiki, there’s a lot of back and forth with Outlook, Teams, Loop, SharePoint, Excel, Access, etc. PowerAutomate can already automatically trigger based on file creation, I’m just missing the last piece of the puzzle.

It doesn’t have to be PowerAutomate, for example if there was the functionality that Jeremy described where a file created into an “Import Folder” could get auto-imported into TiddlyWiki, that’d solve it. I can already do the reverse and have a file created by TiddlyWiki kick off other automations.

I hope that adds clarity, this has been a very interesting thread!

[TW_Tones]

Quite involved then and thus a lot of details

Like you I played with similar uses of tiddlywiki. Using a wiki with local access rights, I have manually triggered external batch files to generate files which are already visible from within TiddlyWiki and whilst I could not interrogate this content to do further processing (I think the HTML get may allow this) it was very helpful.

• My key usecase was to run a sophisticated LAN access test using ping to identify if variouse devices making up the local network and internet were reachable and visible in DNS. Obviuously I only triggered this when I needed to, so it was batch on demand.

There are a few non-standard triggers out there that can occur on a timer etc… and if an external batch is repeated and retains the results sometimes the startup actions can trigger the ingestion process.

• I suppose it depends on the circumstances but ingestion only when you look is a valid model.
• There are tricks but browsers are about browsing so they assume interaction, so this needs to be taken into account, such as if the browser makes tabs go to sleep etc…

It seems there are so many ways and needs in automation, that can influence when processing needs to take place. TiddlyWiki fits nicely into the visulisation and interaction space, or on demand dash board.

• I did a lot of this kind of stuff in my career from downloading and uploading to / from Aircraft when they arrive at the airport to logging load and passenger records, and much more.
• I found the greater challenges when a lot of analysis was needed. I think tiddlywiki is good for this part of automation, but we can still leverage external tools, rather than trying to jam too much into tiddlywiki.

[saqimtiaz]

@stobot

The only way I can think to extend PowerAutomate to do the same input/output in TiddlyWiki is via file creation

The way I went about this was to watch for the creation/modification of files and using that as the trigger for a http request to the TW API. I used fswatch (node.js) to detect file system changes but you might want to have a look to see if you can find an equivalent solution that works with your stack.

[pmario]

@jeremyruston

One possibility might be for the server to watch what used to be called a “dropbox” directory, and now perhaps better called an “inbox” directory. Any tiddler files deposited there would be dynamically imported into the wiki. Perhaps the file would be deleted once it has been imported. Something simple like that would open the door to a bunch of useful scenarios, and the semantics should be a good deal simpler.

I would create a workflow like this.

• The server constantly watches for if a single file exists eg: load.inbox
• If the file exists, that’s the trigger to load all files from the inbox in one go
• using $tw.loadTiddlersFromPath(), which gives us the same functionality as the TW –load command.
• So we can import several files in “mixed” formats: .html, .tiddler, .tid, .json and even tiddlywiki.files
• When a file is imported 2 things can happen:
  1. Log some file info into a log-file → Delete file
  2. Log file info, Move file to an “archive” directory instead of deleting the file
• When everything is imported → Delete the load.inbox file

With this workflow it’s easy to test and prepare the files in the inbox manually.
Also an external batch file can check for the load.inbox file. As long as it exists, no new files should be moved to the inbox, or an error can be raised for manual intervention.

Just some thoughts.

[stobot]

Thanks for the reminder, I’m testing the watch-fs now and it somewhat works in TiddlyWiki App

What I’m testing:
Watch FS — Modern TiddlyWiki plugin developing framework
Releases · tiddly-gittly/watch-fs

With this installed

1. I can edit tiddlers with an external application (such as notepad), save, and wiki will update automatically.
2. I can copy the .tid out of the folder, edit it, and copy it back in and overwrite it and things also go well.
3. When I copy a .tid that wasn’t already in the wiki, it behaves strangely.
   a. It automatically adds a second file with a _1 suffix
   b. It shows in the wiki, but further in-wiki editing only changes the _1 file strangely, even though the _1 is not shown anywhere in the wiki.
4. When I delete some tiddlers in the folder (not in the wiki), I get a Javascript error at sync.

So, while this is a bit finicky, if I can continue to use watch-fs safely within TiddlyWiki App and use my automation to only replace existing tiddlers, then I can have automated refreshes which is what I’m going for.

If anyone who knows more sees a problem with this methodology, please let me know!

[nemo]

@saqimtiaz

somewhat similar setup

That’s a good writeup, thankyou! I’ve not played with nodemon before, but will be keeping it in mind, and watch-fs looks even more useful.

@TW_Tones

I took intergration and Automation as meaning reoccuring, not once off.

I think we’ve been talking a little cross purposes here. I was advocating for node TW to see and pickup changes to the filesystem (and refuting claim that would necessarily be expensive) as useful, and my thinking at the time was primarily my own usecase so far, which is one-off imports.

In general, I think I’d categorise “automation” as being reoccurring, but “integration” could be either reccuring, or once-off/adhoc.

@TW_Tones

Yes, we do need expierence with these tools, but if you are going todo many one off things, or lots of steps for something once off it is worth the investment.

I doubt I’ll ever be importing from tomboy notes ever again. Or from tw classic files. Or from cctiddly, etc. More relevant for me (and only mentioned above as an aside), it’s important to me that I not only have the data in TW, but in the git repo that is storing the history of the data. That is, I’m importing the history into git. The final version is also visible to TW. But afaik, TW doesn’t manage local git repos, certainly not to the esoteric git usage I do. So even if I do end up running similar imports of similar data, I would do the same. If the history-to-git wasn’t a consideration, even then I still think “convert to (relatively simple) .tid format and write to a file” is simpler than “convert to valid json format, and send via API call”.

However circling back to automation of regular tasks, I’ve been thinking of some of the things I’ve got automated to other adhoc solutions and what could be put into TW instead, and I think the watch-fs will suit that well (though I also need to delve into the plotting thread: Does Tiddlywiki have a preferred Plotting/Graphics js tool? - my current adhoc setup is writing data in csv format and visualising it using dygraph, but updating the CSV within TW, having TW pickup the new data, and then visualise that within TW I think will be a much nicer workflow)

@stobot

If anyone who knows more sees a problem with this methodology, please let me know!

Those cases where it works well look ideal! Thank you for testing. Natve watch-fs for live edits to existing files, and nodeTW API for adding / deleting, will make for the ideal mix that never requires node to be restarted?