myTideR: Gotta Tiddl' Em All!

,
#1

Edit: The myTideR package for R can be found at myTideR: Gotta Tiddl' Em All! • myTideR

The new MultiWikiServer plugin is very exciting, and the ability to have many collections of tiddlers gathered in one database - while still being able to keep them separate - revives an old dream that many of us have surely had: to let TiddlyWiki be one’s external brain that collects and remembers everything important.

Imagine having one bag of tiddlers called scripts. These tiddlers contain scripts that run, for example, every hour. We may call them tides, representing waves of tiddlers being added by the external scripts. Some of us prefer R, so I started drafting the myTideR package as a proof-of-concept:

Examples of what the tides can be:

  • Import emails to the mail bag. Using IMAP, the script downloads new emails, converts them to TiddlyWiki format. Attachments are downloaded to the file storage (see below).
  • Import files to the file bag. The file storage contains external files. The script ensures that all files have a corresponding tiddler containing the file hash (to avoid duplicates) and file path. Also, I know Jeremy is working on improved handling of external files. Perhaps we can even use pandoc to convert some file formats to TiddlyWiki.
  • Import calendar events to the event bag. The script creates new tiddlers from a list of iCal files.
  • Import external notes to the external-note bag. The script converts markdown files from Obsidian or Logseq to TiddlyWiki format and creates corresponding tiddlers. Imports file attachments and embeds images.
  • Import web pages to the web-archive bag. If you mention a URL in a tiddler and put it in brackets ([[http://example.com]]) the script will try to archive it through SingleFile and the Internet Archive, and create a corresponding tiddler.
  • Import RSS feed to the RSS bag.
  • Import historical weather data to the weather bag.
  • Export static site. For public - redacts links to private tiddlers - or just for easier interaction with external files etc.

Does anyone have experience with this type of pipeline? Or have ideas for other types of scripts (Python?) that could be useful?

(Somewhat inspired by Peter Hajas: TiddlyWiki Everywhere)

1 Like
#2

I don’t use scripts for task you mentioned, but do use scripts to process publications and references of colleagues.

A few R scripts are developed to

  • download publication list from Google Scholar, ORCID, Web of Science, colleague homepage.
  • Add tags of colleagues to publications.
  • Get reference list from crossref and add into bibtex-entry.

My Tiddlywiki is setup as node.js under windows laptop and I communicate with server with my R package, rtiddlywiki (https://rtiddlywiki.bangyou.me/).

The script is running with schedule task in windows at daily, and with hotkey with auto hotkey at requirement (manually).

2 Likes
Use R scripts to manage reference in Tiddlywiki
#3

Very nice @Zheng_Bangyou! I use R quite often myself, so this is perfect.

As a quick test, I made two new tiddlers: (1) To allow the filter through the web api

title: $:/config/Server/ExternalFilters/[tag[Script]]
text: yes

and (2) a tiddler containing the script, tagged with Script and text:

```r
title <- paste('This', 'title', 'is', 'great')
text <- paste("!! Section",
          "This is a new tiddler", 5+5, sep="\n")
type <- "text/vnd.tiddlywiki"
tags <- c("Tag1", "Tag 2")
fields <- list("field 1" = "value 1", "field 2" = "field 2")
put_tiddler(title = title, 
            text = text,
            type = type, 
            tags = tags,
            fields = fields)
```

(note that we can have the code formatted as codein the tiddler).

@Zheng_Bangyou: I had to make the fields a list variabel and concatenate the text to avoid errors

In R, we can use the script

# devtools::install_github('byzheng/rtiddlywiki')
library("rtiddlywiki")
tw_options(host = "http://127.0.0.1:8080/")

script_tiddlers <- get_tiddlers(filter = "[tag[Script]]")
lapply(script_tiddlers, function(script_tiddler) {
  script_to_parse <- get_tiddler(title = script_tiddler$title)$text
  script_to_parse <- gsub('```r', '', script_to_parse)
  script_to_parse <- gsub('```', '', script_to_parse)
  eval(parse(text = script_to_parse))
})

and it successfully creates a new tiddler, just as expected! To make R do more, we can simply add new tiddlers and tag them Script

1 Like
#4

Glad it is working for you. Just keep in mind. rtiddlywiki is under development and doesn’t have lots of test. I am developing for my own purpose.

Some data might be lost. Create backup before using it.

#5

You can import google calendar with ical-calendar-importer

You can already make TW an all-in-one second brain, by importing all calendar event and webpages (A Web Clipper browser extension for NodeJS wiki
I don’t know what MultiWikiServer can improve with this? Performance?

#6

Thanks @linonetwo, I agree that many functions and plugins already exist. One thought may be that tiddlywiki excels at many things, but not everything and that other tools can offload the wiki at such things. I think the MultiWikiServer will be useful for performance itself, but perhaps also by providing the sqlite database as a new interface, as well as providing easier mechanisms for tracking which tiddlers that have been changed (by id)

Also, I have updated the main post with some further testing in R

#7

That’s true, current nodejs wiki can’t track what tiddler is changed (or deleted), so sync with other services may have bug (cause dupliction, etc.)

#8

It would also be interesting to include information from google takeout - such as youtube and location history (and perhaps contacts, keep, calendar?). The latter is interesting given the new Geospatial Plugin. However, the large number of tiddlers may affect performance so I need to think a bit about structure (e.g., one tiddler per address with appended timestamps?)

#9

The myTideR package can now be found at github: myTideR: Gotta Tiddl' Em All! • myTideR with some support for both the “ordinary” server and MWS!

3 Likes