Exporting all relevant (related/linked) data

#1

I don’t think this has been solved (though my search-fu is lacking) but occurred to me that it might be possible to integrate TiddlyWiki with GenAI by means of using TW’s export feature. Most AI models can likely handle WikiText and an export of an tiddler with all its related tiddlers could be formatted in a textual format well suited as context to an AI model.

However, something I’ve struggled with when exporting is transcluding. Since exporting to a textual format (CSV, JSON, TID) does not expand transcluded content; and because transcluded tiddlers are not part of the [links[]] nor tag tiddlers, etc. it isn’t easy to ask TW to export a tiddler along with all its relevant context tiddlers.

Is it possible to export all relevant tiddlers associated with a filter of tiddlers to make for a complete relevance (context) graph? And if not how might one go about making a plugin to do so?

#2

I suppose it is possible to send the results of a [links[]] filter to the story river and then print to PDF. But that’s kind of a complicated workaround and only works by happenstance that most AI models understand PDF content.

#3

I’m not sure whether it would catch all your relevant tiddlers, but I’d probably start by looking at the relink:references and relink:backreferences operators included in Relink plugin. For example, you can test [tag[Demo]relink:references[]] +[has[title]] in the Advanced Search on the plugin site, which lets you download all the resulting tiddlers in your preferred format. (If you’re not concerned about shadow tiddlers, you could use +[is[tiddler]] in place of +[has[title]]; I used the latter because it ought to catch any shadows referenced by your tiddlers.) You’d probably also need to configure Relink to check any additional fields you may be using prior to using this filter.

I think the relink operators may be more likely to miss associated tiddlers if you make heavy use of ViewTemplates in your wiki — especially if any of your templates reference content from associated tiddlers whose titles are programmatically derived, rather than listed in a field. For instance, if your base filter returns Tiddler A and your ViewTemplate uses [<currentTiddler>addprefix[$:/data/]], relink:references might not catch $:/data/Tiddler A.

Just some quick thoughts. Perhaps @Flibbles can speak to whether this is a reasonable use of Relink.

#4

I’m sure this is doable. I have done this with my genealogy TiddlyWiki solution. I wrote a tiddler “Export ancestors of…”, where the user selects a person tiddler and it generates all the tiddlers to export. The export recursively locates all people, events, places, photos, etc that are associated with the ancestors to the selected person.

#5

Have you tried using export all on the Page Controls menu? I belive it expands transclusions, though keep in mind content may be duplicated, is that transclusion is done more than once.

  • You could clone the existing export all and make new templates and export in a desirable format. I expect LLMs (No AI exists yet) can make use of the headings and other structure to assist with understanding.
#6

I know that some people use the Relink filters for stuff like that. They’re documented, and won’t be changed, so they’re safe to use.

As for picking up stuff in viewTemplates, that is tricky. If you don’t want to write your own module, the best way to go would be to whitelist the fields those viewTemplates use. Between that and Relink-Titles, which would let you pick up stuff like the [<currentTiddler>addprefix[$:/data/]] example @etardiff gave, you should be able to accomplish what you need.

2 Likes