Import-like process that matches title, but updates (adds) new or selected field(s) only?

This may be a long-shot, maybe aspirational question/request.

For many of my books, I have Library of Congress card-catalogue call# info (a call# field) in an older FileMaker database (I lived in FM for many years before finding TW, and still use it for certain workflows). I have managed to develop a workable JSON-generating (bibtex-style) export process that can help get these biblio records across from FileMaker into TiddlyWiki. Mostly good.

Alas, variant tiddler versions of many of those same records already exist in my TiddlyWiki (where I’ve been building a set of bibtex records for use with solutions by @Mohammad), and these tiddlers include data for fields (image-url, notes, tags, etc.) that are not present in my FileMaker records.

Within FileMaker’s own import process, problems like these are relatively easy to solve: given an incoming duplicate (based on a “match” field), the import dialogue allows one to leave various existing fields untouched, and simply to add the content of desired fields. This is fantastic database behavior… Alas, FileMaker doesn’t directly import JSON. So I can’t simply accomplish this merging of fields on the FM side (after which I could re-import, overwriting the tiddlers with the desired merged-field variants).

Questions:

(1) How hard would it be to get a variant of TiddlyWiki’s import process to merge or “zip together” tiddlers – so that existing tiddlers (based on matching title) might be not simply overwritten, but instead modified only by adding incoming field-data pairs that are new (where existing tiddler lacks those fields)? ALTERNATELY, how hard would it be to add some checkboxes to cells in the “Diff” preview pane (in the import interface) so that one could manually accomplish something like a selectively-merged tiddler?

(2) Would something like this be of interest to others? Could we collectively commission someone to build a solution for this workflow challenge?

Note: I realize for many people, the simple question “which tiddler variant is more recent/complete?” is the extent of the import problem. Merging divergent branches of a tiddlywiki project is often handled well just by selecting which (whole) tiddlers to override, based on being able to inspect their fields. This is itself already important and great.

So, I’m asking whether a merge solution at the next level is possible, when two title-matching tiddlers (or in my case, JSON strings) each carry field-value pairs that belong in the result.

(SMALL update: I suppose my JSON should specify the incoming field title as bibtex-LCCN, since some bibtex records do include this field. Still, none of the bibtex records I’ve imported, so far, come with this field included. What info I have has been manually typed in, over the years, within FileMaker.)

1 Like

I would be tempted to bypass the import mechanism after the data is in the $:/Import tiddler, and just process it with wikitext. Adding only fields that do not previously exist would be very straightforward and not too different from the wikitext I have posted earlier for importing tiddlers from one wiki to another at startup. If this is still unresolved after the weekend I can have a look at it for you.

Creating a UI to determine which fields are merged etc would be fiddly and time consuming.

1 Like

I’m afraid I don’t know what you mean with the suggestion

Once data is in the $:/Import tiddler, is there a way to perform batch operations on it? I’m talking about many tiddlers, all of which need to be handled in identical fashion, adding (in this case) just a particular field. Of course, if this field-merge task isn’t a challenge that others also face, then it wouldn’t make sense for anyone to put great effort into solving it!

If it’s worth solving, I imagine the interface could look roughly like the screenshot below (with the overall “selected for import” checkbox for the tiddler itself becoming one of those “interdeterminate” boxes [-] as soon as a partial field-list is specified):

Yes. The code posted at New Features in TiddlyWiki 5.3.0: HTTP request and deserialize operator creates JSON in the same format as the import tiddler and then iterates over it to create corresponding tiddlers. What you would need is the same logic, just adapted to only add new fields for existing tiddlers.

1 Like

@Springer my approach would be to start an import and rename the $:/import tiddler and change its plugin-type from import to plugin. Avoid using the same tiddler names as existing tiddlers.

  • This will change it to a plugin and the tiddlers become shadow tiddlers
  • I would then build a batch process to itterate the shadow tiddlers within the plugin and make changes to the relavant existing tiddler.
  • Once complete you can delete the “import plugin” and recover the space.

Here is a really quick and minimally tested example that adds an additional button to the import tiddler. The button triggers an alternate import which only adds previously non-existent fields to tiddlers that are already present in the wiki: My TiddlyWiki — a non-linear personal web notebook

Please do not use this with your actual wikis without a fair bit of testing!

The minimum improvements that you would probably want to this would be to reset the $:/Import tiddler after the merge, and to display a list of which tiddlers were imported and which were updated.

3 Likes

Saq,

This is fantastic, as far as I can tell in limited testing. So far, I’ve just used your demo site to batch import first from one source and then from another (using the modified merge function on the second import). Then, after being satisfied that the results are exactly what I’m looking for, I port the resulting set of tiddlers back over to my own wiki, safely overwriting the earlier tiddlers that had been missing certain fields.

I’m very grateful for this solution, especially whipped up so quickly!

I’m not sure how many other folks would be interested. But it seems likely that bibliography projects in particular (insert shoutout to @Mohammad for Refnotes here) are likely to face this need to merge tiddlers from multiple sources, where each source fills in certain details but not others.