A little background, to begin with: I recently started exploring @Mohammad’s SearchWikis plugin, which creates an index of (a subset of) all the tiddlers in a given wiki.
- Each wiki index is stored as a single
application/jsontiddler. - Within the JSON tiddler, each tiddler title is an index key, and each value records a few field values from that tiddler. The default index-value used by the JSON generator is
$(title)$|$(caption)$|$(tags)$. - Thus, the JSON output might look like this:
{
"\"A free, open source wiki revisited\" by Mark Gibbs, NetworkWorld": "\"A free, open source wiki revisited\" by Mark Gibbs, NetworkWorld||Articles",
"\"A Thesis Notebook\" by Alberto Molina": "\"A Thesis Notebook\" by Alberto Molina||Examples",
"\"ATWiki\" by Lamusia Project": "\"ATWiki\" by Lamusia Project||Resources",
"\"BJTools\" by buggyj": "\"BJTools\" by buggyj||Resources",
"\"BrainTest - tools for a digital brain\" by Danielo Rodriguez": "\"BrainTest - tools for a digital brain\" by Danielo Rodriguez||Resources"
}
This JSON tiddler gets imported into a “master” wiki, where Mohammad uses split[|] to retrieve the individual field contents from each index value:
<$vars inTitle={{{[<result>split[|]nth[1]]}}}
inCaption={{{[<result>split[|]nth[2]]}}}
inTags={{{[<result>split[|]nth[3]]}}} >
This is all very clever, and it works well, but when I started hacking, slashing, and otherwise adapting the concept for my own purposes, I started thinking that there had to be a better way to write and access these values. A nested JSON structure would take a few more characters, but it would be more legible at a glance…
{
"\"BJTools\" by buggyj": {
"title": "\"BJTools\" by buggyj",
"tags": "Resources"
}
}
And surely, (I thought naively1) it would be easier to access named values. After all, we have purpose-built operators like jsonget!
If you’ve ever tried to work with complex JSON structures in TW, you may already have guessed that this didn’t go as smoothly as I’d hoped.
Pain points in more or less the order I encountered them:
- Right off the bat: There’s no obvious way to create a nested JSON in the first place.
- Adding indexes to a flat structure is very simple: just use
$action-setfield. For instance:
<$list filter="[tag[HelloThere]]" variable="index">
<$action-setfield $tiddler="My JSON Index" $index=<<index>> $value={{{ [<index>get[created]] }}} />
</$list>
yields
{
"A Gentle Guide to TiddlyWiki": "20150325170720682",
"Discover TiddlyWiki": "20140904121000000",
"Some of the things you can do with TiddlyWiki": "20140904090300000",
"Ten reasons to switch to TiddlyWiki": "20140904085700000",
"Examples": "20140320230543190",
"What happened to the original TiddlyWiki?": "20140904085100000",
"Funding TiddlyWiki": "20221204165636777",
"Open Collective": "20221204165636777"
}
- However, it doesn’t seem to be possible to set or modify nested indexes without rewriting the entire text field of the JSON tiddler. Here’s an example by @Scott_Sauyet:
I did a little experimenting with this approach and found:
- It doesn’t seem to be possible to use
jsonsetunless you’re starting with an existing JSON tiddlerSomeTid. - It’s difficult to iterate over a $list as I did with
$action-setfieldabove. I’d hoped I could use a:maprun to add X field/value pairs for each of Y tiddlers, but I couldn’t figure out how to get this to work withjsonset.
Eventually I gave up on this approach and tried to build the structure I wanted without any JSON-specific operators. This is horribly inelegant, but it does seem to work: JSON Builder.tid (1.3 KB)
\define indexTiddler() JSON test
\define include() [tag[HelloThere]]
\function get.fields() created modified caption tags :filter[<..currentTiddler>has{!!title}] :map[field.index[]]
\function make.string() =[["]] [all[]] =[["]] +[join[]]
\function make.key() [{!!title}make.string[]] =[[: ]] +[join[]]
\define lb() {
\define rb() }
\function tiddler.index()
[make.key[]] [<lb>] +[join[]]
[get.fields[]join<comma-break>]
[<rb>] +[join<lbr>]
\end
\function field.index()
[make.key[]] [<..currentTiddler>get{!!title}make.string[]] +[join[]]
\end
<$link to=<<indexTiddler>> />
<$let
lbr="""
"""
comma-break=""",
"""
text=```{
${ [subfilter<include>] :map[tiddler.index[]] +[join<comma-break>] }$
}```>
<$button>
write text
<$action-setfield $tiddler=<<indexTiddler>>
type="application/json"
text={{{ [<text>] +[format:json[4]] }}} />
</$button>
<pre>
<$transclude $variable=text $mode=block />
</pre>
</$let>
`{{{ [[JSON test]indexes[]] }}}`
{{{ [[JSON test]indexes[]] }}}
`{{{ [[JSON test]get[text]jsonget[Examples],[created]] }}}`: {{{ [[JSON test]get[text]jsonget[Examples],[created]] }}}
`{{{ [[JSON test]get[text]jsonget[Examples],[tags]] }}}`: {{{ [[JSON test]get[text]jsonget[Examples],[tags]] }}}
You can see some (working) filter tests at the bottom of that code, which was where I had my second unwelcome revelation:
-
jsongetdoes not work likegetindex.
-
getindexexpects its input to be a title:[[$:/palettes/Vanilla]getindex[background]] -
jsongetexpects its input to be a complete JSON string — so if you’re working with a JSON tiddler, you need to useget[text]before you can retrieve a specific value.
And this felt so inefficient for regular use in a filter — particularly when I’d hoped to use it with large JSON tiddlers like the ones produced by SearchWikis — that I decided Mohammad had had good reasons to choose the format he did, and gave up on my revisions entirely.
What I’d love to see in a future version of TW:
- An
<$action-setfield $index= ...equivalent that makes it easy to add nested, programmatically-determined key-value pairs — whether the target tiddler already exists or not.- Or (ideally?) an extension of the existing
$action-setfieldwidget?
- Or (ideally?) an extension of the existing
- Similarly, an easier alternative to
jsonget. IMO, it’d be most intuitive to extendgetindexto take multiple parameters- Current solution:
[[JSON test]get[text]jsonget[Examples],[tags]] - Ideal solution:
[[JSON test]getindex[Examples],[tags]]
- Current solution:
1 I know very little about JS; I’ve never really used it for anything more complicated that storing data in TW, and even then I generally let $action-setfield do the work for me. It’s very possible that I’m missing some easy solutions (though I did search the docs, this forum, and the top Google results I found, and discovered that I’m not the first to struggle with this, and none of the previous topics seemed to include any satisfactory solutions…) If so, please tell me! I’d love to make this work.
