I think that could be a WikiText widget that makes use of the json* filter operators in a more accessible way. It’d be called <$action.setjson>, then.
Most of the fiddling would probably be to verify that all intermediate index levels are given, and maybe to figure out how many levels were given and to handle that in a flexible way.
In Emily’s case, inside a $list, this would still lead to a lot of tiddler writes.
I am continually frustrated with JSON in TW. If I try hard enough, I can usually get things done, but it’s non-trivial.
You can do it with full paths to your nodes: [<myJSONstring>jsonset[grandpa],[mom],[me],[over21],[yes]]. But you need to set all intermediate nodes before you add a property to them. The above won’t work unless the object your JSON string represents already has a grandpa property, which has a mom property, which has a me property that is an object:
<$let myJSONstring='{"grandpa": {"mom": {"me": {"name": "Scott"}, "bro": {"name": "Michael"}}}}'>
<pre><code><$text text={{{[<myJSONstring>jsonset[grandpa],[mom],[me],[over21],[yes]] +[format:json[4]] }}} /></code></pre>
</$let>
yielding
{
"grandpa": {
"mom": {
"me": {
"name": "Scott",
"over21": "yes"
},
"bro": {
"name": "Michael"
}
}
}
}
But yes, that is awkward as hell.
This is because of what JSON actually is. It is not a data structure; instead it is a string representation of a data structure. That sounds subtle, but the point is important. Although tiddlers are pretty simple data structures (Hash Maps / Dictionaries), most everything else we deal with in TW is a string or a list of strings.
JSON was designed as a transport format: a way to share nested data structures between different systems. But working with JSON in programming languages usually involves parsing JSON into a more sophisticated data structure on receipt, manipulating that structure as needed, and serializing that data structure back into a string before passing it along. In TW, there’s more friction in this than in many languages. But even if there were no friction, the model would still involve converting from a string, making changes, and converting back.
Mostly to sharpen my TW-JSON skills, I tried this, and the results are not too bad:
<$button>Extract
<$action-createtiddler $basetitle="MyExtract" type="application/json" text="{}" >
<$let tid=<<createTiddler-title>> >
<$list filter=[tag[HelloThere]]>
<$action-setfield $tiddler=<<tid>> $field="text" $value={{{
[<tid>get[text]jsonextract[]]
+[jsonset:object{!!title}]
+[jsonset{!!title},[title],{!!title}]
+[jsonset{!!title},[tags],{!!tags}]
+[jsonset{!!title},[caption],{!!caption}]
+[format:json[4]]
}}} />
</$list>
<$action-navigate $to=<<tid>> />
</$let>
</$action-createtiddler>
</$button>
which yields this:
{
"A Gentle Guide to TiddlyWiki": {
"title": "A Gentle Guide to TiddlyWiki",
"tags": "HelloThere",
"caption": ""
},
"Discover TiddlyWiki": {
"title": "Discover TiddlyWiki",
"tags": "HelloThere",
"caption": ""
},
"Some of the things you can do with TiddlyWiki": {
"title": "Some of the things you can do with TiddlyWiki",
"tags": "HelloThere",
"caption": ""
},
"Ten reasons to switch to TiddlyWiki": {
"title": "Ten reasons to switch to TiddlyWiki",
"tags": "HelloThere",
"caption": ""
},
"Examples": {
"title": "Examples",
"tags": "HelloThere Community",
"caption": ""
},
"What happened to the original TiddlyWiki?": {
"title": "What happened to the original TiddlyWiki?",
"tags": "HelloThere",
"caption": ""
},
"Funding TiddlyWiki": {
"title": "Funding TiddlyWiki",
"tags": "About HelloThere",
"caption": ""
},
"Open Collective": {
"title": "Open Collective",
"tags": "About HelloThere",
"caption": ""
}
}
Obviously it would be easy to parameterize the filter used (here [tag[HelloThere]]. I’d have to think about how to do dynamic filters, or possibly use reduce, in order to change the list of fields to include (here title, tags, and caption.) But I’m sure it could be done.
But this is only for a simple, two-level structure. I have some ideas of how to make it simpler to build arbitrary JSON strings from simple parameters. I do it in JS all the time, and so I guess a JS operator might be enough. But I’m trying to figure out how to do it in wikitext.
Extract Tiddlers.tid (676 Bytes)
I think an $action-setjson widget would be an excellent solution, and would be more semantically transparent than further overtaxing $action-setfield.
Would it be prohibitively difficult to allow a path to be expressed via named subindexes as well as numerically? Maybe I’m misunderstanding, but I’d assume that using numeric parameters would require you to know the full structure of the JSON being manipulated in advance—which subindexes a given index has, and in what order. That seems a little inconsistent with the way the index-focused tools currently function: I don’t need to know anything about the current sequence of indexes in a data tiddler to use $action-setfield.
Thank you for your consideration!
My biggest concern is with arrays. I have useful techniques in JS to do this. But it distinguishes arrays from objects by the existence of integer parameters, which we really can’t do in TW. This seems to limit the utility a bit. But that’s only a bit. I think we could generally assume that by
[foo][0][bar]
[foo][1][baz]
the user means
{
foo: ['bar', 'baz']
}
instead of
{
foo: {
'0': 'bar'
'1': 'baz'
}
}
If that’s acceptable, I could probably work on this. I’m leaving in a few days for a two-week vacation, so it wouldn’t be immediately.
The KeyValues plugin doesn’t handle these specific index writing/retrieval issues, but it’s a great tool for anyone working with data tiddlers, so thank you for mentioning it here, @Sunny!
For anyone who hasn’t encountered it before, @pmario’s KeyValues introduces a new filter operator that lets you search index keys and their values simultaneously; then, for any indexes that match on either side, it returns either the key, the value, or both. This is possible but much harder to do with vanilla TW operators, and would typically require multiple filter runs.
Honestly, I’d love to see something like keyvalues in the core as well… 
There has been a PR: Implement a data tiddler keyvalues operator by pmario · Pull Request #3971 · TiddlyWiki/TiddlyWiki5 · GitHub, which was rejected, because data-tiddlers should be discouraged.
There was a PR: Discourage / Deprecate data tiddlers by pmario · Pull Request #4389 · TiddlyWiki/TiddlyWiki5 · GitHub, which was rejected, because we are not there yet.
As the json-operators where introduced, I did have an idea, how I could make it easier to work with them in a similar way as the keyvalues-plugin. I did not immediately write down the mechanism I had in mind. – So I forgot it 
IMO data-tiddlers will go nowhere, since we have the json-operators
That’s a pity! I agree with you (and @Mohammad, as per that GH thread): data tiddlers are perfect for “unique” data. And I say this as a huge proponent of fields and templating! I work with a lot of data that absolutely does not need to be chopped up into tiddlers with all their associated metadata, and I can’t justify making a once-off field for every key-value pair I’d like to record. I can’t imagine any alternatives that wouldn’t be less efficient… even JSON tiddlers are more verbose when it comes to simple key-value pairs.
Edit: Here’s @jeremyruston’s (2019) position as per the first PR you linked:
The implementation of data tiddlers is spread throughout the core code, and hard to work with. I think that the whole idea was misguided: it was an attempt to avoid the proliferation of tiddlers, but I now think that that is a non-goal.
I hope this isn’t indicative of the current design goals. My largest wikis have upwards of 20,000 tiddlers — and the majority of those have some substantive text content, or at least more then 2-3 “user” fields. I’m always trying to cut down on total file size and increase filter efficiency, so keeping my tiddler count as small as feasible is a big priority for me. And similar, my field list is already so crowded that I tend to disable the editor dropdown.
If I were to convert all my data tiddlers to “full” tiddlers, it would bloat my wikis substantially. In fact, the experiments that inspired my OP were born out of a desire to avoid stub tiddlers!
In any case, thank you (to both @pmario and @jeremyruston) for continuing to support those of us who do use and love data tiddlers.
Drifting OT here but;
I don’t understand why we don’t just use fields as a lightweight alternative to dictionary tiddlers. I mean, it is already an established concept in TW. A dictionary list looks pretty much like a bunch of custom fields. If a user understands custom fields, then “fields based data tiddlers” are almost self evident, what it is, how to use etc.
I’d guess fields could be “extended” to allow for nesting if needed (mytiddler!!fieldlevel1!!fieldlevel2!!...) and even visually they could be indented. This is way more “wikitexty” than having to use json.
I was actually dreaming (well, not literally) about this last night! It would really cut down on the the number of prefixes needed to establish a unique field name or to get fields to sort together… and indenting “child” fields would be a nice visual indicator.
It wouldn’t obviate my desire for data tiddlers, though. [fields[]count[]] = 753 in my largest wiki, and [all[shadows+tiddlers]fields[]count[]] gets up to 814. I wouldn’t want to further pollute that space with hundreds (if not thousands!) of single-use field names… indexes are appealing precisely because they don’t “count”, performance-wise, unless I explicitly choose to include them.
I wonder, though, if we could simply handle this at a view/edit level, with storage in JSON, to avoid adding new data types that TW needs to care about. Imagine an editor something like this: JSON Editor Online: edit JSON, format JSON, query JSON.
A good example of a practical use for DataDictionary tiddlers (type=application/x-tiddler-dictionary) is TiddlyTools/Settings/Colors/X11, which defines a mapping from X11 color names to #RRGGBB color values.
Similarly, in the TWCore, the palette tiddlers ($:/palettes/...) are DataDictionary tiddlers that define names used by the <<colour ...>> macro to apply CSS color values in $:/themes/tiddlywiki/vanilla/base and other theme definition tiddlers.
In both cases, the simple name:value format is much easier to read and edit than using JSON tiddlers (type=application/json) which have lots more punctuation (curly braces, quotes around text values, commas) and are much more prone to errors.
Having said that…
For most of my TiddlyTools configuration tiddlers ($:/config/TiddlyTools/...), I use JSON tiddlers that are managed via dedicated UI interfaces that use <$action-setfield $tiddler=... $index=... $value=.../> or <$edit-text tiddler=... index=.../> widgets so users rarely need to hand-edit or even view the underlying JSON configuration.
Still, in some cases, I’ve used fields to store the configuration settings, and in a few instances, I’ve even used a combination of both indexes AND fields in the same config tiddler.
-e
Right. How things are actually stored is probably not so important from a users pov, but the interface is.
I don’t get the functionality in that link to work: “Error: Local document with id “local.bedite” not found” - and same for the right hand window.
This might work better:
But that was just in the first search result. There are dozens or more of these.
Hi @jeremyruston, here is my first quick stab at an <$action.setjson> widget, written completely in WikiText:
\widget $action.setjson($tiddler)
<!-- if any intermediate levels are not given, they are assigned the "0" index -->
\define defaultIndex() 0
\function thisLevelIndex() [<widgetParameters>jsonget<level>else<defaultIndex>]
\function previousLevel() [<level>subtract[1]]
\define check-success() [jsonextract<setLevel>match<setValue>]
<!-- we try to assign the value as a JSON object and fall back to string assignment if that fails (e.g. when assigning the actual "value" parameter) -->
\function .json-then-string(setLevel,setValue) [all[]jsonset:json<setLevel>,<setValue>filter<check-success>] :else[all[]jsonset:string<setLevel>,<setValue>]
<!-- we iterate (in reverse order) and build up the multi-level JSON object -->
\function .jsonBuilder(level) [<level>compare:integer:gt[0]] :then[all[].jsonBuilderInner<level>else[]] :else[all[]]
\function .jsonBuilderInner() [all[]] :map[[].json-then-string<thisLevelIndex>,<currentTiddler>] +[.jsonBuilder<previousLevel>]
<$parameters $params="widgetParameters", 0=<<defaultIndex>> >
<$let input={{{ [<$tiddler>type[application/json]get[text]format:json[]] }}}
level={{{ [<widgetParameters>jsonindexes[]search::regexp[^\d+$]maxall[]] }}}
value={{{ [<widgetParameters>jsonget[value]jsonstringify[]] }}}
jsonObject={{{ [<value>.jsonBuilder<level>] }}}
output={{{ [<input>.json-then-string<0>,<jsonObject>] }}}
>
<!-- this is for debugging when not used as an action -->
<$codeblock code={{{ [<output>format:json[4]] }}} />
<% if [<$tiddler>type[application/json]] %>
<$action-setfield $tiddler=<<condition>> text={{{ [<output>format:json[4]] }}} />
<% endif %>
</$let>
</$parameters>
\end
This will set a string value at a position in the JSON hierarchy which is defined by integer parameters, starting at 0 like so:
<$action.setjson $tiddler=<<jsonTiddler>> value="This is some text." 0=h 1=i 2=j />
The nested hierarchy is created if it does not exist, e.g. if there is no existing index “h” in the above example. If hierarchy levels are omitted in the parameters, they are still created with a default index value of 0.
If anyone wants to play around with this, I made a Sharing Edition.
PS: I realized that this will completely overwrite anything that is already assigned the 0-level index. Therefore, this widget couldn’t be used in its current state to insert some property somewhere in the deeper structure of the JSON object. I’ll try to fix this, but this might be non-trivial and may require a different approach.
I did try this, as a simple variation of jsonset. The code has a very different style from TW core, so is not right for the core, but that would be easy enough to fix. If the current behavior is a bug, as Jeremy says above:
then is this the sort of behavior we’d want?
Original
myJSONstring{ "grandpa": { "name": "Hudge", "mom": { "name": "Gee", "me": { "name": "Scott" }, "bro": { "name": "Michael" } } } }Can add to deepest nodes
[<myJSONstring>jsondeepset[grandpa],[mom],[me],[over21],[yes]]{ "grandpa": { "name": "Hudge", "mom": { "name": "Gee", "me": { "name": "Scott", "over21": "yes" }, "bro": { "name": "Michael" } } } }Can add to first-level node
[<myJSONstring>jsondeepset[grandpa],[aunt],[cuz],[name],[Tom]]{ "grandpa": { "name": "Hudge", "mom": { "name": "Gee", "me": { "name": "Scott" }, "bro": { "name": "Michael" } }, "aunt": { "cuz": { "name": "Tom" } } } }Can use numeric properties
[<myJSONstring>jsondeepset:number[grandpa],[aunt],[kidCount],[2]]{ "grandpa": { "name": "Hudge", "mom": { "name": "Gee", "me": { "name": "Scott" }, "bro": { "name": "Michael" } }, "aunt": { "kidCount": 2 } } }Can add to root node, string together calls in one filter, and use boolean content
[<myJSONstring>jsondeepset[grandma],[name],[Barbara]] +[jsondeepset:number[grandma],[born],[1914]] +[jsondeepset:boolean[grandma],[deceased],[true]]{ "grandpa": { "name": "Hudge", "mom": { "name": "Gee", "me": { "name": "Scott" }, "bro": { "name": "Michael" } } }, "grandma": { "name": "Barbara", "born": 1914, "deceased": true } }Can create arrays using numeric node names
[<myJSONstring>jsondeepset[grandpa],[favoriteColors],[0],[purple]] +[jsondeepset[grandpa],[favoriteColors],[1],[orange]] +[jsondeepset[grandpa],[favoriteColors],[2],[blue]]{ "grandpa": { "name": "Hudge", "mom": { "name": "Gee", "me": { "name": "Scott" }, "bro": { "name": "Michael" } }, "favoriteColors": [ "purple", "orange", "blue" ] } }
You can try this by downloading the following, dragging it to a wiki, saving and reloading, then opening the jsonDeepSet Test tiddler:jsonDeepSet.json (4.0 KB).
Another reasonable API choice might be:
[<myJSONstring>jsondeepset[grandpa.mom.me.over21],[yes]]
where we could override the . separator with a second prefix, in case our node names happen to include .:
[<myJSONstring>jsondeepset:boolean:/[grandpa/mom/me/over21],[true]]
This would be easy to change in my implementation.
The key is to allow the use of variables, and the syntax of variables should not conflict with other syntax. This can be challenging in JSON with multiple levels. Additionally, I believe that fields with deeper values are worth exploring. However, this would undoubtedly increase the workload, as many widgets may require refactoring or the addition of new code. Therefore, I think prioritizing JSON or data entries is a better approach.
There are indeed some scenarios where deeper field levels may be necessary. For example, when working with Markdown files that contain multi-level YAML content, converting these files to TiddlyWiki can be quite challenging.
Of course, a combined approach is also possible. For instance, the field content of a particular entry could be fully controlled by another JSON tiddler, with the text body excluded from it.
Although JSON might be necessary at times, for this, I think that what you did with markdown-importer works well.
---
title: "TestPage"
another:
foo: 1
bar: 2
baz: 3
---
## Content
becomes
title: TestPage
another.foo: 1
another.bar: 2
another.baz: 3
!! Content ##
Of course there are potential issues if the keys contain .'s, but I think it’s mostly a reasonable approach. That’s clearly related to my alternative API above.
Maybe my ignorance is showing through here, but a number of solutions exist that resemble what is being asked for here, and the depth of understanding to make true comparisons or analysis of the options available is difficult for me (at present). For Example I was just looking at @Flibbles Introducing TW5-Graph, Tiddlymap's spiritual successor! which quickly leads to the prior work on XML and the xpath and xselect widgets and filter operators.
If you already have a good understanding of JSON/XML and these tools please have a look and tell us what you think.
It seems to me we need to try and bring together a disparite set of data representation, queries and manipulation and try and use convergence to make skills developed in tiddlywiki or anyone of these global standards applicable accross others. There by “steepening the learning curve” ie: making each quicker to learn and adopt (Note: the correct use of the often missused slope of the learning curve)
This concerns me in so far as I accept this may be depricated in the tiddlywiki core, and we may not wish to promote the use of Data tiddlers, however even when depricated, tools to use to manage and manipulate datatiddlers are essential even if just to assist in migrating away from data tiddlers.
However I also think we are over looking something and that is TiddlyWiki participates in the global internet, app, software and data ecosystem, and regarless of what is chosen to be core behaviours in tiddlywiki there is a whole universe out there. That is it will often be the case than key value data is available out there or used out there and will always be needed as are the import and export and manipulation of such data.
Although the Data Tiddler concept is realy just a “file” of rows or key value pairs, and is one of the easier structures for novice users to understand. To “remove this” is to deminish TiddlyWiki.
There is something called JSONpath which is analogous to XPath. It could in theory work. If people wanted this stuff baked in so that manipulation was simpler, I recently proposed making data tiddlers and index-fetching into modules.
Right now, you’ve got text/x-tiddler-dictionary, and you can index and modify it using things like ##key.
And then there’s application/json which accepts ##jsonKey and such again for single-level objects fetching and setting.
If dataTiddler logic were in modules, then a plugin could make indexing support JSONPath, which would make setting entires in a json tiddler’s tree as simple as:
<$action-setfield $index="$.fieldlevel1.fieldlevel2[*].price" $tiddler="PricesTiddler" />
which would neatly set all json paths that match that pattern. The dollar sign is a standard of JSONPath, I think, which would help preserve backward compatibility.
Just a thought. I know DataTiddlers are kinda passe.