There have been several discussions recently about how to deal with JSON tiddlers (reading, dealing with object structure, deleting, …).
However I am missing some elements to join the dots.
More especially I would like to create / read / write / update JSON tiddlers that contains a set of tiddlers with some selected set of their fields. For instance:
Most of the content is replicated below, but I’ve created a tabbed tiddler with the answers:. You can download the following, drag it to any wiki and import the tiddlers. Then the tiddler JSON Question should show examples of doing each of these (except that the deletes aren’t really working well before 5.4.0.)
Before we get to answering the individual questions, we have some notes.
General Notes
We create a new JSON string
When we do our manipulation, we don’t directly alter the content of that tiddler. We create a new JSON string. This is important to note. It takes a second step to update the original, if that’s what we want.
For a second step, we could do this:
<$button>Add new object to MyJson
<$let result={{{
[{MyJson}]
+[jsonset:object[A third tiddler]]
+[jsonset[A third tiddler],[icon],[icon3]]
+[jsonset[A third tiddler],[color],[#facade]]
+[format:json[4]]
}}} >
<$action-setfield $tiddler="MyJson" $value=<<result>> />
</$let>
<$button>
We could combine these into a single action like this:
<$button>
Add new object to MyJson
<$action-setfield $tiddler="MyJson" $value={{{
[{MyJson}]
+[jsonset:object[A third tiddler]]
+[jsonset[A third tiddler],[icon],[icon3]]
+[jsonset[A third tiddler],[color],[#facade]]
+[format:json[4]]
}}} />
<$button>
The parameters can be any wikitext values.
We are not limited to hard-coded values like [A third tiddler], [icon] and [#facade]. We can use other reference, such as variables <title> or <medium-pink> or tiddler field references such as {!!main-icon} or {some tiddler!!icon}.
So we might include a line like this:
+[jsonset{!!title},[color],<medium-pink>]
Little white-space.
By default, there is no unnecessary white space in JSON output. I add +[format:json[4]] to the display of of each filter here so they are easy to read. But in general, this is only necessary if the result is going to be put somewhere you’re going to be viewing the JSON directly. Otherwise, default output like this is perfectly usable for tools that work with JSON:
{"A tiddler":{"icon":"my-icon","color":"#fedcba"},"Another tiddler":{"icon":"another-icon","color":"#abcdef"},"A third tiddler":{"icon":"icon3","color":"#facade"}}
The object in a JSON string do not have to be tiddlers
The questions seemed to assume that what you store in JSON is always tiddlers. While there are many such uses across TW, what is stored in JSON strings can be arbitrary data.
This would be perfectly legitimate JSON data:
{
"books":[
{
"isbn":"9781593279509",
"title":"Eloquent JavaScript, Third Edition",
"subtitle":"A Modern Introduction to Programming",
"author":"Marijn Haverbeke",
"published":"2018-12-04",
"publisher":"No Starch Press",
"description":"JavaScript lies at the heart of almost every modern web application...",
},
{
"isbn":"9781491943533",
"title":"Practical Modern JavaScript",
"subtitle":"Dive into ES6 and the Future of JavaScript",
"author":"Nicolás Bevacqua",
"published":"2017-07-16",
"publisher":"O'Reilly Media",
"description":"To get the most out of modern JavaScript, you need...",
}
]
}
Note that this isn’t tiddler data, and there is more than one level of nesting.
Answers
We do this by first adding an empty object, and then setting its properties individually:
Code
[{MyJson}]
+[jsonset:object[A third tiddler]]
+[jsonset[A third tiddler],[icon],[icon3]]
+[jsonset[A third tiddler],[color],[#facade]]
Result
{
"A tiddler": {
"icon": "my-icon",
"color": "#fedcba"
},
"Another tiddler": {
"icon": "another-icon",
"color": "#abcdef"
},
"A third tiddler": {
"icon": "icon3",
"color": "#facade"
}
}
Similarly, this is not available for working with a JSON string. But if you want to directly update a field in a tiddler (here the text field), you can use action-setfield, and not supply the usual $value parameter
Nice work @Scott_Sauyet laying this out systematically.
As we can all see the key to most JSON but especially nested multi-level is the use of a “hierarchical key”, built from a number of names found inside a JSON. This results in what looks like repetition or verbosity, although under the current paradigms is “necessary”.
However as a community we are really quite good at handling hierarchical structures, consider the TOC macros, the Table of contents and ancestor and descendant operators, TOCP etc… and now I would argue with the power of filters and functions we can do this better for JSON Structures using what we already have.
It is your/our mission, should you choose to accept it, to design and share a code pattern that makes this easier, for new and existing users alike.
If no one else does I will in good time but this should be a community effort.
Some thoughts
I think the key here may be some custom operators that have simple parameters that then use the various JSON operators, and particularly support the multilevel keys. So that when coding we are no so much referring to JSON but the structure in the JSON, its internal logic and names.
There is also a strong reason to make use of nested lists widgets to process each level of of a JSON and display it all at once with view and edit features allowing users to navigate such data visually.
One thing I have being working on is regular expressions to identify the basic layout of any JSON such as arrays or (name not coming to mind as of writing). The idea would be to identify the basic structure and from there we can create suitable solutions for that type of JSON.
To make this simpler to visualise, we may need to construct names for levels and keys either from the data or a standard set eg when a numeric index is needed to select between items, then within that another numeric index or key names.
Then all of the above should make it easy to go from visualisation, lookup, to retrieving and doing actions against the JSON both in TiddlyWiki Script and interactively such as edit. I do think this can mostly be handled by the existing features but it will also help us determine if the core needs any other minor tweaks to open this out for everyone to make full use of JSON.
Finally we also need to document and discuss the value of enhancing JSON use in TiddlyWiki when it would appear to many to contravene the Tiddler as the basic unit, I will do that soon if no one else does.
I find that relatively straightforward to do… when all we want to do is fetch data from a structure stored in a JSON string. But the minute we want to manipulate it, we run hard into the fact that TW’s data is generally either single strings or lists of strings, themselves stored as single strings. We could certainly build tools that simplified doing this, but I believe they would always be going against the grain of TW.
Fair warning: this is an impossible problem to do thoroughly. JSON is not what is called a regular language, and so cannot be reliably parsed with regular expressions. This has to do with the arbitrary deep nesting it allows. You can use regex to parse specific JSON formats, but things fall down as you get more complex. The same is true with HTML, and my all-time favorite StackOverflow answer shows the exasperation of trying to explain that over and over.
Count me among the sceptics. I’m thinking about trying to do something similar to your Tips and commentary - Notes on tiddlers stored in a data tiddler, to demonstrate other ways to create it and document it, but I don’t expect that this will ever be a large part of my TW experience. (I looked at it a little last night, and if I can do it in a few hours over the weekend, I will probably try to write up my results.)
Thank you Scott for the time you have tanke and this very detailed and useful answer with both key ideas and step by step examples covering all my use cases.
I think that the main idea to remember when dealing with json tiddlers (and that I was missing):
To manipulate json tiddlers, don’t directly alter the content of that tiddler. Create a new JSON string using a filter and jsonset operator then update the original
But at first we can find a simple way to determine if it contains an array or tiddler formats. I already have a set of regular expressions to assist in this. We may even be able to detect when “there go JSON monsters”. Me I am only interested in the cream off the top as a start. 20% of the work to solve 80% of the problems.
As I have posted recently the JSON Mangler tool can collapse a complex structure to a flat JSON with compound keys to every item. This can be handled more easily and would permit the use and editing of a whole range of JSON formats in one hit. Of course we can build the tools to help flaten unflatten, and handle a long compound key. But we could already do this to support [jsonset[A],[B],[C]] and get?
I think reasons why JSON does not have the support it should is because it is thought of from an open ended definition, we need only handle the key uses, and expand as needed. I dont think we should be deciding what designers should use JSON for, just give a complete set of tools and let the designers do what they wish. This opens us to innovation and new ideas, and they will grow or collapse in the market place of ideas.
These sound like good ideas. I will look at both when I’m on a computer and not lying at bed at 4 am having already done Wordle* and my two other puzzles but still unable to get back to sleep.
I don’t know if I can change just my follow-up post to Tips & Tricks. But so long as @Eskha is okay with it, I can do it for the whole thread.
I didn’t write those tiddlers in order to post them. TiddlyWiki has simply gotten to be how I think. It’s how I organize my brain for many problems. When I was done but didn’t have time to write up my complete answer here, I added a brief one with those tiddlers to give the OP as timely a response as I could. Later I came back to give a proper response and decided others might like that approach too, so left them. But they certainly were not intended at first to act as reference docs, or I would have prefaced their titles as you suggest.
This can already be done, although I have not done it recently you can either import a TiddlyWiki into another or treat is as a file containing JSON and extract tiddler and this is even without coming at it in the Node file system.
It is an interesting observation to consider along with its Non-trivial Quine nature.
I do think if you constructed a tiddler containing a tiddler inside it, in the same form as it is stored in the html file I doubt it will be a problem, as there is some encoding that happens. Also keep in mind that the tiddlers are loaded from the HTML into browser memory, then saved back when requested. So as a rule we manipulate tiddlers in memory.
The tiddlers form of tiddler encoding can be used to construct PLUGINS and are read into memory as shadow tiddlers. Eric recently extended his SaveAs solution to allow sending filtered tiddlers in Advanced search to be pumped into a Plugin, Import and JSON tiddler. We can import JSON files while avoiding the Import process using a custom $browse widget, or interrupting the Import mechanism. Add the recently discussed InnerWiki and the ability to store and export tiddlers/files in a zip file, my own work on no-touch flags and we have a plethora of interesting possibilities.
TiddlyWiki is the only tool I know that can build its own SDK Software development Kit/Platform.
As I gave recently in a discussion with @EricShulman JSON containing tiddlers allows you to take tiddlers out of the tiddler store and save a version, and other tricks. Imagine, you could save a days work in one tiddler.
TW’s basic internal data structure is a collection of tiddler objects. Those tiddler objects themselves are hash maps/dictionaries, that is, mappings between string names and string values.
TW’s basic interface is either a single string or a list of strings (titles or otherwise.). That is what we use to display content and what we use to query and manipulate that internal data store.
That is true even for operations that look to involve other data types: the multiply operation accepts a list of strings (perhaps a list of only one) that might represent numbers and a separate string that might represent a number and returns a new string that represents a list of numbers. 1 2 3 4 5 +[multiply[5]] is a list of the strings “1”, “2”, “3”, “4”, and “5” along with the multiply operator and the parameter “5”. It returns the list of strings “5”, “10”, “15”, “20”, and “25”, usually stored as a single space-separated string.
JSON is not a data structure. It is the transport mechanism for a data structure, a single string format for passing simple or complex data structures between systems. On your file system, or on the other end of a web request, we do not find TW’s collection of dictionaries. That would make no sense; such a structure is an in-memory construct. Instead we find a string to represent that structure, one that is passed to your browser and parsed into the actual structure when TW starts.
This is not just picking nits. There is a very important word in that last sentence: “parsed”. To load JSON into TW, we first need to parse it. This is a much more time-intensive process than the manipulations we perform on the in-memory object store. We want to do that as infrequently as possible. Similarly we want to serialize the data back to a string only when we save. It’s less expensive than parsing, but still far more expensive than working with our in-memory model. JSON was built to be as inexpensive as possible to parse and serialize from JS, but it is still tremendously faster to work with the in-memory structures.
The same is true for any in-wiki JSON. Part of the reason that it is hard to work with is that when we store data in this transport format, every query and every manipulation involves parsing , then the work we want to do against an in-memory structure, then, often, another serialization.
We might want to suggest that this could be cleaned up by adding a system that somehow stored such JSON data in memory, only parsing it on startup and only serializing it on save. We could. But then we’d need an interface for manipulating it in memory; we wouldn’t want to require users to be JS experts.
We already have such a system for working with in-memory data. That’s what our filter language is for! Building a second, parallel system offers very little bang for its buck.
This is why some of us are recommending using JSON tiddlers very sparingly. They can’t become easy to use without a lot of work. But for the most part, we can convert our JSON constructs into collections of tiddlers, which we can work with easily with existing tools.
No. Again, that is what I called the transport mechanism. There’s probably a better term, since it’s also what’s stored in the TW file. But it’s not what’s in memory as the wiki operates. Think of this string:
{"name": "Jill", "age": 19, "eyes": "hazel"}
What sorts of things can you do to it as a string?
You could take the first ten characters:
{"name": "
You could upper-case all the characters
{"NAME": "JILL", "AGE": 19, "EYES": "HAZEL"}
You could reverse it
}"lezah" :"seye" ,91 :"ega" ,"lliJ" :"eman"{
You could apply some regex, and do a number of other things.
But in order to answer the question “Is this person over 21 years old?”, you will need to parse it into an actual in-memory object. In order to say, “No, her eyes are actually green,” you would need to parse it, make the change, then serialize it back into a string to save in the text field of the tiddler in question.
The text you quoted is like this. It’s what TW can parse to create its in-memory store. And TW can serialize its store to save the wiki. But its not what is used to run the wiki. If every TW interaction involved reparsing and/or reserializing the data, it would extremely, paralyzingly, sssssssslllllllloooooooowwwwwwww.
At a smaller scale, that is what we’re doing whenever we use JSON tiddlers. We first parse, then manipulate the JSON data , and then if we’re updating, serialize it back to a string to save. If the equivalent data was in tiddlers, we would have only done this parsing once, and only done the serializing on save.
It’s not that JSON is never appropriate. For deeply nested structures, it might well be your best bet. I’m all in favor of adding the jsondelete operator: it’s necessary in to be able to use JSON well. Although I’ve tried to demonstrate that no such tool can ever cover all potential JSON, I have no objection to adding some operators to allow you to use keys such as mother.address.city instead of the three separate parameters. But I don’t think its worth spending a great deal of time on optimizing the usage of JSON. Most uses of JSON tiddlers, in my opinion, would be better written as a collection of standard tiddlers.
That’s right, but it is used to store our data as serialized text, which is machine readable and somewhat human readable too.
As Scott wrote. At the wiki startup we need to take that string and parse it. Since JSON is (now) used all over the web, all browsers support parsing and serializing JSON strings natively. So it has good performance, but is slow compared to “in memory” object handling.
I can 100% sign that. Using data tiddlers, that contain a structure that looks like a tiddler, is like: “Why make it simple, if it can be complicated.”
IMO the JSON-operators have been primarily designed to be used with the tm-http-request message.
They should allow us to read data from 3rd party sites as JSON and convert them into tiddlers, so we can easily work with them.
It is true, that working with JSON operators should be easier, especially if the 3rd party structure is completely different to what we need.
But imo working with “tiddlers” inside a data-tiddler is the wrong way to go. Except to use them as a transport medium, for export / import.
