JSON Tiddlers -- Ease of Use

Developers
#1

Nice work @Scott_Sauyet laying this out systematically.

As we can all see the key to most JSON but especially nested multi-level is the use of a “hierarchical key”, built from a number of names found inside a JSON. This results in what looks like repetition or verbosity, although under the current paradigms is “necessary”.

However as a community we are really quite good at handling hierarchical structures, consider the TOC macros, the Table of contents and ancestor and descendant operators, TOCP etc… and now I would argue with the power of filters and functions we can do this better for JSON Structures using what we already have.

It is your/our mission, should you choose to accept it, to design and share a code pattern that makes this easier, for new and existing users alike.

  • If no one else does I will in good time but this should be a community effort.

Some thoughts

I think the key here may be some custom operators that have simple parameters that then use the various JSON operators, and particularly support the multilevel keys. So that when coding we are no so much referring to JSON but the structure in the JSON, its internal logic and names.

There is also a strong reason to make use of nested lists widgets to process each level of of a JSON and display it all at once with view and edit features allowing users to navigate such data visually.

One thing I have being working on is regular expressions to identify the basic layout of any JSON such as arrays or (name not coming to mind as of writing). The idea would be to identify the basic structure and from there we can create suitable solutions for that type of JSON.

To make this simpler to visualise, we may need to construct names for levels and keys either from the data or a standard set eg when a numeric index is needed to select between items, then within that another numeric index or key names.

Then all of the above should make it easy to go from visualisation, lookup, to retrieving and doing actions against the JSON both in TiddlyWiki Script and interactively such as edit. I do think this can mostly be handled by the existing features but it will also help us determine if the core needs any other minor tweaks to open this out for everyone to make full use of JSON.

Finally we also need to document and discuss the value of enhancing JSON use in TiddlyWiki when it would appear to many to contravene the Tiddler as the basic unit, I will do that soon if no one else does.

More help needed with json tiddlers
More help needed with json tiddlers
#2

I find that relatively straightforward to do… when all we want to do is fetch data from a structure stored in a JSON string. But the minute we want to manipulate it, we run hard into the fact that TW’s data is generally either single strings or lists of strings, themselves stored as single strings. We could certainly build tools that simplified doing this, but I believe they would always be going against the grain of TW.

Fair warning: this is an impossible problem to do thoroughly. JSON is not what is called a regular language, and so cannot be reliably parsed with regular expressions. This has to do with the arbitrary deep nesting it allows. You can use regex to parse specific JSON formats, but things fall down as you get more complex. The same is true with HTML, and my all-time favorite StackOverflow answer shows the exasperation of trying to explain that over and over.

Count me among the sceptics. I’m thinking about trying to do something similar to your Tips and commentary - Notes on tiddlers stored in a data tiddler, to demonstrate other ways to create it and document it, but I don’t expect that this will ever be a large part of my TW experience. (I looked at it a little last night, and if I can do it in a few hours over the weekend, I will probably try to write up my results.)

1 Like
#3

Please say how?

But at first we can find a simple way to determine if it contains an array or tiddler formats. I already have a set of regular expressions to assist in this. We may even be able to detect when “there go JSON monsters”. Me I am only interested in the cream off the top as a start. 20% of the work to solve 80% of the problems.

As I have posted recently the JSON Mangler tool can collapse a complex structure to a flat JSON with compound keys to every item. This can be handled more easily and would permit the use and editing of a whole range of JSON formats in one hit. Of course we can build the tools to help flaten unflatten, and handle a long compound key. But we could already do this to support [jsonset[A],[B],[C]] and get?

I think reasons why JSON does not have the support it should is because it is thought of from an open ended definition, we need only handle the key uses, and expand as needed. I dont think we should be deciding what designers should use JSON for, just give a complete set of tools and let the designers do what they wish. This opens us to innovation and new ideas, and they will grow or collapse in the market place of ideas.

I would like to see this, and wonder why as well :nerd_face:

#4

A vero.

Given JSON is a fundamental base structure in TW I’d guess the more one understands JSON the more one can do?

Of course there is a self-protect in TW from auto-cannibalism.
Meaning: it’s data store is itself IN JSON so activity IS limited to in-tiddler?

That said, it is interesting to consider one TW eating another.

Meaning parse and manipulate the JSON data store in A TW from inside a tiddler in another TW?

Just a comment
TT

#5

This can already be done, although I have not done it recently you can either import a TiddlyWiki into another or treat is as a file containing JSON and extract tiddler and this is even without coming at it in the Node file system.

It is an interesting observation to consider along with its Non-trivial Quine nature.

I do think if you constructed a tiddler containing a tiddler inside it, in the same form as it is stored in the html file I doubt it will be a problem, as there is some encoding that happens. Also keep in mind that the tiddlers are loaded from the HTML into browser memory, then saved back when requested. So as a rule we manipulate tiddlers in memory.

The tiddlers form of tiddler encoding can be used to construct PLUGINS and are read into memory as shadow tiddlers. Eric recently extended his SaveAs solution to allow sending filtered tiddlers in Advanced search to be pumped into a Plugin, Import and JSON tiddler. We can import JSON files while avoiding the Import process using a custom $browse widget, or interrupting the Import mechanism. Add the recently discussed InnerWiki and the ability to store and export tiddlers/files in a zip file, my own work on no-touch flags and we have a plethora of interesting possibilities.

  • TiddlyWiki is the only tool I know that can build its own SDK Software development Kit/Platform.

As I gave recently in a discussion with @EricShulman JSON containing tiddlers allows you to take tiddlers out of the tiddler store and save a version, and other tricks. Imagine, you could save a days work in one tiddler.

#6

I’m afraid that this is a misconception.

TW’s basic internal data structure is a collection of tiddler objects. Those tiddler objects themselves are hash maps/dictionaries, that is, mappings between string names and string values.

TW’s basic interface is either a single string or a list of strings (titles or otherwise.). That is what we use to display content and what we use to query and manipulate that internal data store.

That is true even for operations that look to involve other data types: the multiply operation accepts a list of strings (perhaps a list of only one) that might represent numbers and a separate string that might represent a number and returns a new string that represents a list of numbers. 1 2 3 4 5 +[multiply[5]] is a list of the strings “1”, “2”, “3”, “4”, and “5” along with the multiply operator and the parameter “5”. It returns the list of strings “5”, “10”, “15”, “20”, and “25”, usually stored as a single space-separated string.

JSON is not a data structure. It is the transport mechanism for a data structure, a single string format for passing simple or complex data structures between systems. On your file system, or on the other end of a web request, we do not find TW’s collection of dictionaries. That would make no sense; such a structure is an in-memory construct. Instead we find a string to represent that structure, one that is passed to your browser and parsed into the actual structure when TW starts.

This is not just picking nits. There is a very important word in that last sentence: “parsed”. To load JSON into TW, we first need to parse it. This is a much more time-intensive process than the manipulations we perform on the in-memory object store. We want to do that as infrequently as possible. Similarly we want to serialize the data back to a string only when we save. It’s less expensive than parsing, but still far more expensive than working with our in-memory model. JSON was built to be as inexpensive as possible to parse and serialize from JS, but it is still tremendously faster to work with the in-memory structures.

The same is true for any in-wiki JSON. Part of the reason that it is hard to work with is that when we store data in this transport format, every query and every manipulation involves parsing , then the work we want to do against an in-memory structure, then, often, another serialization.

We might want to suggest that this could be cleaned up by adding a system that somehow stored such JSON data in memory, only parsing it on startup and only serializing it on save. We could. But then we’d need an interface for manipulating it in memory; we wouldn’t want to require users to be JS experts.

We already have such a system for working with in-memory data. That’s what our filter language is for! Building a second, parallel system offers very little bang for its buck.

This is why some of us are recommending using JSON tiddlers very sparingly. They can’t become easy to use without a lot of work. But for the most part, we can convert our JSON constructs into collections of tiddlers, which we can work with easily with existing tools.

All the above also answers the following:

#7

Right.

For mortals an example might be useful.

Footnote: “Eating” a TW inside another in one Tiddler is likely easier when the “Eaten” TW is one that externalizes it’s core. Less weighty.

It’s Robinson Crusoe all over again.

TT

#8

I thought the TW data store was structured in JSON?

Screenshot 2025-11-15 12.52.54
Screenshot 2025-11-15 12.52.541634×229 135 KB

I guess you trying to go further.

Right now I’m confused.

TT

#9

No. Again, that is what I called the transport mechanism. There’s probably a better term, since it’s also what’s stored in the TW file. But it’s not what’s in memory as the wiki operates. Think of this string:

{"name": "Jill", "age": 19, "eyes": "hazel"}

What sorts of things can you do to it as a string?

You could take the first ten characters:

{"name": "

You could upper-case all the characters

{"NAME": "JILL", "AGE": 19, "EYES": "HAZEL"}

You could reverse it

}"lezah" :"seye" ,91 :"ega" ,"lliJ" :"eman"{

You could apply some regex, and do a number of other things.

But in order to answer the question “Is this person over 21 years old?”, you will need to parse it into an actual in-memory object. In order to say, “No, her eyes are actually green,” you would need to parse it, make the change, then serialize it back into a string to save in the text field of the tiddler in question.

The text you quoted is like this. It’s what TW can parse to create its in-memory store. And TW can serialize its store to save the wiki. But its not what is used to run the wiki. If every TW interaction involved reparsing and/or reserializing the data, it would extremely, paralyzingly, sssssssslllllllloooooooowwwwwwww.

At a smaller scale, that is what we’re doing whenever we use JSON tiddlers. We first parse, then manipulate the JSON data , and then if we’re updating, serialize it back to a string to save. If the equivalent data was in tiddlers, we would have only done this parsing once, and only done the serializing on save.

It’s not that JSON is never appropriate. For deeply nested structures, it might well be your best bet. I’m all in favor of adding the jsondelete operator: it’s necessary in to be able to use JSON well. Although I’ve tried to demonstrate that no such tool can ever cover all potential JSON, I have no objection to adding some operators to allow you to use keys such as mother.address.city instead of the three separate parameters. But I don’t think its worth spending a great deal of time on optimizing the usage of JSON. Most uses of JSON tiddlers, in my opinion, would be better written as a collection of standard tiddlers.

#10

That’s right, but it is used to store our data as serialized text, which is machine readable and somewhat human readable too.

As Scott wrote. At the wiki startup we need to take that string and parse it. Since JSON is (now) used all over the web, all browsers support parsing and serializing JSON strings natively. So it has good performance, but is slow compared to “in memory” object handling.

I can 100% sign that. Using data tiddlers, that contain a structure that looks like a tiddler, is like: “Why make it simple, if it can be complicated.”

IMO the JSON-operators have been primarily designed to be used with the tm-http-request message.

They should allow us to read data from 3rd party sites as JSON and convert them into tiddlers, so we can easily work with them.

It is true, that working with JSON operators should be easier, especially if the 3rd party structure is completely different to what we need.

But imo working with “tiddlers” inside a data-tiddler is the wrong way to go. Except to use them as a transport medium, for export / import.

1 Like
#11

This is a good point. I’m sure there are other appropriate reasons too, but this is the best reason I’ve seen for using JSON operations in TW.

#12

Okay. From a user view of me I get confused. I am now hearing two stories.

  1. TW data is stored in JSON

  2. AFTER that there can be complications of the @Scott_Sauyet kind?

How, practically, should I respond?

Seems a bit complex?
TT

p.s. My complication got complicated …

#13

That’s right. It is parsed once at startup and is serialised once on save. All the time in between we work with JavaScript objects that are handled efficiently in the JS engine.

In contrast all JSON operators have one function in common. See line 29

Everytime we use eg: jsonextract the input data needs to be parsed and the result needs to be serialised for the output (see line 33).

On the input side of every other json* operator there is a new parse step. … and so on.

For small data-structures that’s OK but it seems users want to create their own tiddler-data-store. Doing it that way is complex and completely inefficient. A data-tiddler is no database.

Converting JSON data into tiddlers once and use tiddlers IMO is much more efficient.

I did create a write-up a GitHub discussions 4 years ago: How the TW internal data structure looks like and why data-tiddlers are not optimal · TiddlyWiki/TiddlyWiki5 · Discussion #6116 · GitHub which goes a bit deeper and describes a data-dictionary tiddler. But the problem with the text-index parsing is the same as with JSON data in the text field.

#14

With a career in IT behind me I see that what is happening yet again.

I takes time to understand json and more so when there are gaps in the tools and documentation. even then once you understand it it needs to be understood in the perspective of the core and how it’s used in tw and code environments.

so if you are aware of tw in general but not json you will look around to fill a few gaps in the tools you have.
If json is not the answer we need to first understand the questions.

the premise we need to start with is there are gaps we need to fill and we need to find out how to, rather than trying to resist the demand and blocking progress.

:thinking:

#15

I think you’re misunderstanding the fundamental nature of the problem here. Sure, there are some gaps that should be filled. jsondelete, for instance, is an essential companion to the other json* operators! But what I am trying to point out (and I would guess that @pmario agrees) is that the very basics of how JSON works is so at odds with the core of Tiddlywiki that an attempt to fully integrate JSON tiddlers with the rest of TW is the matter of a full core rewrite, and the resulting tool would likely not be entirely compatible with current TW. It would overturn our “everything is a tiddler” mantra. It would require totally new data storage mechanisms. Its deeply nested nature would most likely mean a hugely expanded collection of filter operators. It’s a gigantic undertaking. And for little benefit, as most anything we can do with JSON tiddlers we can also do with normal tiddlers.

I’m not merely suggesting that such tools would be too hard to write. I believe that such a tool would need a brand new name because its too different from current TW to be called the same thing.

2 Likes
#16

computer says no?

I am not even saying the solution needs to be JSON. But the solution may be adjacent, but we need a solution as is evidenced by related questions popping out all over the place.

lets us find the root causes without dismissal of people’s needs perceived or otherwise.

#17

This is a really interesting discussion!

To me it kinda illustrates a difference between folk (you & Scott?) deeply aware of how TW’s fundamentals work (“basement thinking”) and end users not interested, or capable, in the basement.

Your data-dictionary comment really illustrates the polarity.

For practical purposes it is entirely healthy and functional to see data-tiddlers as databases.

Why not?

Just a comment
TT

#18

So? I’m trying to understand, practically where this thread comes out?
Are there any changes you are suggesting?

Just a comment
TT

Mantra: OM MANI PADME HUM (Tibetan style)

#19

I’m sorry, but I thought this thread was about JSON. It started with @Eskha’s list of very specific how do I do X with JSON? questions and has proceeded to discuss JSON in every post.

If you want to talk about some abstraction from there, might I suggest you start a new topic?

I’ve put in a lot of effort to thoroughly explain deep objections to expanding TW’s core JSON capabilities. To hear it suggested that I’m merely being dismissive is disheartening.

1 Like
#20

@Scott_Sauyet, it looks like you inadvertently edited @TiddlyTitch’s post instead of replying him, would you mind reverting to his original post please?

Fred

1 Like