JSON Tiddlers -- Ease of Use

[Scott_Sauyet]

Oh damn. I won’t try it from my phone. That was probably the cause of the problem in the first place. But I’ll try to straighten it out ASAP.

Thanks for the heads-up.

Edit: fixed. Sorry about the mess up. And again, @tw-FRed, thanks for catching it.

[Scott_Sauyet]

@TiddlyTitch

So? I’m trying to understand, practically where this thread comes out?
Are there any changes you are suggesting?

Except for the minor addition of jsondelete, I, for one, am not suggesting any changes.

Obviously, people can add their own tools to work with their particular JSON formats, but I am trying to demonstrate that there is no reasonable approach to integrating more general JSON-related tools into the core. A superficial integration would be very slow because of the parse-manipulate-serialize cycle on every transaction. A deeper integration would require a radical restructuring of the core and a huge expansion of the operator space, with very little benefit to end users.

@TW_Tones seems to find my responses dismissive. I don’t accept that. I’m trying very hard to explain why I see the suggested expansions as a problem. I am certainly not trying to appeal to my TW expertise.

@TiddlyTitch

To me it kinda illustrates a difference between folk (you & Scott?) deeply aware of how TW’s fundamentals work (“basement thinking”)

I reject that. Mario clearly has a deep knowledge of the core, alongside @saqimtiaz and @jeremyruston, and, I think, in a slightly different way, @EricShulman. I’m not a novice there, but I barely make it to intermediate.

I am, though, an advanced JS developer. I understand the capabilities of the language TW is written in, and have nearly three decades of experience with it. I’ve been trying to explain (and not simply asset) the fundamental difference between TW’s tiddler store, which is a functional database, and JSON, which is primarily a storage/transport format.

When @pmario says that a data tiddler is not a database, it’s like saying that sheet music doesn’t groove or swing or jive. It takes musicians “parsing” it to bring it to life. Yes, those who read JSON/music notation can get some sense of what the parsed information would be like, but it’s nothing like attending a concert.

@TiddlyTitch

Mantra: OM MANI PADME HUM (Tibetan style)

Damn it, Titch, now you’ve got me humming everything from Hair!

[TW_Tones]

@Scott_Sauyet

I’m sorry, but I thought this thread was about JSON. It started with @Eskha’s list of very specific how do I do X with JSON? questions and has proceeded to discuss JSON in every post.

if you are saying json is not going to work or should not be used don’t be surprised if the conversation strays.

@Scott_Sauyet I have valued your contribution but you recent responses suggest you are not hearing what I said, nore being generous in your interpretation of what I have said. it is irrelevant if you think you have not being dismissive if another feels you have. let us just try to listen a little more deeply.

[Scott_Sauyet]

@TW_Tones

if you are saying json is not going to work or should not be used don’t be surprised if the conversation strays.

Ok. I still feel like a new topic is a better idea, but if you want to discuss it here, then…

@TW_Tones

I am not even saying the solution needs to be JSON. But the solution may be adjacent, but we need a solution as is evidenced by related questions popping out all over the place.

A solution to what? What fundamental problem are you trying to solve? I feel as though I gave a quite comprehensive answer to @Eskha’s original question, along with some additional information about the overall use of JSON.

You suggested a mission, which I’ve steadfastly refused to accept, to create an improved code pattern for using JSON. I’ve tried very hard to explain why I think this is a bad idea: it would make it easier for users to misuse the fundamentals of TW.

But if it’s not specifically JSON then I really don’t know what it is you’re looking for.

[TW_Tones]

There is JSON both as an input and as a data structure, this Topic presumably is about its use as it is about a gap, ease of entry deletion. The General heading is now “JSON Tiddlers – Ease of Use” so presumably we can at least try and “ease its use”,

Now If I am to take @pmario and @Scott_Sauyet advice that JSON is not an appropriate tool to deal with “extra tiddler data”, as given and I have not argued otherwise;

• we still can work on the JSON forms already available and used within TiddlyWiki however there is a reoccurring need identified by others who seem to have data in JSON forms that are deeper and more structured that the basic forms TiddlyWiki can currently handle.
• What we need at a minimum is to help people transfer such structures into a form that we can handle in TiddlyWiki, if we are to do this we need to start with those new forms.

I cant speak for everyone, but I am an observer of what/everything has being discussed in the last couple of months and it is clear better access and ease of use into JSON structures is being raised for a number of reasons;

• People want to process or store more complex data structures
• Handle arbitrary JSON they have imported or found
• Maintain and export such data

Now these problems may be solved by alternate

Personally I am looking for the following as it stands;

• Easing the use of JSON in TiddlyWiki especially for the forms it comes in
  • Tiddlers as in plugins
  • JSON Data tiddlers as occurs using $index

I also expected it to help me “going forward”;

• To use it as an index between two keys and store additional information for that entry, with multiple items in one tiddler.
• To use it for data structures beyond the uses “As it Stands”

So for now I am assuming JSON is not suitable for these uses “going forward”, and the gap may be filled outside JSON, so how?

• I have taken this no JSON as a premise and think perhaps if we could convert complex JSON structures into a familiar structure, based on tiddlers and then ensure we can use and navigate them, that way may be a compromise. I could not see a way forward until I revisited the JSON Mangler approach to FLATTENing JSON into compound keys, where the compound key could be like system tiddlers; $:/datasetname/topkey/subkey/parameter with the values if any stored in the text field (or the title)

• $:/datasetname/ would remain the same

• There could be multiple topkey/subkey/parameter values Including;

  • $:/datasetname/topkey1 $:/datasetname/topkey2
  • $:/datasetname/topkey1/subkey1 $:/datasetname/topkey104/subkey10
  • $:/datasetname/topkey1/subkey1/parameter1 …

This approach may be able to handle the arbitrary structures people are presenting.

• We can then create ease of use tools for this “accepted method”

[EricShulman]

@TW_Tones

FLATTENing JSON into compound keys, where the compound key could be like system tiddlers; $:/datasetname/topkey/subkey/parameter with the values if any stored in the text field (or the title)

Way back in 2017, long before the JSON filter operators and the JSON Mangler, I created TiddlyTools/Widgets/action-unpackdata.js, which takes a tiddler containing JSON text, and converts it into a set of tiddlers as you’ve described, where the JSON “path” (aka “compound key”) is used as a tiddler title and the value is stored in the text field of that tiddler.

Note that it also handles XML formatted text via a built-in “convert XML to JSON” function, as well as custom Google Places API JSON data output retrieved via remote XMLHttpRequest and also TWCore-formatted tiddler definitions (i.e., plugins, themes, imports, etc).

It’s initial use was so that TiddlyTools/Panels/Location panel could submit latitude/longitude or address text queries to the Google Places API, and extract/process the returned JSON data to feed into the TiddlyTools/Panels/Maps and TiddlyTools/Panels/StreetView panels.

When the TWCore added the json filter operators, I added a “fallback” method that can perform the same “unpack” operation using just wikitext coding (see the getData_unpack() macro in TiddlyTools/Panels/Location).

One very interesting note about these two methods: the javascript-based $action-unpackData widget is more than 10 times faster than the wikitext-based code. For example, a typical response from the Google Places API might take 3 seconds to unpack using wikitext and json filter operators, but less than 0.3 seconds using the pure javascript widget coding.

-e

[saqimtiaz]

The main use case driving the introduction of the JSON filter operators was to support making HTTP requests from wikitext.Since JSON is one of the most common formats returned by HTTP APIs, we needed basic tools to read that data. Without those operators, the ability to make HTTP requests would have had very limited practical use. Because of this, most of the current operators focus on accessing and reading JSON.

The initial draft of the JSON operators worked with tiddlers instead of strings, so that the parsed JSON for each tiddler could be cached. However, this would have required that JSON first be stored in a tiddler before it could be read or manipulated. To avoid this requirement, the operators were refactored to work with strings, parsing them as JSON each time.

With the existing operators, you can parse JSON in any structure, read its contents from wikitext, and even convert it into tiddlers if you want. There are examples of this approach in the forum. Naturally, this does require a basic understanding of JSON, and I don’t see a realistic way around that. For the common JSON formats TiddlyWiki itself uses to store or export tiddlers, the deserialize[] filter operator gives access to deserializer modules, so you don’t need to work with raw JSON directly.

If you have workflows involving JSON that the current operators don’t support, please share them, ideally as short user stories or concrete examples. This will help us understand the real use cases, and whether any gaps exist in existing operators that have solutions that would be universal enough to include in the core.

Example hypothetical user stories:

• “As a user fetching data from an external API, I want to extract just the title and address fields from a deeply nested JSON object so I can display it on a wiki page without writing custom JS.”

• “As a plugin developer, I want to store configuration data as JSON and allow users to adjust individual values from wikitext, so that configuration remains both compact and accessible.”