Okay. From a user view of me I get confused. I am now hearing two stories.
TW data is stored in JSON
AFTER that there can be complications of the @Scott_Sauyet kind?
How, practically, should I respond?
Seems a bit complex?
TT
p.s. My complication got complicated …
That’s right. It is parsed once at startup and is serialised once on save. All the time in between we work with JavaScript objects that are handled efficiently in the JS engine.
In contrast all JSON operators have one function in common. See line 29
Everytime we use eg: jsonextract the input data needs to be parsed and the result needs to be serialised for the output (see line 33).
On the input side of every other json* operator there is a new parse step. … and so on.
For small data-structures that’s OK but it seems users want to create their own tiddler-data-store. Doing it that way is complex and completely inefficient. A data-tiddler is no database.
Converting JSON data into tiddlers once and use tiddlers IMO is much more efficient.
I did create a write-up a GitHub discussions 4 years ago: How the TW internal data structure looks like and why data-tiddlers are not optimal · TiddlyWiki/TiddlyWiki5 · Discussion #6116 · GitHub which goes a bit deeper and describes a data-dictionary tiddler. But the problem with the text-index parsing is the same as with JSON data in the text field.
With a career in IT behind me I see that what is happening yet again.
I takes time to understand json and more so when there are gaps in the tools and documentation. even then once you understand it it needs to be understood in the perspective of the core and how it’s used in tw and code environments.
so if you are aware of tw in general but not json you will look around to fill a few gaps in the tools you have.
If json is not the answer we need to first understand the questions.
the premise we need to start with is there are gaps we need to fill and we need to find out how to, rather than trying to resist the demand and blocking progress.
I think you’re misunderstanding the fundamental nature of the problem here. Sure, there are some gaps that should be filled. jsondelete, for instance, is an essential companion to the other json* operators! But what I am trying to point out (and I would guess that @pmario agrees) is that the very basics of how JSON works is so at odds with the core of Tiddlywiki that an attempt to fully integrate JSON tiddlers with the rest of TW is the matter of a full core rewrite, and the resulting tool would likely not be entirely compatible with current TW. It would overturn our “everything is a tiddler” mantra. It would require totally new data storage mechanisms. Its deeply nested nature would most likely mean a hugely expanded collection of filter operators. It’s a gigantic undertaking. And for little benefit, as most anything we can do with JSON tiddlers we can also do with normal tiddlers.
I’m not merely suggesting that such tools would be too hard to write. I believe that such a tool would need a brand new name because its too different from current TW to be called the same thing.
computer says no?
I am not even saying the solution needs to be JSON. But the solution may be adjacent, but we need a solution as is evidenced by related questions popping out all over the place.
lets us find the root causes without dismissal of people’s needs perceived or otherwise.
This is a really interesting discussion!
To me it kinda illustrates a difference between folk (you & Scott?) deeply aware of how TW’s fundamentals work (“basement thinking”) and end users not interested, or capable, in the basement.
Your data-dictionary comment really illustrates the polarity.
For practical purposes it is entirely healthy and functional to see data-tiddlers as databases.
Why not?
Just a comment
TT
So? I’m trying to understand, practically where this thread comes out?
Are there any changes you are suggesting?
Just a comment
TT
Mantra: OM MANI PADME HUM (Tibetan style)
I’m sorry, but I thought this thread was about JSON. It started with @Eskha’s list of very specific how do I do X with JSON? questions and has proceeded to discuss JSON in every post.
If you want to talk about some abstraction from there, might I suggest you start a new topic?
I’ve put in a lot of effort to thoroughly explain deep objections to expanding TW’s core JSON capabilities. To hear it suggested that I’m merely being dismissive is disheartening.
@Scott_Sauyet, it looks like you inadvertently edited @TiddlyTitch’s post instead of replying him, would you mind reverting to his original post please?
Fred
Oh damn. I won’t try it from my phone. That was probably the cause of the problem in the first place. But I’ll try to straighten it out ASAP.
Thanks for the heads-up.
Edit: fixed. Sorry about the mess up. And again, @tw-FRed, thanks for catching it.
Except for the minor addition of jsondelete, I, for one, am not suggesting any changes.
Obviously, people can add their own tools to work with their particular JSON formats, but I am trying to demonstrate that there is no reasonable approach to integrating more general JSON-related tools into the core. A superficial integration would be very slow because of the parse-manipulate-serialize cycle on every transaction. A deeper integration would require a radical restructuring of the core and a huge expansion of the operator space, with very little benefit to end users.
@TW_Tones seems to find my responses dismissive. I don’t accept that. I’m trying very hard to explain why I see the suggested expansions as a problem. I am certainly not trying to appeal to my TW expertise.
I reject that. Mario clearly has a deep knowledge of the core, alongside @saqimtiaz and @jeremyruston, and, I think, in a slightly different way, @EricShulman. I’m not a novice there, but I barely make it to intermediate.
I am, though, an advanced JS developer. I understand the capabilities of the language TW is written in, and have nearly three decades of experience with it. I’ve been trying to explain (and not simply asset) the fundamental difference between TW’s tiddler store, which is a functional database, and JSON, which is primarily a storage/transport format.
When @pmario says that a data tiddler is not a database, it’s like saying that sheet music doesn’t groove or swing or jive. It takes musicians “parsing” it to bring it to life. Yes, those who read JSON/music notation can get some sense of what the parsed information would be like, but it’s nothing like attending a concert.
Damn it, Titch, now you’ve got me humming everything from Hair!
if you are saying json is not going to work or should not be used don’t be surprised if the conversation strays.
@Scott_Sauyet I have valued your contribution but you recent responses suggest you are not hearing what I said, nore being generous in your interpretation of what I have said. it is irrelevant if you think you have not being dismissive if another feels you have. let us just try to listen a little more deeply.
Ok. I still feel like a new topic is a better idea, but if you want to discuss it here, then…
A solution to what? What fundamental problem are you trying to solve? I feel as though I gave a quite comprehensive answer to @Eskha’s original question, along with some additional information about the overall use of JSON.
You suggested a mission, which I’ve steadfastly refused to accept, to create an improved code pattern for using JSON. I’ve tried very hard to explain why I think this is a bad idea: it would make it easier for users to misuse the fundamentals of TW.
But if it’s not specifically JSON then I really don’t know what it is you’re looking for.
There is JSON both as an input and as a data structure, this Topic presumably is about its use as it is about a gap, ease of entry deletion. The General heading is now “JSON Tiddlers – Ease of Use” so presumably we can at least try and “ease its use”,
Now If I am to take @pmario and @Scott_Sauyet advice that JSON is not an appropriate tool to deal with “extra tiddler data”, as given and I have not argued otherwise;
I cant speak for everyone, but I am an observer of what/everything has being discussed in the last couple of months and it is clear better access and ease of use into JSON structures is being raised for a number of reasons;
Now these problems may be solved by alternate
Personally I am looking for the following as it stands;
I also expected it to help me “going forward”;
So for now I am assuming JSON is not suitable for these uses “going forward”, and the gap may be filled outside JSON, so how?
I have taken this no JSON as a premise and think perhaps if we could convert complex JSON structures into a familiar structure, based on tiddlers and then ensure we can use and navigate them, that way may be a compromise. I could not see a way forward until I revisited the JSON Mangler approach to FLATTENing JSON into compound keys, where the compound key could be like system tiddlers; $:/datasetname/topkey/subkey/parameter with the values if any stored in the text field (or the title)
$:/datasetname/ would remain the same
There could be multiple topkey/subkey/parameter values Including;
This approach may be able to handle the arbitrary structures people are presenting.
Way back in 2017, long before the JSON filter operators and the JSON Mangler, I created TiddlyTools/Widgets/action-unpackdata.js, which takes a tiddler containing JSON text, and converts it into a set of tiddlers as you’ve described, where the JSON “path” (aka “compound key”) is used as a tiddler title and the value is stored in the text field of that tiddler.
Note that it also handles XML formatted text via a built-in “convert XML to JSON” function, as well as custom Google Places API JSON data output retrieved via remote XMLHttpRequest and also TWCore-formatted tiddler definitions (i.e., plugins, themes, imports, etc).
It’s initial use was so that TiddlyTools/Panels/Location panel could submit latitude/longitude or address text queries to the Google Places API, and extract/process the returned JSON data to feed into the TiddlyTools/Panels/Maps and TiddlyTools/Panels/StreetView panels.
When the TWCore added the json filter operators, I added a “fallback” method that can perform the same “unpack” operation using just wikitext coding (see the getData_unpack() macro in TiddlyTools/Panels/Location).
One very interesting note about these two methods: the javascript-based $action-unpackData widget is more than 10 times faster than the wikitext-based code. For example, a typical response from the Google Places API might take 3 seconds to unpack using wikitext and json filter operators, but less than 0.3 seconds using the pure javascript widget coding.
-e
The main use case driving the introduction of the JSON filter operators was to support making HTTP requests from wikitext.Since JSON is one of the most common formats returned by HTTP APIs, we needed basic tools to read that data. Without those operators, the ability to make HTTP requests would have had very limited practical use. Because of this, most of the current operators focus on accessing and reading JSON.
The initial draft of the JSON operators worked with tiddlers instead of strings, so that the parsed JSON for each tiddler could be cached. However, this would have required that JSON first be stored in a tiddler before it could be read or manipulated. To avoid this requirement, the operators were refactored to work with strings, parsing them as JSON each time.
With the existing operators, you can parse JSON in any structure, read its contents from wikitext, and even convert it into tiddlers if you want. There are examples of this approach in the forum. Naturally, this does require a basic understanding of JSON, and I don’t see a realistic way around that. For the common JSON formats TiddlyWiki itself uses to store or export tiddlers, the deserialize[] filter operator gives access to deserializer modules, so you don’t need to work with raw JSON directly.
If you have workflows involving JSON that the current operators don’t support, please share them, ideally as short user stories or concrete examples. This will help us understand the real use cases, and whether any gaps exist in existing operators that have solutions that would be universal enough to include in the core.
Example hypothetical user stories:
“As a user fetching data from an external API, I want to extract just the title and address fields from a deeply nested JSON object so I can display it on a wiki page without writing custom JS.”
“As a plugin developer, I want to store configuration data as JSON and allow users to adjust individual values from wikitext, so that configuration remains both compact and accessible.”
Thanks @saqimtiaz for pencilling out the best way to look at enhancing the current operators. I do think in keeping focused on the title of this topic it is not only about the JSON operators as they stand. But what you said is important.
I think there would be value in further collating and providing guidance as we already have from a number of people including @pmario and @Scott_Sauyet.
But I also see room for code patterns, perhaps custom operators and also methods to move JSON models into tiddler structures where most users are more familiar. See @EricShulman and my suggestions. I think if we could “extract” a JSON into tiddler structures, and write them back to JSON as needed we may be able to avoid some complexity for the general audience.
I suppose you could say guiding and democratising access to data we may find in JSON forms.
It appears to me that since JSON seems to have an origin in JavaScript data manipulation and structures used there in, that it is not as independent of code as we may consider other structures, like dictionary and data tiddlers, and TiddlyWiki tiddler/field structures. If we consider structured JSON data in a tiddler to make use of it at present we need to establish code to iterate such structures and access the contents. So it is not so easy to give general users access to that data. It is harder to separate the data structures from the code needed to interrogate them, let alone update them. Using compound keys, nesting filters or building sophisticated filters are almost essential to make use of JSON as it stands.
I can foresee a few things we could do in code patterns and methods to make this easier for a more general audience, but as I am still coming to terms with JSON as used in TiddlyWiki, I won’t present any examples yet. I am still trying to classify the differences between TiddlyWiki native formats, arrays and hash maps, cases where we only have a numeric index etc…
In contrast TiddlyWiki has the TOC and Tree macros, the tag, tiddler title, and filter operators and fields to assist with more complex structures within tiddlers.
One thing that I find interesting is …
Q: Could you edit the tiddler JSON data store of one TW from another?
TT
This is why: Alternate version of "Notes on tiddlers stored in a data tiddler" (VERY long!). It was a long time coming, though.