The thing is if you import a json file into a tiddler and it remains text you can look at it, there one can interrogate its format. I think its at this point some smart analysis of the JSON can present the users options, transfer between JSON formats and or trigger a reformat.
Only once you do this can you consider automating its import a particular way.
JSON data of the correct format is imported as tiddlers
Other formats can be interrogated with the various index and JSON operators.
I regularly point out JSON Mangler which is a suite of tools and I think operators and widgets to help handle JSON beyond the default support to import tiddlers.
There are some good threads on JSON in talk.tiddlywiki (try searching for “JSON formats”) that can help navigate them, because JSON is just a data format and the data can be represented many ways, perhaps we can call it a data transport protocol.
JSON Manger has a tool that can collapse a hierarchical JSON into a flat one with a compound key and back again. The flat one is a more logical one to import in many cases.
This is our challenge, if we don’t have tools that know generic JSON and provide import, interrogation and reformat tools it will be different for every piece of data you may wish to import depending on the source. We can never have perfect knowledge of how others and other systems will format this data.
In part for this reason I source csv files and use the JSON Mangler tools to generate a plugin of imported records as tiddlers, as I did for the Periodical table.
Exactly. There is still some trouble navigating to the first – or any – sample record, but once you’ve figured that out, I think it’s quite possible to show the user a reasonable interface for mapping the input structure to tiddler fields. I figure that we could offer to store this mapping mechanism itself in a tiddler (maybe a JSON one! ) to reuse in future imports.
I’m going to have to look again. When I got more seriously involved in TW, I looked at it for something I was doing. When it wasn’t able to handle what I wanted, I dropped it and never looked back. Part of this, of course, is that my own skills are quite tuned to doing data transformations in JS. But I will take a look.
No, but I think we can make a reasonable approach to the core problem: identifying and applying mappings from a fixed JSON structure to a collection of tiddlers.
Perhaps with this knowledge you can identify if any gaps exist in the tools available. We may only need a little putty, that tool to collapse and expand json structures goes a long way.
From my own exploration I identified a number of mechanisms and methods we could use, but i dont yet have a full understanding of json, and possibly will not until i can explore json in tiddlywiki. A bit of a paradox really. I just need help getting over the hump.
I may just add we should do our best to empower the community and let them explore the design space rather than setting standards or developing opinionated solutions. Standards will most likely emerge from usage.
Also JSON being a standard along side javascript means it is possible to use vibe coding (sorry about the jargon).
I do recall trying to work with JSON Mangler at one point years ago when I had a potential optional project that would need to parse a complex nested JSON as its primary data source. JSON Mangler is clearly a very powerful tool, and it felt like a pretty steep learning curve. (I got far enough to take a few steps into that project, which then lost priority for other reasons.)
Since my regular workflow often involves importing just a few tiddlers’-worth of data at a time, and these are otherwise simple and flat JSON arrays (just key-value pairs), I haven’t tried to navigate JSON Mangler again. But it’s absolutely fantastic to have it out there as an option.
) to reuse in future imports.