what ever your pov on should some form of json be accessible / addressable for users / from with in ui.
more interesting (imho,…because: im sorry i haven’t a clue )
is why is json not easily accessible ?
which seams a little counterintuitive to me
(as i used to use tw before it was as complex / feature rich as it is now)
is JSON data … a second-class citizen in TW?
It isn’t.
TW merely objects to shorting the system.
how do you mean
like …
??!
“NO disassemble!!!”
json obviously!
I can only hazard a guess but from the top of my head - tiddlers are a flat structure of data with many known keys (and one, title, being required) wherein JSON can be infinitely large and there are entire huge libraries in existence for querying and handling it in various ways. There are infinitely many complex ways you can try to interact with it so it’s always going to be tougher to deal with it in TW.
It is available. There are JSON operators that make it easy enough to address parts of an object stored as a JSON string, either to get or set properties.
What we don’t have are ergonomic means to query data stored that way.
It’s easy to find all tiddlers with certain properties:
[tag[Book]]
<!-- or -->
[tag[Book]author[Neil Gaiman]]
<!-- or -->
[[tag[Book]] :filter[get[publication_year]compare:number:gte[2000]]
But we don’t have easy equivalents for these. We could write them in JS, like this:
JSON.parse(json_string)?.filter(({tags = []}) => tags.includes('Book'))
<!-- or -->
JSON.parse(json_string)?.filter(({tags = [], author}) => tags.includes('Book') && author=='Neil Gaiman'))
<!-- or -->
JSON.parse(json_string)?.filter(({tags = [], publication_year}) => tags.includes('Book') && publication_year >= 2000)
But we have no simple query mechanism in TW to do such things.
Why? I wasn’t paying attention during the discussions or decision, so my answer is just a guess. Take it with a grain of salt:
One of the mantras of TW is “everything’s a tiddler”. If you need a new configuration parameter, put it in a tiddler. If you want a new procedure/function/macro/widget, put it in a tiddler. If you want a temporary global value, put it in a tiddler. Etc.
This universality is very powerful. One of my favorite programming quotes is Alan Perlis’s, “It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures.” The reason is simple: you always know what you’re working with. This show in the way operators tend to transform lists of strings into other lists of strings; it’s a very powerful construct.
JS Objects, serialized from JSON strings, are a competing data structure. The ability to nest them, and the multiple types of values (strings, numbers, booleans, objects, arrays) makes for a more expressive data structure than TW’s main store, which is a Dictionary/HashMap of Dictionaries/HashMaps. There would be a strong case to make for a tool similar to TW, built on such a store.
But that is not how TW is designed; we can’t change that without massive backward compatibility problems. So the only other thing we could do, besides ignoring JSON, is to allow it as a source of data parallel to our main store, either as a single big object or as a number of little ones… We’ve done the latter, and done something similar with Dictionary tiddlers.
This seems a reasonable compromise, but it’s possible that it was a big mistake. (I believe I’ve read somewhere that @jeremyruston views it as one.) The trouble is that we now have to support multiple data types, and if we want sophisticated querying—similar to what we have for tiddlers in our various operators—we will have to write a lot of code to support that. If we want to have equivalent behavior to our tiddler store, we will have to carefully keep any changed synchronized between JSON and plain tiddler manipulation. This is a maintenance nightmare for any team, but particularly bad for the relatively small team of core coders.
And to @Maurycy’s point, to be complete, a robust querying API for plain JS Object stores would have to be huge.
Right. It is JaSON & the Argonauts all over again.
TT
javascript library’s ?
cant help pondering what is at the top of your list of huge libraries
in my mind at least due to it being a sub set of js
id have thought all that was needed would be part of js-lang
that said i guess a lot of (parser) effort is put into avoiding “eval”
and …
( ** never found a case that prefixing with “in” was not more appropriate )
parse and sanitise
a topic which seams to be wandering toward the territory of
GitHub - Engid/TDOPjs: TDOP parser in and for JavaScript or some subset of that or similar ( ftr i read that
)
More than 30kByte of minified JS code.
In the early days of TW5 there has been an external js parser, that did sanitise the TW core code.
We also would have had better control of the code itself. But for the amount of code it had to handle, it was so slow, that it was completely unusable. So in the end it was removed.
so a (historical) implementation exists in distant log/history
that might potentially be pluginizable ?
intresting
ahh ! ftr
i did not mean upper size limit
but the titles of the json-library’s !!!
Those examples look messy to me.
I and @EricShulman discussed this in our zoom meeting just closed. He has a better grasp of it. He showed me how to query JSON containing tiddlers, like plugins. However with one of more jsonextracts and jsonget at different depths it is not strait forward while still being logical.
However the above may be nice but it will focuse on a JSON full of tiddlers, and this may be a smart way to leverage JSON, just stick to JSON to represent virtual tiddlers, when building your own solution after all the support is there and you can turn it into a plugin or export it as a file with current tools.
However they, JSON can be structured quite different and includes different JSON structures such as an array and heirachical or flat depth.
I cant promise to be able to simplify this any further until I understand it more deeply, and I cant understand it more deeply until I can simplify this.
I worry that if we dont have the choice of unsanitised/unsanitary? core code, tiddley wiki looses is “open source” nature in a sanitised way. I think it should be an option, that can be done and undone if possible, then there is also the externalised core to consider.
Oh, and if you really want to know JSON I suggest you get to Know JSONMangler. Its in git hub but the demo is broken so I installed it here https://jsonmangler.tiddlyhost.com/
Huge might have been a bad choice of words since I was thinking about complexity and features, not necessarily the size. And I wasn’t just talking about JS, json is used all over the place.
Like see how huge the manual for jq is.
Or the many features of jsonata.
What I am getting at is that even with how simple tiddlers are (essentially a string-string dictionary) we have so many filters for tiddler-specific things. JSON structure is arbitrary and you have to deal with both arrays and objects. Naturally any solution you can come up with or chose will be more complicated and have more edge cases than what you need for tiddlers.
And given that tiddlers are first class citizens in TW world it, from my perspective, makes perfect sense why JSON is not treated with the same amount of care :).
Diclaimer: I haven’t done anything with JSON in TW so maybe things are not that bad; I am really addressing the hypothetical of “why json is treated worse than tiddlers”
That was part of my point. I probably could have simplified them a little by removing the initial JSON.parse call, since presumably if we’re using JS objects stored as JSON, we’ll do that conversion once up front (somehow).
But mainly, the JS API is much lower-level than the TW operator API, which means that it will generally take more code to handle the equivalents of our operators. That is why we’d need a new API for this. But it would be much harder to write than our operator API, which needs to support only a flat structure.
That is a much simpler construct to deal with. I don’t know how we’d support it, but at least I can imagine the possibilities. However, I didn’t think that’s really what people were talking about when discussing JSON support. I’m pretty certain a lot of people are talking about better, more ergonomic support for arbitrary JSON content, something like an operator api for nested objects. That’s what I’m arguing is far too much work.
So a “so called simple” example of a JSON query against a "JSON of formatted tiddlers:, as in plugins and import.
[{Demo}jsonextract[tiddlers]jsonextract[$:/config/TiddlyTools/AutoScroll]jsonget[description]]
In Prose:
Using the tiddler “Demo” and its text field, extract values below tiddlers, one of which is $:/config/TiddlyTools/AutoScroll and from there get the description(s) value.
Now let us define a function/custom filter operator
\function get.data(source tiddler item) [<source>get[text]jsonextract[tiddlers]jsonextract<tiddler>jsonget<item>]
So then the filter query at the top would now read;
{{{ [get.data[Demo],[tiddler],[item]] }}}
However structured and defined differently our code could be designed to look like this;
\function source.titles() [<source>jsonextract[tiddlers]]
\function get.item(item) [jsonextract<tiddler-record>jsonget<item>]
<$let source="Demo">
<$list filter="[source.titles[]sort[]]" variable=tiddler-record>
<!-- for each tiddler stored in the JSON -->
<<tiddler-record>> is described as {{{ [get.item[description]] }}}
</$list>
This example above shows how we can use operators to abstract away the use of jsonextract and jsonget filter operators, and that tiddlers within this JSON are nested under the tiddlers key.
\function source.titles() [<source>jsonextract[tiddlers]]
\function get. Item(item) [jsonextract<tiddler-record>jsonget<item>]
So Now I need to “collect other JSON formats we come across” and write new custom filter operators for each JSON structure, also actions procedures for writting to values.
i guess that is looking at the “problem” in isolation
as any potential solutions have to intertwingle with existing filter etc syntax
What were you doing?
Not something I have expierenced relating to JSONMangler
