Alternative Ways to Read JSON Data

[TW_Tones]

@EricShulman

The $transclude widget processing would first attempt to find a top-level key with the specified “d.e” key name (as it currently does)

One concern is that often we want to know when a key is NOT present, so we need to test for missing or blank values and return the result “Before splitting the key and looking deeper”. Perhaps a setting to allow or deny this would be wise. Whatever “synthetic key” is used to read needs to be returned as a literal key so it can be used.

I am not wishing to put a “kibosh” on this but just as TW5-Graph allows one to use the same data through multiple graphing engins we could also provide "a complete set of database query and write, operators and actions. which whilst they default to JSON can use another engin like SQL, IndexDB etc…

• Perhaps this need justifies an independant set of tools rather than trying to wedge them into existing ones, as clearly this seems to be a barrier to development.

Note:
I belive there are some more comprehencive JSON tools (I am looking for now)

Care of @Flibbles, one of his many impactful solutions.

See this TiddlyXML — XML features for Tiddlywiki

It refers to XML but as I understand it XML is a more structured way to handle what JSON does.

Features

• Query XML and HTML with XPath using the xpath widget and operator.
• Query XML and HTML with CSS selectors using the xselect widget and operator.
• XML tiddlers display correctly, and support XML processing instructions to render using a template tiddler.
• Allows for importing and exporting XML files.

See the demo website for documentation and examples.

[Scott_Sauyet]

@EricShulman

Still… it might be a nice TWCore enhancement if the syntax suggested by Gemini actually could produce results for nested JSON objects. What would be needed is a “fallback” approach:

One small wrinkle. We would have to decide if we let the choice of ambiguous values fall out of the implementation or have a stronger assertion.

For instance, in the following, what would be the value of a.b.c.d ?

{
  "a": {
    "b": {
      "c": {"d": 1}, 
      "c.d": 2
    },
    "b.c": {"d": 3},
    "b.c.d": 4
  },
  "a.b": {
    "c": {"d": 5},
    "c.d": 6
  },
  "a.b.c" : {"d": 7},
  "a.b.c.d": 8
}

There are eight choices. There seems to be few strong reasons to choose one over another. We won’t solve this by introducing a different separator, where perhaps a[b.c]d should return 4, since this is also a legal object:

{"a": {
  "[b": {
    ".c]": {
      "d": 42
    }
  },
  "b.c": 99
}

@TW_Tones

It refers to XML but as I understand it XML is a more structured way to handle what JSON does.

Not exactly. XML predates JSON. If anything JSON is more structured than XML, as it was designed for data structures, where XML was initially designed for loosely-structured documents, later adapted to data structures.

A (strict) version of HTML is a dialect of XML So while JSON is conceptually very close to JS, XML is closer is spirit to HTML.

This does not detract from your point that

@TW_Tones

we could also provide “a complete set of database query and write, operators and actions. which whilst they default to JSON can use another engin like SQL, IndexDB etc…”

That’s a very nice idea.

But I don’t think we could provide a single set of operators that each acted in the same way across such varied sources (presuming we want them to also act on lists of tiddlers as most operators currently do.) JSON and IndexDB might be close enough, but SQL is very different from both, as are tiddler-lists. It’s hard to imagine any clean interface that handles all of them.

But we could have separate operators for the different types. We already have a sketch of this in the JSON operators.

I don’t know whether this is a good idea or not. There would definitely be some added convenience. But there’s a good argument that even adding JSON is a step away from a nice purity of “everything is a tiddler”.

[pmario]

@Scott_Sauyet … The jsonget-operator has a “syntax” to retrieve values from a JSON file.

{{{ [[test.json]get[text]jsonget[a],[b],[c],[d]] }}}

{{{ [[test.json]get[text]jsonget[a],[b.c],[d]] }}}

@Scott_Sauyet

There are eight choices. There seems to be few strong reasons to choose one over another. We won’t solve this by introducing a different separator, where perhaps a[b.c]d should return 4, since this is also a legal object:

No it should not. The “[c.d]” is an index string and not an object reference.

[EricShulman]

ok… how about this:

When using $index="..." params

• if the param value does not contain square brackets, it is handled normally (i.e., as a top-level index key)
• if the param value uses the jsonget parameter syntax (i.e., terms within square brackets), it is handled using a jsonget filter

Applying these two simple rules, we get:

• <$transclude $tiddler="test.json" $index="a"/>
  returns a top-level index using key=“a”
• <$transclude $tiddler="test.json" $index="a.b"/>
  returns a top-level index using key=“a.b”
• <$transclude $tiddler="test.json" $index="[a.b]"/>
  uses jsonget and returns the value of a top-level JSON object named “a.b”
  (note that this effectively the same result as in the previous example)
• <$transclude $tiddler="test.json" $index="[a],[b]"/>
  uses jsonget and returns a nested value for “b” contained within JSON object “a”
• <$transclude $tiddler="test.json" $index="[a],[b.c],[d]"/>
  uses jsonget and returns a nested value for “d” contained within JSON object “b.c” within JSON object “a”

Also note that I am suggesting these rules should only be applied when using the “modern” $-prefixed param names. Thus, they won’t have any impact on “legacy” syntax that doesn’t use $-prefixed params.

-e

[Scott_Sauyet]

@pmario

The jsonget-operator has a “syntax” to retrieve values from a JSON file.

Right. This was a response to @EricShulman’s suggestion that we might also find convenient a syntax that uses dot-separated paths: [d.e] instead of (or as well as) [d],[e]. My point was simply that however we did that, we would have to recognize that there are legal JSON strings whose nested properties we couldn’t unambiguously address. The current syntax is unambiguous, and reasonably ergonomic, so long as our object property names do not contain square brackets. (That’s not impossible; it just has an awkwardness shared with our regex handling of square brackets.)

@pmario

@Scott_Sauyet

where perhaps a[b.c]d should return 4, since this is also a legal object:

No it should not. The “[c.d]” is an index string and not an object reference.

This was intended to dispense with an anticipated alternative syntax, where instead of plain dot-separated strings, we wrapped up any node keys that contained the dot in brackets, with the idea that we could capture the 42 in {"a": {"b.c": {"d": 42}}}} with a syntax like a[b.c]d or perhaps a.[b.c].d. The point is simply that this won’t work in a universal manner. You can always construct string keys that would confound your mechanism.

[wiki_user]

@TW_Tones

set of database query and write, operators and actions. which whilst they default to JSON can use another engin like SQL, IndexDB etc…

@Scott_Sauyet

But I don’t think we could provide a single set of operators that each acted in the same way across such varied sources (presuming we want them to also act on lists of tiddlers as most operators currently do.)

idk …but it sounds a bit like

Transforms every database into a object key value store. - ether/ueberDB

GitHub - ether/ueberDB: Abstract your databases, make datababies. Transforms...

Database Support
Couch
Dirty
Elasticsearch
Maria
memory: An in-memory ephemeral database.
Mongo
MsSQL
MySQL
Postgres (single connection and with connection pool)
Redis
Rethink
rustydb
SQLite
Surrealdb

[pmario]

@wiki_user … Stuff like this does not really make any sense for TW. It would add a 13MByte package to a tiddlywiki single file wiki, for no win. ./dist/index.js would have to be included.

[wiki_user]

i get that is not practical in standalone.html context

but also i think you get it is (or appears to be/if i understand the comment context correctly ) a
(simpler) implementation of a similar db-adapter concept

not practical in standalone context

makes me wander about what
usage metrics might be for such different cases
(though id guess that data is not practical to attain )

[TW_Tones]

I am talking about how to address the current store of tiddlers not changing how we store them.

[Scott_Sauyet]

@EricShulman

When using $index="..." params

• if the param value does not contain square brackets, it is handled normally (i.e., as a top-level index key)
• if the param value uses the jsonget parameter syntax (i.e., terms within square brackets), it is handled using a jsonget filter

I’m just realizing that I never responded to this. My point was that we can’t do this unambiguously, since any string is an acceptable JSON key. In the following [a],[b.c],[d] could reasonably have value 1, 2, 3, or 4.

{
    "a": {
        "b.c" : {
            "d":  1
         },
        "[b.c]" : {
            "d": 2
        },
        "[b.c],[": {
            "d]": 3
        },
        "... and many": "more"
    } ,
    "[a],[b.c]" {
        ",[d" : {
            "]": 4
        }
    },
    "etc., etc": "etc"
}

That’s not necessarily a show-stopper. If you have such crazy keys, you’ll have to explicitly specify them, and not use this new index mechanism.

Your proposed rules would unambiguously choose a value, if the path exists in the object. But there are a number of legal keys it could not handle.

It does get a little squirrely.

[EricShulman]

@Scott_Sauyet

there are a number of legal keys it could not handle.

Doesn’t this mean that the current jsonget parameter handling also has the same limitation?

My proposal simply makes the current jsonget reference handling syntax available for use in the $transclude widget, so instead of writing:

<$let value={{{ [[SomeJSONTiddler]jsonget[a],[b],[c],[d]] }}}><<value>></$let>

you can write:

<$transclude $tiddler="SomeJSONTiddler" $index="[a],[b],[c],[d]"/>

It also occurs to me that this same enhanced $index handling could also be useful in $action-setfield. Consider… to update a value in a JSON tiddler, you currently need to fetch a tiddler’s text field value, apply jsonset to that value, and then use $action-setfield to re-write the text field, like this:

<$let old={{{ [[SomeJSONTiddler]get[text]] }}}
      new={{{ [<old>jsonset[a],[b],[c],[d],<somevalue>] }}}>
<$action-setfield $tiddler="SomeJSONTiddler" text=<<new>>/>
</$let>

Even if you combine these steps in single widget, it’s still a somewhat awkward (and error prone) syntax:

<$action-setfield $tiddler="SomeJSONTiddler"
   text={{{ [[SomeJSONTiddler]get[text]jsonset[a],[b],[c],[d],<somevalue>] }}}/>

compared with:

<$action-setfield $tiddler="SomeJSONTiddler" $index="[a],[b],[c],[d]" $value=<<somevalue>>/>

-e

[Scott_Sauyet]

@EricShulman

@Scott_Sauyet

there are a number of legal keys it could not handle.

Doesn’t this mean that the current jsonget parameter handling also has the same limitation?

No. The issue is when you combine the nested keys into a single string. If you have separate keys for each level, there is no issue.

There is some complexity, which is similar to what happens in our regex handling. If your keys contain square brackets or other characters not allowed in a hard parameter, you would need to put them in a variable:

<$let key1="[b.c],[", key2="d]">
    {{{ [[SomeJSONTiddler]jsonget[a],<key1>,<key2>] }}}
</$let>

But we should be able to address any nested JSON properties using that. For the reasons I’ve already mentioned, any attempt to combine the keys into a single string will not allow for this universality.

That doesn’t mean that it’s a terrible idea. But we shouldn’t go into this without understanding the limitations.

[TW_Tones]

@EricShulman

Even if you combine these steps in single widget, it’s still a somewhat awkward (and error prone) syntax:

Even more so if the key is realistically driven by variables jsonset<a>,<b>,<c>,<d> This is a critisisum of mine about the Tiddlywiki documentation, most examples have literals not variables, fields or transclusions. Kind of an example how to go to the library and get a specific book out, but not teach you how to get any book in the library

[EricShulman]

@Scott_Sauyet

The issue is when you combine the nested keys into a single string.

Ah. But my proposal doesn’t “combine the nested keys into a single string”:

If the $index param doesn’t contain square brackets, it’s handled as a single top-level key. If it does contain square brackets, it’s handled the same as the jsonget filter parameters (i.e., “separate keys for each level”).

-e

[Scott_Sauyet]

@EricShulman

@Scott_Sauyet

The issue is when you combine the nested keys into a single string.

Ah. But my proposal doesn’t “combine the nested keys into a single string”:

I’m pretty sure it does, unless I’m drastically misunderstanding your proposal. You say, for instance:

@EricShulman

<$transclude $tiddler="test.json" $index="[a],[b.c],[d]"/>
<!--                                     ^^^^^^^^^^^^^^^  -->

That’s now a single string, instead of, say, the three strings "a", "b.c", and "d".

I’m almost certain that any way you choose to combine them into a single string won’t be able to cover all legal JSON.

For this example, how would you distinguish between the paths to 1 and 2 in this?:

{
    "a": {
        "b.c" : {
            "d":  1
         },
    } ,
    "a],[b.c" {
        "d" : 2
    }
}

Wouldn’t both of them be given by [a],[b.c],[d]?

---

I’m not suggesting that we don’t do this. But if we do, we need to be careful about documenting the sorts of key structure we can support. “Not abnormal”, for some definition of “abnormal”, I guess.

[TW_Tones]

Perhaps we can have an interving macro or custom widget or genesis widget to suppor the key structuring?

Such as

<$get-json-item L1keyname=<<var1>> L2keyname=<<var2>> L3keyname=<<var3>> >
and the result is also returned in here as <<L1value>>, <<L2value>>, <<L3value>>
 </$get-json-item>

and let that handle the composition of the key. In this example we may only want to use <<L3value>>

[EricShulman]

@EricShulman

[a],[b.c],[d]

I think we’re talking past each other…

Obviously, the syntax I am suggesting uses a single $index="..." parameter.

However, what I am proposing is that, if the $index parameter contains square brackets, it should be parsed using the same logic as the jsonget filter parameters, where it would be unambigously treated as three bracketed, comma-separated keys, "a" , "b.c" , and "d".

The example you’ve given that uses two keys: “[b.c],[” and “d]” requires extraordinary effort to bypass the normal jsonget operator’s “bracketed key” parsing by using variables. I would argue that this is clearly an “abnormal” use case that we don’t need to consider within the scope of this proposal.

-e

edit:
My proposal could be made to handle “abnormal” usage by changing the suggested parsing strategy just a little so that it doesn’t depend upon the use of square brackets to trigger the secondary parsing logic:

• First, attempt to use the entire $index parameter value to retrieve a top-level key (as it currently does).
• If that fails to return a result, then use the $index parameter value as if a jsonget filter operator were being processed.

Thus, it would permit the use of variables so you could write $index="<key1>,<key2>" to handle your odd example where the key values contain square brackets.

-e

[Scott_Sauyet]

@TW_Tones

<$get-json-item L1keyname=<<var1>> L2keyname=<<var2>> L3keyname=<<var3>> >

If you do that, you don’t run into the issue I’m describing, but you also don’t get the convenience that Eric is seeking. That is still a list of separate node names.

@EricShulman

The example you’ve given that uses two keys: “[b.c],[” and “d]” requires extraordinary effort to bypass the normal jsonget operator’s “bracketed key” parsing by using variables.

Yes, and it would get progressively sillier the more sophisticated the operator logic became. But I’m pretty certain that it would always remain possible to construct such ridiculous keys no matter our level of sophistication.

@EricShulman

I would argue that this is clearly an “abnormal” use case that we don’t need to consider within the scope of this proposal.

Yes, I can certainly live with that, should we decide to go this route, We’d simply need some reasonable documentation about the limitations. But I’m not sure this really adds all that much convenience to our current capability.

---

If you want to see a long, drawn-out argument about this, I participated in one fifteen years ago. I wouldn’t recommend it.