Alternative Ways to Read JSON Data

One concern is that often we want to know when a key is NOT present, so we need to test for missing or blank values and return the result “Before splitting the key and looking deeper”. Perhaps a setting to allow or deny this would be wise. Whatever “synthetic key” is used to read needs to be returned as a literal key so it can be used.

I am not wishing to put a “kibosh” on this but just as TW5-Graph allows one to use the same data through multiple graphing engins we could also provide "a complete set of database query and write, operators and actions. which whilst they default to JSON can use another engin like SQL, IndexDB etc…

  • Perhaps this need justifies an independant set of tools rather than trying to wedge them into existing ones, as clearly this seems to be a barrier to development.

Note:
I belive there are some more comprehencive JSON tools (I am looking for now)

Care of @Flibbles, one of his many impactful solutions.

See this TiddlyXML — XML features for Tiddlywiki

It refers to XML but as I understand it XML is a more structured way to handle what JSON does.

Features

  • Query XML and HTML with XPath using the xpath widget and operator.
  • Query XML and HTML with CSS selectors using the xselect widget and operator.
  • XML tiddlers display correctly, and support XML processing instructions to render using a template tiddler.
  • Allows for importing and exporting XML files.

See the demo website for documentation and examples.

One small wrinkle. We would have to decide if we let the choice of ambiguous values fall out of the implementation or have a stronger assertion.

For instance, in the following, what would be the value of a.b.c.d ?

{
  "a": {
    "b": {
      "c": {"d": 1}, 
      "c.d": 2
    },
    "b.c": {"d": 3},
    "b.c.d": 4
  },
  "a.b": {
    "c": {"d": 5},
    "c.d": 6
  },
  "a.b.c" : {"d": 7},
  "a.b.c.d": 8
}

There are eight choices. There seems to be few strong reasons to choose one over another. We won’t solve this by introducing a different separator, where perhaps a[b.c]d should return 4, since this is also a legal object:

{"a": {
  "[b": {
    ".c]": {
      "d": 42
    }
  },
  "b.c": 99
}

Not exactly. XML predates JSON. If anything JSON is more structured than XML, as it was designed for data structures, where XML was initially designed for loosely-structured documents, later adapted to data structures.

A (strict) version of HTML is a dialect of XML So while JSON is conceptually very close to JS, XML is closer is spirit to HTML.

This does not detract from your point that

That’s a very nice idea.

But I don’t think we could provide a single set of operators that each acted in the same way across such varied sources (presuming we want them to also act on lists of tiddlers as most operators currently do.) JSON and IndexDB might be close enough, but SQL is very different from both, as are tiddler-lists. It’s hard to imagine any clean interface that handles all of them.

But we could have separate operators for the different types. We already have a sketch of this in the JSON operators.

I don’t know whether this is a good idea or not. There would definitely be some added convenience. But there’s a good argument that even adding JSON is a step away from a nice purity of “everything is a tiddler”.

@Scott_Sauyet … The jsonget-operator has a “syntax” to retrieve values from a JSON file.

{{{ [[test.json]get[text]jsonget[a],[b],[c],[d]] }}}

{{{ [[test.json]get[text]jsonget[a],[b.c],[d]] }}}

No it should not. The “[c.d]” is an index string and not an object reference.

ok… how about this:

When using $index="..." params

  • if the param value does not contain square brackets, it is handled normally (i.e., as a top-level index key)
  • if the param value uses the jsonget parameter syntax (i.e., terms within square brackets), it is handled using a jsonget filter

Applying these two simple rules, we get:

  • <$transclude $tiddler="test.json" $index="a"/>
    returns a top-level index using key=“a”
  • <$transclude $tiddler="test.json" $index="a.b"/>
    returns a top-level index using key=“a.b”
  • <$transclude $tiddler="test.json" $index="[a.b]"/>
    uses jsonget and returns the value of a top-level JSON object named “a.b”
    (note that this effectively the same result as in the previous example)
  • <$transclude $tiddler="test.json" $index="[a],[b]"/>
    uses jsonget and returns a nested value for “b” contained within JSON object “a”
  • <$transclude $tiddler="test.json" $index="[a],[b.c],[d]"/>
    uses jsonget and returns a nested value for “d” contained within JSON object “b.c” within JSON object “a”

Also note that I am suggesting these rules should only be applied when using the “modern” $-prefixed param names. Thus, they won’t have any impact on “legacy” syntax that doesn’t use $-prefixed params.

-e

Right. This was a response to @EricShulman’s suggestion that we might also find convenient a syntax that uses dot-separated paths: [d.e] instead of (or as well as) [d],[e]. My point was simply that however we did that, we would have to recognize that there are legal JSON strings whose nested properties we couldn’t unambiguously address. The current syntax is unambiguous, and reasonably ergonomic, so long as our object property names do not contain square brackets. (That’s not impossible; it just has an awkwardness shared with our regex handling of square brackets.)

This was intended to dispense with an anticipated alternative syntax, where instead of plain dot-separated strings, we wrapped up any node keys that contained the dot in brackets, with the idea that we could capture the 42 in {"a": {"b.c": {"d": 42}}}} with a syntax like a[b.c]d or perhaps a.[b.c].d. The point is simply that this won’t work in a universal manner. You can always construct string keys that would confound your mechanism.

idk …but it sounds a bit like
:safety_pin: :eyes: :left_speech_bubble:
Transforms every database into a object key value store. - ether/ueberDB

Database Support
Couch
Dirty
Elasticsearch
Maria
memory: An in-memory ephemeral database.
Mongo
MsSQL
MySQL
Postgres (single connection and with connection pool)
Redis
Rethink
rustydb
SQLite
Surrealdb

@wiki_user … Stuff like this does not really make any sense for TW. It would add a 13MByte package to a tiddlywiki single file wiki, for no win. ./dist/index.js would have to be included.

i get that is not practical in standalone.html context

but also i think you get it is (or appears to be/if i understand the comment context correctly ) a
(simpler) implementation of a similar db-adapter concept

not practical in standalone context

makes me wander about what
usage metrics might be for such different cases
(though id guess that data is not practical to attain )

I am talking about how to address the current store of tiddlers not changing how we store them.

I’m just realizing that I never responded to this. My point was that we can’t do this unambiguously, since any string is an acceptable JSON key. In the following [a],[b.c],[d] could reasonably have value 1, 2, 3, or 4.

{
    "a": {
        "b.c" : {
            "d":  1
         },
        "[b.c]" : {
            "d": 2
        },
        "[b.c],[": {
            "d]": 3
        },
        "... and many": "more"
    } ,
    "[a],[b.c]" {
        ",[d" : {
            "]": 4
        }
    },
    "etc., etc": "etc"
}

That’s not necessarily a show-stopper. If you have such crazy keys, you’ll have to explicitly specify them, and not use this new index mechanism.

Your proposed rules would unambiguously choose a value, if the path exists in the object. But there are a number of legal keys it could not handle.

It does get a little squirrely.

Doesn’t this mean that the current jsonget parameter handling also has the same limitation?

My proposal simply makes the current jsonget reference handling syntax available for use in the $transclude widget, so instead of writing:

<$let value={{{ [[SomeJSONTiddler]jsonget[a],[b],[c],[d]] }}}><<value>></$let>

you can write:

<$transclude $tiddler="SomeJSONTiddler" $index="[a],[b],[c],[d]"/>

It also occurs to me that this same enhanced $index handling could also be useful in $action-setfield. Consider… to update a value in a JSON tiddler, you currently need to fetch a tiddler’s text field value, apply jsonset to that value, and then use $action-setfield to re-write the text field, like this:

<$let old={{{ [[SomeJSONTiddler]get[text]] }}}
      new={{{ [<old>jsonset[a],[b],[c],[d],<somevalue>] }}}>
<$action-setfield $tiddler="SomeJSONTiddler" text=<<new>>/>
</$let>

Even if you combine these steps in single widget, it’s still a somewhat awkward (and error prone) syntax:

<$action-setfield $tiddler="SomeJSONTiddler"
   text={{{ [[SomeJSONTiddler]get[text]jsonset[a],[b],[c],[d],<somevalue>] }}}/>

compared with:

<$action-setfield $tiddler="SomeJSONTiddler" $index="[a],[b],[c],[d]" $value=<<somevalue>>/>

-e

1 Like