How to write nested JSON structure in tiddler fields, like Array or Object

Developers
, , ,
#1

I’m letting my game numerical setup (game planning?) intern using tw to write game data JSON like the one similar to CDDA’s JSON
Or dwarf fortress’s token

Here is an example from CDDA:

  {
    "result": "hollow_cane",
    "type": "recipe",
    "activity_level": "LIGHT_EXERCISE",
    "category": "CC_WEAPON",
    "subcategory": "CSC_WEAPON_BASHING",
    "skill_used": "fabrication",
    "difficulty": 4,
    "time": "40 m",
    "autolearn": true,
    "using": [ [ "adhesive", 1 ] ],
    "proficiencies": [ { "proficiency": "prof_carving" }, { "proficiency": "prof_carpentry_basic" } ],
    "qualities": [ { "id": "HAMMER", "level": 1 }, { "id": "CUT", "level": 2 } ],
    "components": [ [ [ "2x4", 1 ] ], [ [ "nail", 5 ] ] ]
  }

See the components field, it is an array.

I’m now asking my intern to write a components field for tiddler “XXX” as components: [[XXX's components]],
and then in another tiddler “XXX’s components” write fields like "2x4": 1, "nail": 5.

Fields like qualities will be similar. Write qualities: Hammer1 Cut2 (tiddler title separated by space means list in tiddlywiki) in tiddler “XXX”,
and in tiddler “Hammer1” write fields like "id": "HAMMER", "level": 1

Do you have better solutions? Possible solutions I think can be:

  • using json path string like the one in lodash.get: a[0].b.c: xxx to express { a: [{ b: { c: 'xxx' } } ] }
  • write JSON in the text field, and set type to application/json, but my intern doesn’t know what “JSON” means. And I also want the input process to be simple.
#2

Point them to https://JSON.org to see the spec and tell them, that all browsers can deal with JSON out of the box now. So no 3rd party stuff needed for parsing and creating.

With the new json-related filter operators it is easier to read content from application/json tiddlers, but writing more than 1 level deep is not part of the TW core yet.

So plugins will be needed if you want to do so. … Or create your JSON tiddlers from the command line and import them to TW.

2 Likes
#3

Okay, I think I will have to remaintain my Smart Form plugin

And when editing nested JSON structure in a field, just store a JSON in that field. Later I can use jsonget operator for them.

What I concerned is, when I export tiddlers as JSON, these fields will become stringified JSON string, instead of object.

1 Like
#4

For security reasons.

#5
  • :frowning_face: this is a sad gap, I hope can it be added, ideally for the next release. Gaps like this can cause users waste hours trying to find the what should be a complementary feature only to discover its missing.
  • A similar example is allowing convert to Unix time but not from Unix time to TW serial date.
#6

See proposed addition (including code that works) here:

4 Likes
#7

I added a suggestion in the GitHub issue for consideration for a input date format to TiddlyWiki Serial Date or Unix Time stamp.

#8

This can already be done using https://tiddlytools.com/#TiddlyTools%2FTime%2FParseDate:

For any ISO 8601 or RFC2822 compliant date text:

To convert to a TWCore UTC Serial Date, use:

[[inputdate]parsedate[]]

To convert to a Unix Time stamp, use:

[[inputdate]unixtime[]]
or
[[inputdate]parsedate[unixtime]]

… and, to convert any signed integer value (i.e., a Unix Time stamp) to a formatted date, use:

[[inputtimestamp]unixtime[outputformat]]
or
[[inputtimestamp]parsedate:unixtime[outputformat]]
#9

So the argument goes, parsedate in the core,

or my proposed enhancement?

  • Which I would argue is simple an intuitive.
#10

I think the meta file can use yaml-like format to express nested structure. Currently, it seems already like a subset of yaml syntax.

If @jeremyruston also agree with this, I may be able to add this.

#11

What security problem we may have when exporting?

I thought the reason is that fields are all string by definition currently.

#12

Sorry but I do not understand this comment.

#13

Hi @linonetwo

Are you asking how to extract data from a JSON object into tiddler fields? The new JSON operators can accomplish that.

Or are you asking for techniques for generating JSON? That’s something I’d like to add to the core, but there are already a variety of techniques available. One simple approach is to treat them like we do stylesheets. For example:


\define payload-source()
\rules only transcludeinline transcludeblock filteredtranscludeinline filteredtranscludeblock
{
  "departure_searches": [
    {
      "id": "My first isochrone",
      "coords": {
        "lat": {{!!lat}},
        "lng": {{!!long}}
      },
      "departure_time": "2021-09-27T08:00:00Z",
      "travel_time": 5400,
      "transportation": {
        "type": "driving"
      }
    }
  ]
}
\end
...
<$action-setfield $tiddler=<title> text=<<payload-source>>/>
#14

And I think manual labor may produce JSON syntax error easily. This happened many time on the hand of professional game Mod authors like

That is why I ask my intern use builtin field form to input the value.

#15

Yes, and I mean using syntax like this in .tid file:

created: 20220528065611981
creator: LinOnetwo
modified: 20221121084558819
modifier: LinOnetwo
tags: AAA BBB
title: XXX
type: text/vnd.tiddlywiki
YAML Resources:
  YAML Specifications:
  - YAML 1.2:
    - Revision 1.2.2      # Oct 1, 2021 *New*
    - Revision 1.2.1      # Oct 1, 2009
    - Revision 1.2.0      # Jul 21, 2009
  - YAML 1.1
  - YAML 1.0

CC @TW_Tones

(example comes from https://yaml.org/)

I thought they were about manipulating JSON in the text field? So they can handle the whole tiddler as a JSON?

#16

Yes, I’m asking this too… Currently when I create a tiddler field with JSON content

截屏2022-12-16 18.09.48
截屏2022-12-16 18.09.481524×238 6.79 KB

It generates this in .tid file, which is fair enough I think, maybe we don’t need YAML syntax to express the nested JSON in .tid file

截屏2022-12-16 18.10.09
截屏2022-12-16 18.10.091254×202 6.9 KB

But when I try to get the whole tiddler as a JSON, this transportation field becomes a string instead of an object

get
get1058×366 22.7 KB

And my usage is to export the whole wiki as an array of game JSON data! So I hope when I getTiddler, the transportation field becomes a JSON object directly.

Maybe this has to be done by a custom serializer…

#17

The YAML library for JS is super heavy weight in terms of file size and IMO it’s as error prone to manually create as JSON. It replaces different types of braces with different type of indentation and other chars.

So in terms of manual errors IMO it’s the same amount of work to maintain

#18

Actually, I don’t mean manually editing yaml inside .tid file, I mean storing it that way automatically by the tw core. So when loading it, tw core knows it is a JSON object instead of a string with brackets.

So when I export tiddlers as JSON, it will become nested json, instead of currently a flat json with stringified value.
And I can provide a nested field value editor input for it.

#19

Unfortunately, the last time I looked, the smallest YAML parser I could find was about 80KB, so I don’t think that it is appropriate for the core. But there’s nothing to stop somebody making a plugin that adds support for YAML encoded data tiddlers. I’m pretty sure it can be done already, but if not I’d be happy to see core changes to make it happen.

There was an early version with those restrictions. By the time we merged them, the JSON operators operate on filter values, and so they can be used with variables or any other text source.

That’s correct. In order to be able to automatically parse them into JSON objects we’d have to have a way of knowing which fields contain JSON. I don’t think it’s a problem; there’s some inefficiency but we already suffer from that with the way that plugins are encoded.

#20

Maybe the jsontiddler Macro could help