How can I get the indexes of a data tiddler in the order they are given there?

[JanJo]

…obviously when retreiving them by

<$list filter="[$:/data/Fehleranalyse]indexes[]]">{{!!title}}<br></$list>

they are sorted by title.

Best whishes Jan

[TW_Tones]

@JanJo I would have expected what you are expecting as well, it is implied here https://tiddlywiki.com/#indexes%20Operator%20(Examples) the palet is a dictionary tiddler, but its already in title order.

What Kind of data tiddler is yours?. There is another way to list the “dictionary tiddler” keys.

Like other data tables they are possibly sorted internally to help with key lookups but in this case you would expect it need not sort first.

[jeremyruston]

@JanJo

they are sorted by title.

Behind the scenes, the name/value pairs in a dictionary tiddler are represented as a JSON object. JSON objects do not maintain a guaranteed ordering and so we can’t reliably retain the ordering of the data in the file.

[TW_Tones]

Perhaps @JanJo could use a full JSON data tiddler and include an additional sort filed, perhaps an incremental number or time date stamp so he can get what he is after?

It should be relatively easy to generate the replacement data tiddler from the existing dictionary tiddler, but then he will also need to update the code where the former dictionary tiddler is referenced.

• in this case the tiddler title $:/data/Fehleranalyse suggests it is custom data and this should be possible.

[JanJo]

@jeremyruston

Behind the scenes, the name/value pairs in a dictionary tiddler are represented as a JSON object. JSON objects do not maintain a guaranteed ordering and so we can’t reliably retain the ordering of the data in the file.

Thank you for the info - I thought I was missing something in the syntax. Anyhow it would be nice to have the possibility to refer to the order. In this usecase for example I wanted to build a table with checkboxes. It seems that a simple list is more practical then.

[TiddlyTweeter]

@jeremyruston

Behind the scenes, the name/value pairs in a dictionary tiddler are represented as a JSON object. JSON objects do not maintain a guaranteed ordering and so we can’t reliably retain the ordering of the data in the file.

Thanks for that info! Very useful. I did think I must be doing something wrong that I couldn’t get dictionary entries to list in my entered order

@JanJo

Thank you for the info - I thought I was missing something in the syntax. Anyhow it would be nice to have the possibility to refer to the order. In this usecase for example I wanted to build a table with checkboxes. It seems that a simple list is more practical then.

I’m wondering, now, if there is a way (trick) to a “hidden” way to add to the indices such that the JSON ordering is forced appropriately?

I tried adding redundant <spans> to a DD but it messes up wikification in the calling macro …

Just out of interest I tried, and failed, with fake starting spans …

Just a comment, TT

[joshuafontany]

You have to hold that data in another structure (a list field or array JSON object), and then update it when the parent-data changes. A little rough, but doable, if you figure out which user interactions need to change the list.

A great example of this is how TW handles the TAGS list. A preferred sort order (alpha by title) allows building the list virtually, but on drag/drop a list field is created on the parent tag tiddler. The list logic also uses “list-before” and “list-after” fields to perform some smart sorting, even when the main “list” field is BLANK.

[JanJo]

Hi Joshua,
that seems to be a smart solution.
May you post a json or demo?

[TW_Tones]

If I may suggest “Depending on physical order” when you have a strong desire to make use of that order in different ways, is more fragile than adding (one or more) sort keys to every record. Whilst you can use “compound keys” in dictionary tiddlers this is unnecessarily complex and it is worth moving to a more suitable data layout, with multiple fields pre row, perhaps even individual tiddlers.

Alternatively you can make use of another index to a data tiddler, as raised by @joshuafontany where the index is the list field of a tag, basically it is a list of the same tiddlers in another order. But you could build other types of indexes. Also the sortby operator is your friend here.

I am yet to research it, but since the way we retrieve indexes and values is similar, if not identical with both dictionary and json data tiddlers it may be possible to replace one with the other with little or no code changes, just a batch transformation process.

• Well worth the research because we then have an upgrade/downgrade option for any data tiddler including ones in the core.
  • This is useful if we want to add more related information within the one data tiddler.

[JanJo]

@TW_Tones

it is worth moving to a more suitable data layout, with multiple fields pre row, perhaps even individual tiddlers.

I would like to have this…but is there a solution for multi value data in TW?

[TiddlyTweeter]

@TW_Tones

If I may suggest “Depending on physical order” when you have a strong desire to make use of that order in different ways, is more fragile than adding (one or more) sort keys to every record.

That may be true in a big design perspective.

Just FYI I only use DD’s when it is elegant.
Why would I want to maintain 10 Tiddlers when one is all that is needed?

Just a comment
TT

[TiddlyTweeter]

@JanJo

I would like to have this…but is there a solution for multi value data in TW?

We should ask @pmario??

I think he made a tool to expand the DD format to handle keys well?

Just a comment. I’m not sure.
TT

[pmario]

TLDR;
As described later, there may be a possibility to control the order of the output reliably, if data-tiddlers follow some rules and modern browsers are used. So no IE

Since the default behaviour of the indexes filter operator can not be changed in a backwards compatible way, we would need a new operator or may be a new “operator suffix” like

The new suffix should be backwards compatible.

[$:/data/Fehleranalyse]indexes:raw[]]
or
[$:/data/Fehleranalyse]indexes:chronological[]]

@TiddlyTweeter

@JanJo

I would like to have this…but is there a solution for multi value data in TW?

Sadly, there is no “out of the box” solution at the moment.

@TiddlyTweeter

We should ask @pmario??

I think he made a tool to expand the DD format to handle keys well?

My keyvalues plugin won’t help here, since it also uses the core function, that Jeremy mentioned in his post.

@jeremyruston

Behind the scenes, the name/value pairs in a dictionary tiddler are represented as a JSON object. …

“Behind the scenes” here means — The browser javascript engine. So not TW data, that we can control, but how the data is organized internally by the browser vendor. … So the “problem” is “deep down in the rabbit hole”. …

@jeremyruston

… JSON objects do not maintain a guaranteed ordering and so we can’t reliably retain the ordering of the data in the file.

The wording here is a bit vague. “the ordering of the data in the file” … actually can be controlled, because that’s the text the user enters. …

We also can control the order how the text is converted to a JSON object. … We read a data-tiddler “line by line” and store it in the object. So this order is chronological …

The problems starts, when we want to “convert” the data in an “ordered” way form the JSON object back to a “string” using object.keys(data). Our filters use “space separated strings”.

A bit of history: As the development of TW was started the JavaScript specification ES5 (from 2009) was active. The specification of the object.keys() function left some room for interpretation. So the different browser vendors did implement the “output” that function produces differently. That’s the reason, why “we can not reliably reproduce the ordering of the data, created by object.keys()”.

BUT … As I did investigate the details for my post here I did find some interesting infos

Important: The ordering is still a bit strange, but it should be predictable and consistent with different browsers. (If I understand it right)

… The ES6 spec defines that Object.keys also relies on ownPropertyKeys which makes it predictable in today’s browsers, too.

ES6 was defined in 2016 and all major browser should already use it and go with the new specs.

---

The strange behaviour looks like this:

A data tiddler with the following content

2: true
1: true
00: true
b: true
a: true
3: true

Would return:

[ "1", "2", "3", "00", "b", "a" ]

So as long as the “keys” don’t look like integer values, the order should be chronological.

eg:

b: true
a: true
c: true

will return

[ "b", "a", "c"]

as expected

More details can be found if you click me :)

Property order is predictable in JavaScript objects since ES2015

and

The Order of Javascript Object Keys

[TiddlyTweeter]

@pmario

TLDR;
As described later, there may be a possibility to control the order of the output reliably, if data-tiddlers follow some rules and modern browsers are used

Thanks very much for taking the time to respond! Much appreciated. Clarifying.

TT

[JanJo]

I would like to give you five likes for this!

@pmario

we would need a new operator or may be a new “operator suffix” like

The new suffix should be backwards compatible.

[$:/data/Fehleranalyse]indexes:raw[]]
or
[$:/data/Fehleranalyse]indexes:chronological[]]

@pmario, when do you think you get this running?

[TW_Tones]

As I think it suggested earlier, if you want you can access a dictionary tiddler without using indexes but parse as a text file;

<$set name=keys-in-order-found filter="[{data-tiddler}splitregexp[\n]split[:]first[]]">

Then you can “enlist” <<keys-in-order-found>> to process via indexes in the physical order.

The above is untested but I am confident it is close;

• The only issue may be if writing to an index then causes the data tiddler to become sorted.

Also I do wonder if the need for physical order indicates the selected key to the data is the wrong one.

[TW_Tones]

@TiddlyTweeter

Just FYI I only use DD’s when it is elegant.
Why would I want to maintain 10 Tiddlers when one is all that is needed?

My approach too but fields are just as useful, especially since the introduction of the JSON store fieldname can be almost any value.

[edited] but of course this can potentially “pollute the field namespace” and can happen with the overuse of tags “pollute the tags namespace”. But I expect it depends on if you want to reuse the keys multiple times, or have multiple “data sets”. One offs are perhaps best in a data tiddler.

[TW_Tones]

@JanJo

but is there a solution for multi value data in TW?

The core does this (tiddlers in JSON store, history list) and the way data tiddlers is implemented does not make easy to see how to within wikitext/macros and widgets.

• I am investigating this now in Native tiddlywiki.
• I will start other Topic(s) and link in this reply

[JanJo]

Hi @TW_Tones

@TW_Tones

<$set name=keys-in-order-found filter="[{data-tiddler}splitregexp[\n]split[:]first[]]">

…then “enlist” <<keys-in-order-found>>

this sounds like a great idea. I’d love to see a working example.

[TW_Tones]

Can you share the data set?