A question about .tid file format

#1

I exported a couple of random tiddlers to .tid format and looked at those files. It looks like the fields (all but text, which is at the bottom of the file) are sorted alphabetically by name.

Yet, https://tiddlywiki.com/#TiddlerFiles does not elaborate on the order of fields in the file.

A third “source of truth” on this matter would be the TiddlyWiki source code repository, but it is too big and scary for me to study, hence the question.

Is the order of fields in a .tid file relevant or not? Is TiddlyWiki able to import a .tid file with fields sorted in arbitrary order? If yes, then why do the fields seem to be sorted in exported .tid files, is it just a side effect?

#2

Yes, in many ways it does not matter the field order, only that the text field with its multiline content is clear.

We have moved the tiddlers in a single file wiki into a JSON format, look inside the html file to see them. Node also does this now. So tid files are more of a legacy :nerd_face:

Personally I always export tiddlers in a bundle and using the JSON export, for sharing etc… The JSON can have any meaningful name and contain one or more tiddlers.

  • I have played with the tiddler ordering but its safer not to depend on it because fields can be added by different parties.
  • Core fields especialy have special handling once in tiddlywiki but to my knowledge we have not influence over this, but export could be sorting them first.
#3

I would not call them a legacy and I would be very surprised if at some point they will be entirely deprecated.

Sure JSON is a cool format, very popular overall, and especially in the JavaScript ecosystem. But it’s a computer friendly format, while .tid is human friendly. A big tiddler with a multiline text field looks like a mess, with lots of escaped things, it’s not readable at all.

Think about the need to write a tiddler externally, outside of TiddlyWiki, for further importing it. Will you write in in JSON format? I doubt it. .tid is the clear winner here.

3 Likes
#4

That’s exactly what I have done, and still have some more todo (and didn’t even give thought to the order of fields). Specifically so far: importing notes from MacOS “Stickies”, Linux “sticky notes” and extracting from legacy tw2 html files. (the two “stickies” were plain text, but close enough to markdown for my personal style that sometimes I think I used that format. Remaining on my todo is converting from tomboy notes’ format - which is xml.

The tid format is also very close to (I think I read somewhere that it was directly based on, but maybe I assumed it) the Internet Message Format (perhaps better known as rfc822 / rfc2822 / rfc5322, or simply plain text email format), and based on that similarity I have a basic proof of concept script which converts a directory of .tid files (as provided by a multifile node TW5 setup) into a Maildir, which can then be read by an imap server and thus by any email client. Only of use for tid files which are based on human readable text, but interesting nonetheless to me and could be a door to other features down the track.

1 Like
#5

Perhaps Legacy means something different to me, it has a history, although depricated in most places, and this does not mean it does not have its place.

  • I even wrote a preview that shows the tiddler in tid format, with a copy to clipboard for sharing here, but never used it :frowning_face: as I prefer code format and json bundles (so far)

No, with all due respect I use the best tiddler editor on the market, the tiddlywiki platform :nerd_face:

Dont get me wrong I empathise with people wanting to handle single file tiddlers, use standard editors anfd the command line, go for it. Even the .tid format has its place.

  • I do think if this were your focus perhaps we could even develop another format with even more features. For example by giving the editor access to tags, and titles in the target wiki, at least the system tags, support for common braces such as [ { < and more.

By the way if I source data or content from other systems I like direct import to the tiddlywiki platform where I can build batch and parsing solutions primariy with tiddlywiki script. Each converter I have the more power and ability to adapt to other formats I get. The logic becomes part of my wiki.

The platform provides rapid paths to;

  • Preview or viewer formats
  • Syntax highlighting
  • Alternate editors
  • Organising within logical structures
  • Taging/flaging/cateloguing
#6

His is quite a hot take, the TiddlyWiki platform is not the best editor on the market. It is not backwards compatible with older hardware that can not run a modern browser or Node.js. Yes, such hardware still exists and is functional. And can be used for writing .tid files by using a text editor. And searching through them using simple tools like grep. Then the .tid files can be imported into real wikis for further use.

#7

Arguably it is for tiddlers.

Each to their own, and by the way TiddlyWiki makes this easy. Use what ever format or editor you want, on whatever platform you need.

#8

No, as long as the text field if any comes last and is separated by a blank line from other fields.

Yes, apart from the caveat about the text field mentioned above.

Tid files are generated using a template that sorts the fields, since internally tiddler fields have no specific order, see https://tiddlywiki.com/#%24%3A%2Fcore%2Ftemplates%2Ftid-tiddler:%24%3A%2Fcore%2Ftemplates%2Ftid-tiddler%20FieldsWidget

The tid file format is neither a legacy format nor deprecated. TiddlyWiki on node.js still uses tid files as the default storage format for non-binary tiddlers, and uses JSON only when the tid file format cannot handle the tiddler data (most commonly when there are mutliline fields other than the text field) or unless otherwise configured to do so.

2 Likes
#9

I completed the little quest :+1:

While the template you mention does not contain explicit sorting, it uses the <$fields> widget, which has this detail documented in https://tiddlywiki.com/#FieldsWidget

#10

The main problem here is the internal wiki store, which is an “object” store. For objects there is a mechanism how browser vendors store variables. But there may be differences in implementation. So the TW core can not really guarantee, that the order of adding fields can be retained. The simplest way to make it consistent between OSes and browsers, was to sort stuff before returning it to users.

That’s the main reason, why most of the stuff eg: tags are sorted alphabetically, even if users store them in a different order.

With new Javascript functions and data-types, that are consistent between browsers, we are able able to change that. But it will take time and convincing Jeremy. He prefers alphabetically sorted lists over “chaotic” ones :wink:

So time will tell.

#11

The only caveat being, as I understand it, that single file wikis do only use the json format to store tiddlers in the file.

#12

Yes, that’s true, but remember what JSON is at its heart: a serialization of an Object into a string, using a restricted version of JavaScript’s object literal syntax. While we can say that TW uses that format to store its tiddlers in the HTML, we might just as easily note that these JavaScript objects are simply stored in a JavaScript syntax. Any other choice would have meant a much more difficult startup process involving custom parsing of the tiddlers.

#13

I am coming to some interesting conclusions when thinking about the question more.

In an abstract scenario, as in a tiddler being just a collection of key:value mappings, the order of fields in a serialization format like .tid, doesn’t matter indeed.

The fact that fields are sorted in .tid files is actually useful for humans - a bit easier to visually look up things.

In a parsing scenario, the title field seems to be more “important” than the rest, since title is the key field for tiddlers when looking up in a collection of tiddlers. If one wants to parse a string that contains the serialized data that gets dumped into a .tid file, the input data is just a string. If one would want to produce useful parsing error messages (as in “in which tiddler did the parsing problem happen”), one needs to know the tiddler title, but for this, the string has to be pre-parsed to find it (because the tiddler title is not in the first line, and is not guaranteed to be in the last line before the empty line delimiter, after which text field follows). Does this mean that in this situation the tiddler data has to be parsed twice, or is there a better way?

#14

I think (possibly naively) that in practice, parsing errors are not going to be that big a deal.

In the case of a single-file TW, each tid is a single line json object, so it’ll either the line (and thus the entire tid) will parse, or it’ll fail. The field order wont make a difference.

In a multi-file node TW, each tid is a file, so if the file fails to parse, then in theory (maybe practice? I couldn’t say) that could be reported as an error to the node console and report the filename at fault.

So in either case, the order of fields isn’t much of a deal.

(tangent: for human-readability of .tid files, I’d argue that title should be first, and perhaps a few other very standard/default fields (created, modified, tags, type) also be at the top. But in practice, so long as it can read the tid files I occasionally create with fields in an arbitrary order, I don’t care what it does when it rewrites them and sorts the fields.

#15

That may be true for some humans. Others would prefer a position based order. I personally want the

  • title-field to be the very first one.
  • tags and list should be next, if they exist
  • created and creator
  • modified and modifier
  • any other field in the order I defined it in a configuration

With that mechanism position matters. For example: mom, dad and siblings fields belong together, because positioning them in a group matters (for me).

Forced alphabetic sorting, destroys this positional relation – So it depends on the usecase and the users preference.

Alphabetical sorting in the TW core, was partially forced by ES5 JavaScript object store behaviour in browsers and by Jeremy’s preference for alphabetical sorting.

The .tid is a file format. So if there is a problem with parsing we can use the file-name to tell the user where the problem comes from. So no parsing at all is needed for an error message.

If the .tid format is wrong it will be imported in a wrong way silently.

#16

the format reminds me of

generic-message = start-line
*(message-header CRLF)
CRLF
[ message-body ]