Storing data in an additional text area field within tiddlers

[TW_Tones]

Folks,

Background

Storing info related to a tiddler outside the tiddler

There are a number of ways to store additional information for a tiddler outside a tiddler, for example in a tiddler with a prefix to the current tiddler name `$:/data/tiddlername,

Storing info related to a tiddler inside the tiddler

Using the text field
Of course text fields in a tiddler could store a lot of information but tiddlers can get too long, and we tend not to mix to many different data structures.

Using fields
We can also store a lot of additional info in additional fields within a tiddler, filename1 fieldname2 etc… including using list fields.

• just as for tags this can overuse the fieldnames and “pollute the fieldnames” with too many names, just as having too many tags can.
• It is possible to create “additional” multiline text fields inside a tiddler if needed.

forground

Using a field to store data, an indexed data, JSON or data dictionary?
I would like to create an extra field when needed on a tiddler, however I would like it to store indexed data like for a data dictionary or JSON tiddler.

• In this case I would need to provide the tiddlername (or current) a fieldname AND in index.
• Typically data is only stored in tiddlers text fields with a type set, which are effectively “mime types”, but we cant assign a “mime type” for a field.
• The current filters and widgets do not seem to allow both fieldname and index to be used, they default to the text field.

This last option is a gap in the functionality available to us in tiddlywiki, that is to use other than the text field as a data store. If it were possible we could;

• Make larger self contained tiddlers, simplifying transfer to other wikis with drag and drop
• Be able to delete a tiddler and have additional data within the tiddler be deleted as well.
• Export a tiddler in a way that a history or log relating to that tiddler, is exported with it, because it is just another field.
• Use the text field to document the data stored within its additional fields.
• store data with a full qualification of tiddler/fieldname/index and allow one or all such values to be swapped for example but changing the tiddlername but using the same fieldname/index we could “switch datasets”.

The question?

Can anyone see a way to introduce or extend tiddlywiki to allow for data to be stored and retrieved from fields other than the text field, ideally using the existing tools or at least an easy to use workaround to enable this?

[CodaCoder]

I use the following as titles for meta/ancillary data:

• about>tiddlertitle
  • normally for image tiddlers
• tiddlertitle-notes
• tiddlertitle-info
• tiddlertitle-notices
• tiddlertitle-questions
• tiddlertitle-related
• tiddlertitle-stash
• tiddlertitle-overview

[TW_Tones]

@CodaCoder yes, thanks for your response, these are the in the first example

Storing info related to a tiddler outside the tiddler

And I to do something similar often, you can create ways to bundle this for easy transfer such as tagging them all with tiddlertitle and dragging the tagpill.

• I do understand this code pattern you shared, it is a helpful workaround
• I am specifically highlighting the specific case of making a field within a tiddler to store a data structure.
• In such a data structure I would like to store at least key/value pairs I can read and write, ideally multiple column rows

Storing data in an additional text area field within tiddlers

We could even use fields;

• tiddlertitle!!about
• tiddlertitle!!notes
• tiddlertitle!!info
• tiddlertitle!!notices
• tiddlertitle!!questions
• tiddlertitle!!related
• tiddlertitle!!stash
• tiddlertitle!!overview

The above fields can even be multi-line text fields, for example consider tiddlertitle!!questions

• This could contain multiple rows; yet we can’t use indexes to reference these and programmatically lookup, read or write such rows inside the field using

If however you try and do the following the $field is ignored

<$button>
<$action-setfield $tiddler="test" $field="data" $index=<<now YYYY0MM0DD0hh0mm0ss0XXX>> $value="detail"/>
Place detail in index
</$button>

Perhaps a simple use case;

Please this in a tiddler field eg tiddlertitle!!data

Name, Age, Gender, City
John, 25, Male, New York
Emily, 30, Female, London
Bob, 42, Male, San Francisco

*This can be done, but no supported way to lookup, insert, edit entries in the field

• This could be a minor extencion to core code to extend into this functionality.

[Charlie_Veniot]

@TW_Tones

The question?

Can anyone see a way to introduce or extend tiddlywiki to allow for data to be stored and retrieved from fields other than the text field, ideally using the existing tools or at least an easy to use workaround to enable this?

I have an implementation idea. I’ll try to prototype something in the next few days.

[TW_Tones]

Thanks @Charlie_Veniot, can you drop some keywords as a hint?

• I have just started considering the idea of having an open and close function that copies the content to a temp tiddler, allows actions against it, then allows a simple commit, that updates the fieldname, and a close that removes the temp tiddler.

[pmario]

@TW_Tones

The question?

Can anyone see a way to introduce or extend tiddlywiki to allow for data to be stored and retrieved from fields other than the text field, ideally using the existing tools or at least an easy to use workaround to enable this?

I think it would be straight forward to implement a very basic function as described for the usecase lined out in the OP. … But it would result in a “tiddler in a tiddler” situation as stated at:

@TW_Tones

forground

Using a field to store data, an indexed data, JSON or data dictionary?

IMO the increased complexity and the flexibility downsides are not worth it. … The current core code to read and write data is “battle hardened” and “well optimized”.

Dealing with a “tiddler in a tiddler” situation will increase complexity for no real win. All the problems the existing data-dictionary tiddlers have will be extended to the “main tiddler fields”, which makes the situation worse.

So linking to a separated data-tiddler form the main tiddler works out of the box today.

---

We did brainstorm about a similar idea at GitHub Discussions in 2021. I think Jeremy’s summary at: How could system-fields look like with 5.2.x is still valid.

There are some more of my thoughts about 1 year later while thinking about a completely new page-template from scratch, which is OT here

[TW_Tones]

@pmario thanks for your response, I hope the following explains why I think this is more important, and easier than you think.

@pmario

tiddler in a tiddler

• We can already have tiddlers within tiddlers, like a tiddler containing JSON or a plugin. But that is not what I want, actually its simply allowing me to do with another field what I can currently do with the text field.
• Throughout tiddlywiki we can treat the content of any field the same way and search, filter, transform etc… with the exception of data tstructures, this includes JSON and Data dictionary etc… which rely on the text and type fields.

@pmario

increased complexity and the flexibility downsides are not worth it

• This needs to be demonstrated in my view that it is in fact complex.
• I would suggest complexity also depends on the requirements I am putting forward and the methods needed.

@pmario

We did brainstorm about a similar idea at GitHub Discussions in 2021

• I recall those conversations and I was involved for a number of reasons, but these conversations had different initial motivations.

The Tiddler as an object

• title, created and modified and the type fields are effectively metadata fields for tiddlers and it helps present the content to the user in a default manner.
• It makes sense that the text field is nominated as a primary field containing content and we construct various handling and presentation defaults. Mime types editions etc…
  • The text field is the default content displayed and thus it needs view templates, cascades and mime types
• But this is not a reason to limit the data structure functionality to the text field only, and not allow other fields to be used to store other structured data.
  • These other structured data fields do not need the same full treatment we give the text field.
  • We already allow additional fields to be a list of titles like the tags field, why not allow additional fields to have a structured data as well?

The Tiddler fields

• Users and designers are free to add additional fields and with some exceptions, beyond Mear text fields, we have to and are free to manage the field handling and type.
  • With a little tweaking its quite easy to set up a multi-line field.

Structured Data in a tiddlers

• If we are trying to maintain structured data anywhere we need a set of tools including index references etc… presently these can only reference the text field.
• There is no reason why we could not apply these to fieldnames other than text, and like all current fields leave it to the designer to handle types.
  • At worst we may need to delimit structured data so its compatible with the JSON store.

@pmario

So linking to a separated data-tiddler form the main tiddler works out of the box today.

• To me this is at odds with the philosophy of tiddlers, the smallest unit of information is not the tiddler, if you need more than one to store information pertaining to one tiddler.

Fields can be related to the tiddler, the whole tiddler and nothing but the tiddler.

• Until we can access other fields to store structured data, without being forced to create additional tiddlers, we can not meet the data normalisation view of tiddlers.

Practical examples.

• Time stamped log against a tiddler, saved in the tiddler
• storing changes to the tiddler as they occur match/patch
• ability to access data by key stored within a tiddler eg a todo list - without polluting the fieldname space.
  • or needing to structure the text field into subsections.

A Simple view

• Just allow the $field parameter to work along side the $index parameter to allow us to act against other fields. If necessary also provide an effective “type” value so the commands know what mime type belongs to that fields content.

[Charlie_Veniot]

How about a rough prototype instead?

KeyValuePairs_in_TiddlerField.json (1.2 KB)

[TW_Tones]

Nice examples @Charlie_Veniot make your own multi-variable data fields.

Thanks for the contribution.

[Charlie_Veniot]

You ain’t seen nothing yet.

That’s just proof in the pudding concept of the foundational pieces. Much easier to test/evaluate the foundational pieces when one can see them and then poke at the pieces with well-placed action-confirm widgets as the poking tool.

Pending macros: keySet, keyGet, keyDelete, keysList

[TW_Tones]

I will share my additional text are fields tool soon, perhaps a hybrid?

[Charlie_Veniot]

keySet and keyGet macros done.

keyDelete and keyList macros next time.

KeyValuePairs_in_TiddlerField.json (2.8 KB)

Three tiddlers in there:

• Key-Value Data for Fields (beginnings of usage notes)
• Key-Value Macros (the two macros)
• Test Key-Value Macros (test/try tiddler)

The test/try tiddler has two tests for both keySet and keyGet:

• one set of tests to put key-value pairs in a “data” field of tiddler “aaa”
• one set of tests to put key-value pairs in a “greetings-data” field of the current tiddler.

You won’t notice without digging: setting a key-value pair for some tiddler (say “A”) will generate a dedicated temporary tiddler ("$:/temp/A") for tiddler “A”. A dedicated temporary tiddler will be generated for every tiddler which as key-value work done to it.

[Charlie_Veniot]

keyDelete macro done.

KeyValuePairs_in_TiddlerField.json (4.2 KB)

[Charlie_Veniot]

Added the “keyList” macro, intended for use as per this example:

<$list filter={{{ [<keyList "tiddler" "field">] }}}>
...
</$list>

KeyValuePairs_in_TiddlerField.json (6.3 KB)

Keeping in mind that I am developing/testing this with TW 5.2.3, you’ll find some trickery going on in the code to make up for the lack of goodies from 5.2.4 onward.

In particular, I use some unicode characters to make filtering a little easier.

• Keys get book-ended by: ⯈ and
• And values get book-ended by: and