[Proposal] Updating field handling functionality in TiddlyWiki

Developers
, ,
#15

This is my preference for the idea. Having a field name be a tid feels like a powerful way to make connections. I use the ability for any tid to tag any other tid as a way to organically grow structure, and feel the same would be true for fields. I dont really see tids that are tag names as “tag tids” they are just tids. Similarly, I wouldn’t expect I’d see “field tids” as, well, “field tids”, they’d just be tids. From what you’ve written, I’d just expect them to be tids with a few optional extra fields that control how the equivalent field works, but none of them required (eg: field-type: which might have “list” (as the “type” field has now), or “color” (as the “color” field has now), each resulting in a change of the field entry UI to suit, but without a field-type the default would be “short text”). But the same name being used as a normal tid is also possible, and therein lies the potential for organic and emergenct structure.

That’s not to say there isn’t a place for a $:/fields/ style namespace, but I would assume from the style of name, that it’d ne roughly analagous to the $:/tags/ namespace - in that it holds the system ones. But it’s an imperfect analogy since field names as shown/saved don’t have this prefix. (but maybe they could have a field caption field or something to hold the name as displayed?)

Relevant (I hope) tangent: I know the tid format was modeled after HTTP headers, and they were explicitly modeled after RFC822 - the text message format which has evolved since (RFC5322) mainly on the demands of email, and that evolution originally had a mechanism to ensure official headers were separate from user-defined ones (by promising no official header would begin X-), but later updates didn’t even mention it - the distinction between “official” and “user-defined” became irrelevant. I mention all this because in a way, I feel it’s relevant to here too - a tid can be as small as title: x …and other fields are optional. This is part of why I feel the distinction between system and user-defined doesn’t need to be super struct.

The rfc822 message format also evolved into handling the other thing I read in that issue 5805 thread - handling multiple multi-line data. Email solved it by mime attachment, which in the vast majority of cases is just a simple attachment, but at it’s core effectively defines a mechanism by which a full nested tree of attachments can exist (including other full messages). So on the one hand, the problem has been solved, but it’s been solved in a way that is effectively the issue that Jeremy raised, in that it turns one field into a full sub-tid!

(final note: I’m not suggesting tid format follow rfc5322 format - it’s already diverged in a few keys way, though the continued similarity it has made it convenient for the tid->Maildir proof of concept script I’ve drafted up to give me IMAP read-only access to the raw content!)

#16

It would impact search-ability as well as ease of creation/filtering, though. When I want to figure out how the core (or a third-party wiki) accomplishes something, I generally go to Advanced Search first — and similarly, it’s useful to be able to filter for certain types of system tiddlers in my own wikis. Introducing unicode characters like ▭ adds a lot of (IMO, unnecessary) friction. I couldn’t type that on the fly even if I wanted to; I had to copy-paste it from your post.

I understand and appreciate your point about developing tools to manage tiddlers of this type; I think it’s important to provide GUI interfaces whenever possible (as, in fact, the core does with the field editor). At the same time, I’d say I use the core field editor for less than 10% of the changes I make to fields; most of my work is done with $action-setfield and $edit-text. Now, I’m sure this isn’t at all representative of most TW users… but I do think it’s probably a more common workflow among the power-users and developers who would presumably be making and sharing field templates the most. So I think it’s important not to make it too onerous to search, filter for, or even modify these tiddlers, even if “standard” users will never interact with anything beyond the default GUI.

I think that’s one possible definition. I’m not sure whether it’s quite broad enough for your purposes… but on the other hand, if a given field isn’t displayed in the field editor, you probably don’t need a custom template for it, anyway. And conversely, if a user has deliberately modified the display state of a given system field, it’s reasonable to think they might want a custom display, too. So on reflection, it might work as-is.

If you wanted a more complete list of fields used by the core, I think you could omit the :filter run and simply use

[all[shadows]removeprefix[$:/language/Docs/Fields/]]

This could conceivably catch some plugin shadows, too… I’m not sure whether anyone is actually using the $:/language/Docs/Fields/ namespace in their plugins.

If you don’t want to catch any non-core tiddlers, you could use this instead:

[[$:/core]plugintiddlers[]removeprefix[$:/language/Docs/Fields/]]
#17

I understand where you’re coming from, and generally I’d agree; this is how I use tag tiddlers in my own wikis as well. My concern is that one of @TW_Tones’s goals is specifically to make it easier for users to share configurations… and if I have content in the “author” tiddler, and you have your “author” tiddler set up for a certain field configuration, I can’t import your “author” without losing my own content. Instead, I’d have to open it and manually copy all the relevant fields into my own version of the tiddler.

That’s not too difficult if there are only a few extra fields, of course, though I imagine it could get tedious on mobile. But I think the fact that you can typically “copy” a configuration just by dragging some plugins/system tiddlers — without altering any of your own content tiddlers — is a major selling point of TW. I think it’s risky not to follow that precedent — not to mention potentially confusing for users.

#18

Thanks for the feedback. Yes I see value in this kind of transparency fieldname=tiddlername and do think it could be a good source of innovation. As a result I think it importiant to at least allow this to be the case even if ultimatly its not the default. To clarify example approaches, they currently include

  • fieldname = name of field tiddler “fieldname”
  • fieldname = name of field tiddler “▭ fieldname”
  • fieldname = name of field tiddler “$:/fields/fieldname”
  • fieldname = name of field tiddler specified in the fiddler in which its used using a filter eg: [<fieldname>addprefix[#:/]]

However @nemo one example where this may not be helpful is an area we have not explored much yet and that is using fieldnames based on existing tiddler names, independant of field handling. For example the Tiddler “Tony Smith” may have a field named “Cathy Smith” with the value “sister”. Because there is a tiddler called “Cathy Smith”. You can see here we can search for all tiddlers containing “Cathy Smith” to discover all relationships. I will have a closer look at your references to standards in your reply.

  • So in this case there will be a lot of fieldnames, arguably up to the number of tiddlers, and we need not define every field, but could.
  • We can see here we may want to differentiate “field definition” tiddlers using a “namespace”.
  • Actualy not really. Consider the use of ▭ name it is easy to search for any full or partial string in name the difference is in the search results you will see ▭ name in the list and know what you are looking at. You may also see name which is “just a tiddler” not behaving as a field definition tiddler.
  • Similarly to tags tiddlers is[tag]you can use is.field[] later is[field]. also with all operator.

Arguably this unicode prefix method is more powerful and easier to search than tag tiddlers, and there is otherways not indicator in the search drop down for tag tiddlers/ tiddlers that are also tags.

I get what you are saying here, but I think you may be the exception. This solution should support your approach by providing an implied standard way to untroduce field edit code to replace the default field editor. especialy for the suit of different input types, ranges and validation.

This seems like a good definition or method to identify “core system fields”, and allows us to stil leverage the $:/config/EditTemplateFields/Visibility/ mechanisium :+1:

My uncertiantly now exists around language changes. Are these tiddlers simply overidden by other languages? Observing $:/language/Docs/Fields/name contains “The human readable name associated with a plugin tiddler” which is a system field, but users may very well define differently.

Some further thoughts

Thanks @nemo and @etardiff for your continuing feedback. I see here how I/we need to juggle two factors, one is the value of standards and guidence, the other innovation and creativity.

standards and guidence

For want of a better term “field handling” is already something anyone can do in tiddlywiki by making their own rules. I have gone down this path a number of times, and find lots of nessasary details bog down development of field handling, and tends to make the solution more and more bespoke and not nessasarily keeping things open for intergration with other solutions, plugins and tiddlers.

This is why I have ventured into this fairly complex project because if we can establish a robust yet flexible solution for frield handling on which users/designers can build advanced field handling we provide suggested “Standards and guidence” and enable cross solution intergration.

#19

Just for clarity, I think this could be a cascade “looking for the field definition tiddler”.

  • ordered bottom up
#20

Also on lurking over here Check for empty fields with a "generated" field name I need to add to the requirments a way to group fields into sets.

Even if a wiki built by a single user there will need to be logical groupings of field tiddlers, we could call these “forms” or “tiddley types” eg all the fields that belong to a “contact tiddler”, I am tempted to recomend tags eg each field definition that belongs to a “contact tiddler” would have the tag “contact tiddler” or “contact-fields”.

  • You can see here how field handling starts to quickly bleed into other “nomenclatures” like field-types, tiddler types, forms etc…
1 Like
#21

My point was that it would be significantly more difficult to search for ▭ — a character I can’t type or even Google efficiently without copy-pasting — or for [prefix[▭]] or similar filter constructions I might use if I was trying to list all the tiddlers of this type. $:/fields (or any of your other alternatives) is conversely very easy.

Personally I’m happy to agree to disagree about this. But if you’re hoping to ultimately get some or all of this work into the core, I just don’t see it being adopted if it relies on unicode titles, so it seems prudent to design with that in mind.

#22

Comments accepted.

Although just as we have filters to find tags I think we can have filters to find field tiddlers, field definitions., with or without a unicode character.

  • I would like make an arbitary prefix possible, if not a unicode one.
  • Similarly I would like to be able to add a new system namespace eg with arbitary prefix, so If I used ▭ then they remain hidden like $:/ tiddlers.

Now I need to find a pithy name for “field definition tiddler”, then build some filter operators to find them eg: [all[fields]] or [is[field]]. [all[field.definitions]] or [is[field.definition]].

  • perhaps “definition” is too general, and needs the field qualification?
  • of course suplimenting the current fields, has, get…

You can use these operators rather than the prefix to find field definitions just as you do when searching for tags with or without tiddlers system or otherwise [tags[]] etc…

#23

Notes on my Design strategy

I am trying to bring together knowledge from my own and others use of tiddlywiki from over more than a decade. In this case speficialy to handle fields, field-types and may extend into “tiddler-types” and forms, Hopefully this will converge on a good solution.

However: As an expierenced designer I know it is importiant to keep the initial design development phase open as long as practical, not to become locked into a particular path. I am always looking for solutions that;

  • Avoid forced compromises
  • Leverage existing standards or philosopies
  • Are as intuitive as possible

In this case I feel there may be more to be discovered here!

so I continue to encorage ideas here.

Note on missing tiddlers;

I have come to realise that the current missing tiddler mechanisium is less than optimal. The $:/language/MissingTiddler/Hint is only invoked at the lowest point in the existing cascads from within the “view template body”, making it hard to ultimatly have different “missing tiddler types”. Relavent here is “missing field (definition) tiddlers”, but the same could be said also for missing tag tiddlers, or missing alt-tag tiddlers.

  • I suspect a new cascade would make sense here to determin by filter which missing tiddler hint to use.
  • Realise this is a somewhat advanced subject that I know perhaps only myself, @etardiff and @Springer have focused on of late.
  • I am now going to study the Info > Advanced > Cascades to look into introducing additional ones for field definitions and missing tiddlers.
#24

I was going to say I’ve never had this problem… and then I remembered that I’d altered my cascade to avoid it. Here are the simplest-possible changes I’d recommend:

  1. Edit $:/config/ViewTemplateBodyFilters/default to the following:
[has[title]then[$:/core/ui/ViewTemplate/body/default]]

This preserves the default view template for extant tiddlers.

  1. Create a new cascade tiddler, $:/config/ViewTemplateBodyFilters/missing (or your preferred title). This will become our new final fallback for missing tiddlers.
title: $:/config/ViewTemplateBodyFilters/missing
tags: $:/tags/ViewTemplateBodyFilter
text: [is[missing]then[$:/core/ui/ViewTemplate/body/default]]
list-after:

list-after ensures this will always be the last item in the cascade, and again, it applies $:/core/ui/ViewTemplate/body/default. Assuming you haven’t edited that default template, it contains

<$transclude>

<$transclude tiddler="$:/language/MissingTiddler/Hint"/>

</$transclude>

And there’s nothing to transclude in a missing tiddler, of course, so you’ll always be transcluding $:/language/MissingTiddler/Hint. (Conversely, our earlier invocation of $:/core/ui/ViewTemplate/body/default never transcludes the hint, because the tiddler always exists.)

  1. Between $:/config/ViewTemplateBodyFilters/default and $:/config/ViewTemplateBodyFilters/missing, add as many cascade tiddlers (each defining their own ViewTemplates) as you like. You will only see the default missing-tiddler hint if none of the intervening cascade filters apply.

For instance, I have a “missing-only” cascade filter that looks like this:

[<currentTiddler>minlength[3]listed[author]limit[1]then[AuthorTemplate]]

This applies a programmatically generated “author” template to any missing tiddlers that appear in at least one author field. The minlength[3] isn’t strictly necessary; I added it so I could avoid the presumably more costly listed step when a title is undoubtedly too short to qualify.

That’s the basic overview. Please feel free to adapt this if it’s useful to you.

#25

My first type of solution (back when I first starting leveraging nodes that weren’t tiddlers) was actually to hijack the $:/language/MissingTiddler/Hint shadow, overriding it with a portal to all sorts of virtual-node offerings. What we’d see there, in the story river, was conditional on whether we seemed to be at a tag node, a fieldname node, (and/)or an old-fashioned obsolete hard link.

I think what @etardiff is suggesting — a whole cascade for non-tiddler nodes in the story river — sounds more semantically appropriate than stuffing conditional logic, and substantive template transclusions, into the missing tiddler hint! :wink:

Recently I’ve been just vacating that $:/language/MissingTiddler/Hint shadow — since it’s just visual clutter for me to encounter that reminder about how to create a tiddler. Most nodes are already useful when open, for reasons that have nothing to do with wanting to create a tiddler.

In a solution bundled for others, I think I’d restore the default missing tiddler hint (click edit-button to create…), but have it show up only when there’s no relevant automatic-template content at the node. Perhaps for nodes that do have relevant automatic templates, perhaps it still should appear as a reminder, but only on hovering near the title-bar, or something like that.

Currently my overview of what I call the virtual-node-approach mostly relies on cascade conditions (within the usual body-cascade). Beyond the old-fashioned “missing/lost links”, I focus on '‘three’ interesting kinds of non-tiddler nodes :

  • tags
  • fieldnames
  • filter expressions

These may have somewhat different needs, actually. Whatever view template we want for empty-node tagnames may well also want to show up by default on tag nodes that are tiddlers. The same may be true of fieldnames (maybe I want a template that automatically summarizes field-connections there, whether or not a tiddler has been created at that node to hold directly-entered content, such a description of the field’s purpose).

Filter expressions, of course, are a totally different kind of node, and a bit of a niche interest… and for these I even recommend a modification to the toolbar, since it’s essential that tiddlers not be created for such nodes (since filter expressions feature square brackets).

#26

That’s a good point, and my first thought was that the ‘Export Tiddler’ menu item could (optionally) go to a more complex export dialog, where fields could be selected/excluded from the export - and then from there could be dragged with just the specific data being exported.

My second thought was that on import, the same thing - if there is a conflict between the import data and existing data, that it prompt for which fields to keep from which side of the conflict.

On reflection, both of those are arguably good things to have in general, and whilst solving this problem, are not specific to here.

I think that’s a good point!

I expect this is the sort of thing I’d end up using a bit, though I dont have examples to mind just yet. Your example however doesn’t resonate with me, since my instinct would be to have the field name be the more generic thing (“sister”) with value “Cathy Smith” (indeed, values as a list since there may be multiple sisters. I do see the simplicity and value of having ‘Cathy Smith’ as the field name though as a discovery mechanism!

I wonder if another solution to the field-information/tid-information overlap described above, is an “excise” mechanism to convert the field values from a “fieldname” tid to a dedicated “▭ fieldname” or “$:/fields/fieldname” tid?

#27

With apologies for the slight digression, @TW_Tones, and in hopes that this is still at least somewhat useful to you…

I think this is an important point! I use some similar ViewTemplates for tags and other field-based connections, but I’ve found it easier to apply those with the $:/tags/ViewTemplate rather than the ViewTemplateBody cascade. That way, if I decide to add some text content to a previously-empty node, I won’t lose the visual summary of its (pre)existing connections. A “missing” tiddler may thus display a custom “body” template, or a ViewTemplate summary segment, or both, or neither, if I’ve never referenced it before at all.

I don’t use backlinks[] much myself — I’ve found it can get slow in large wikis, and many of my links get stored in non-text fields anyway — but I can imagine it being useful in a ViewTemplate segment or even a missing tiddler template, particularly if it’s a node that might merit its own content.

This is a fascinating application, and not one that had ever occurred to me. I’m not sure I have any immediate need for it, but I think I felt my mind expanding a bit, just seeing it.

Do you find that it replaces standard filter search for you? I was going to suggest adding a link from the search results to the relevant node, but I see you’ve already done that. :clap:

1 Like
#28

Exactly my approach. Nice to know your adventure here.

Exactly. So is this a story tiddler cascade to control buttons, or a View Template Body cascade for just the content?

I may have to put filter and extended “virtual tiddlers” aside for now in this project. Although I stand to be corrected. This may be a good place to discuss Create and Use of Virtual Tiddlers

I am not exactly sure what you mean by excise. Please tell me more. But just so you know I have a number of ideas here, the one I like the most is hiding this in the existing UI.

For example this is how I am crafting it;
2025-08-13_16-06-24

The fieldname is tied into a link to a missing tiddler, clicking on it creates the missing field tiddler “▭ New field” (as an example).

  • In this case the prefix indicates it is a field definition tiddler.
  • Perhaps the way to define the field is to set the field-type field on the tiddler (rather than edit to create).

This was just a quick grasp at an example to illustrate, and although not compelling on its own. I think in the fullness of time examples that are compelling may be found. Although this is where its nice to handover a capacity to do something and let others explore the “innovation space”. Sister may be a bad example, it should be sibling with the Cathy Smith tiddler providing information about the “personal pronouns”.

  • My suspicion is such use cases will emerge when it comes to bilateral relationships". For example when the relationship is different going in a different direction.

Actualy backlinks[] links[] and now transcludes[] are operators that may be able to shed light on what a “missing tiddler” is about. A fieldname could be (placed) in the list field of a tiddler “my fields” to actualy define it as a field definition tiddler.

@Springer I assume a virtual tiddler, define witrh a filter as title can

  1. List the results of the filter and
  2. Have a button to create a filter tiddler, as available in Advanced Search > Filter > Dropdown
#29

The TWCore currently has 14 shadow tiddlers containing pre-defined filters. Each shadow tiddler is tagged with $:/tags/Filter, and contains a filter field (the saved filter syntax) and a description field (the text to show in the $:/AdvancedSearch > Filter dropdown list).

To make it easy to add filters to the dropdown list, add TiddlyTools/Search/SaveFilter to your TiddlyWiki. This tiddler is tagged with $:/tags/AdvancedSearch/FilterButton, which adds it to the other buttons that appear next to the $:/AdvancedSearch > Filter input field.

When you press the save filter button, it displays a popup for you to enter a tiddler name, edit the current search filter syntax, and add description text. By default, filters are saved with a prefix of $:/config/TiddlyTools/Filter/..., but you can edit TiddlyTools/Search/SaveFilter to change the config() definition to any other location you prefer.

enjoy,
-e

1 Like
#30

Perhaps “excise” is not quite the right term here (esp since it has a meaning within normal tid editing). Anyway, idea is that if a tid exists named “New field”, and recognised as a field definition tid via it’s own “field-type” field (and perhaps others?), then just the bits that define it as a field definition tid could be elevated out to “▭ New field” style - the new one having the fields that are relevant to it being a field definition tid copied over, and the normal tid fields remaining with the original. (Whether the field-type (and others) fields are removed from the original, or remain (but ignored because of the precedence of the new field definition tid), is a matter of lesser importance. I’d be inclined to leave them be, and those fields within the ‘New Field’ tid effectively become “shadow field definitions” - they exist but are unused.

That’s my feeling on the matter too. Tid’s being able to define fields to multiple other tids via field definitions, feels like it’ll enrich the interconnectedness and organisational possibilities of TW.

#31

Thanks @EricShulman I seem to have lost my nicely crafter responce. I think this could be a button on missing tiddlers that have a title equal to a filter as per @Springer’s filter titles idea.

Be aware I have two cats that keep tring to stand in front of my screens, keyboard or on my lap. So dont be supprised if some typos slip in. Or worse as my spellchecker is not working in my app window to talk.tiddlywiki.

@nemo see Virually merging two tiddlers? Triddler stacks and form of monkey patch which is complemetary and somwhat related to your approach perhaps?

#32

My habit is to hit the keyboard shortcut for advanced search, and use the filter tab. Then if I realize I’d like the big overview afforded by a dynamic table (rather than a mere list of links), I hit the button to open the filter expression as a node in the story river. I can keep the node open as long as I want during a session, but it never becomes a real tiddler.

The other place I’ve tried to embed the “open filter expression as node in story river” option is in the filter-pill dropdowns by @TW_Tones. It works great for one-off filters, but still needs more development to pass the granular filter to the tm-navigate widget, if the filter-pill is itself embedded in a list widget.

#33

Off Topic: Just as we have the ability to find transclusions and back links thoughout tiddlywiki including text fields perhaps an operator that can find and return filters as delimited by [ ] but also if starting with :runprefix and extending between " quotes " including ' and now back ticks.

  • I expect relink has already solved this.
#34

I will just note there are two other smart approaches to field handling I do want to facilitate or at least not hinder as a result of improved field handing these are;

  • wild card based handling
  • protocols in the field value

wild card handling is where you may use a filter to invoke custom handling such as with a fieldname prefix or suffix eg all fields ending in ‘-link’ or prefix ‘name-’

by protocols in field values is when the fields value names a protocol the most common examples are the url protocols such as http https mail to etc…

  • that is field handling is driven by something in the value rather than the name of the field
  • this can free up the field names to be flexible and thus descriptive

the issue becomes determining precedence

Also note @pmario addresses many of the issues about implementation for fields in the view template HowTo: Use Custom and Complex ViewTemplates (Tagging and Cascade)

  • Of particular note to me is his use of the info tab