[Proposal] Updating field handling functionality in TiddlyWiki

Developers
, ,
#1

As recently triggered here Macros listing fields with spaces in their names - #11 by TW_Tones I would like to propose “Updating field handling functionality in TiddlyWiki”.

  • I have researched this at length over the years and have plenty of ideas and methods to share but raise this topic to include those with the appropriate expertise.
  • If implemented correctly this will add functionality but not demand any changes unless a user wants to leverage this functionality.
  • The following is a full and systematicaly developed “requirements document” so please forgive me for its verbosity.

To keep this simple I will list the features I would like to see and simply assume the “use case” is obvious to readers rather than justify the requirements. I happy to address that later or elsewhere if contested;

  • Initialy introduce as a plugin then propose migration to the core once mature.

Features

Develop a system that allows “Field Tiddlers” missing or otherwise. Similar to tag tiddlers.

  • Provision the following filter operators. If this is crowding “is” and “all” perhaps allfields and isfield?
    • [all[fields] alternativly use “fieldnames”
    • [is[field] alternativly use “fieldname”
    • I expect we can already create custom filter operators such as all.fields[] and is.field[] but I would like to see this prepared for inclusion in the core one day.
    • Another approach may be to expand the functionality of the current fields[] operator
  • Make indexed filter operators for fieldnames (if not already) see Performance
    • Also use in the “fields Operator”
  • Provision list/filters of registered fieldnames for include or exclude as needed, see TiddlerFields perhaps intergrate with an operator and indexing eg;
    • standard system fields
    • core fieldnames
    • System/Plugin and user fieldnames
    • Find a way to systematicaly non-distructivly add/remove from such lists.
    • We may wish to extend the way we currently determin if a field is a User or System field including a system field prefix (less vferbose than $:/… or $:/plugin/pluginname/…) generic and custom (eg pluginname_fieldname)
    • I am raising this now because I belive we should implement it before the next major release if possible.

Cascade features

  • Introduce a “Field View” Cascade that can use a view template for that field, within or referenced from a “fieldtiddler” if it exists.
  • Include in the existing “Field Editor” cascade, that can use an edit template for that field, within or referenced from a “field tiddler” if it exists.
    • If we have field tiddlers it makes sense to consolodate various field handling features in a field tiddler. It would then be possible to share such field tiddlers to share the definition with others. It would also show when someone attempted to import a new field definition if one already existed (similar behaviour exists for tags already).
  • Introduce a widget or two that allows the above edit or view code to use used on a nominated fieldname via the define cascade (or alternative).

Notes on naming

There are variouse opinions as to wether it is permissible to name fields with plane english words or enforce prefixes. This should remain posible and at the behest of developers and designers not enforced by the core.

  • I have made a number of solutions using plain english words for field names without issues and would not be happy if a fieldname prefix was enforced. That is not the case with existing system fieldnames so this should be maintained.

Subsequent features

Once the above becomes mature we could include “field tiddlers/definitions” in the core or as installable packages (plugins) sets of field definitions and fieldname cascades that provide view and edit code for named fields both that come in the core and with plugins or as the result of a design/edition.

  • A designer may be able to pull all defined fields in one wiki to use in another new wiki, there by quickly reusing substantial “prior art”.
  • Existing field edit tools like color and additional ones for date, tag and title pickers, select and select multiple etc… could then be migrated to the field handling mechanisium.
  • Just as tiddlers can be placed in both view and edit modes a future development should introduce a similar mechanisium for fields. Again I have done substantial design work already, and will share / propose in good time. The simplest form of view/edit is in the above proposal, however there should also be mechanisiums to allow modes such as field edit from within forms used in view mode. Including an update mode where the field is in view but a mechanisium to edit inline made available.

Please thumbs up this :+1: if you would like us to proceed but not exactly As presented please :heart: if you agree with what is proposed as a whole. Please reply if you have specific concerns or are in a position to contribute.

2 Likes
Virtually merging two tiddlers? Tiddler stacks and form of monkey patch
Proposal: render fieldnames as links, and see what's possible with fieldname nodes!
Proposal: render fieldnames as links, and see what's possible with fieldname nodes!
Use tag-picker to dynamically pick target tiddler
I believe the TiddlyWiki editing experience is its most significant area for improvement
Modal request for next version of TiddlyWiki
Seeking Advice For Designing Custom EditTemplates
#2

I don’t understand all the nuances in your proposal but, why yes of course it would be useful if fields could be tiddler titles (which already might work well since arbitrary field names are allowed) and if there were corresponding filter operators and functionality to manipulate them.

You defer giving examples of use cases but it is beneficial to get context in order to understand things properly. Better yet are demos that clearly show what problems the lacking features cause.

#3

This is true, but if you look at the details above you will see they are mosly enablers, so it is hard to demonstrate as these features would be pervasive and in the details of solutions than people build on the top of it.

  • I know this is often the nature of some of my ideas as I use my imagination of the possibilities and hastles I have had when writting solutions because these need manual equivalents.

I will see if I can expand on it.

I stumbled upon this related issue How could system-fields look like with 5.4.x #5805

#4

See links re using titles as fieldnames. I want to keep this a possible approach.

#5

That’s a “thought experiment” of mine at GitHub discussions that started June 2021. So it is no actionable issue that can be or should be solved. Just some thoughts articulated, so others may be able to understand it and that they are not forgotten.

The simplified conclusion is: Fields with mata-data become tiddlers. – Very, very simplified.

1 Like
#6

5 posts were split to a new topic: A (flawed) thought experiment about fields as tiddlers

#10

To put this tread back on track and away from the more speculative paths a few words.

  • just as with tag tiddlers, there would be one field tiddler or each field name
    • although there may be room for a different tiddler for each tiddler type
  • like tag tiddlers, who’s fields such as caption, colour and icon, field tiddlers are effectively field definitions.
    • thus field tiddlers could form part of field focused view and edit templates.

for example our field tiddler may have a field values filter that defines what values can be assigned to that field or for manual entry validation eg numeric only, a future date etc…

note: in addition to a possible field tiddler for each tiddler type (method to be developed) I am also working on a way to hide field tiddlers from standard search without the field having a $:/ prefix. I expect to use an additional system name space such as #

  • perhaps a tiddler could have a field such as field-namespace with a value such as $:/name/ which is used to prefix all custom fields to name the field tiddlers. eg I may use $:/PSaT/fields/. Field cascades for viewing and editing fields would apply this prefix to find the field tiddler. this would simultaneously address the namespace, alt field definitions and field types. in face the field-namespace could also be a filter.
#11

One key question

I need to resolve is what if any namespace, or prefix shall I add to “field tiddlers” that define the fieldname. The following are at the top of my list;

  1. Prefixed ▭ eg ▭ fieldname, Searchable but we could hide them with and additional system namespace
  2. Prefixed $:/fields/ eg $:/fields/fieldname, only searchable in advanced search
  3. Or just fieldname = tiddler name eg fieldname, searchable and may conflict with tiddlers of the same name, especially if a tiddlername is used as a fieldname.

Keeping in mind I plan to permit a custom namespace filter on each tiddler. Thus a tiddler may include a field field-namespace containing [prefix[$:/brand/fields/]] which is applied to find field definition tiddlers for the fields on the current tiddlers

A second key question

How do I handle existing standard and core fieldnames?

  1. Include them in this field handling mechanisium ie the field tiddler according to the method chosen above
  2. Provide a seperate namespace to use for known core fieldnames eg prefix $:/system-fields/`
  3. Or just fieldname = tiddler name eg fieldname, searchable and may conflict with tiddlers of the same name, especially if a tiddlername is used as a fieldname.
  4. Just exclude them from field handling altogether?

Keep in mind this will provide additional handling and not impact default tiddlywiki field behaviours.

Proposal: render fieldnames as links, and see what's possible with fieldname nodes!
#12

This is my favorite of the options you’ve presented. I don’t think “configuration” tiddlers’ names should ever overlap with (potential) content tiddlers, and I’m also not fond of any solutions that require characters not found on a standard English keyboard.

I think $:/config/field/... or $:/config/Field/... (following the pattern used by other config tiddlers, which seem to use CamelCase for their broad category) would also be a reasonable solution, as more savvy users may already have absorbed that $:/config tiddlers are safe to modify.

This is certainly what I’d expect. I don’t think it’s a good idea to use separate handling for system fields vs. user fields, particularly as users may not recall which category a given field belongs to. And the documentation implies several potential “system” categories:

  • "standard" system fields (as per that docs tiddler)
    • some of which (like created) are hidden from the standard field editor, some of which are not
    • some of which (again, like created) are written primarily by the core draft system
    • some of which (like caption) are likely to be populated almost exclusively by users
    • some of which (like type) will not be present on many/most tiddlers
  • "other" system fields, which receive special handling in the core — in terms of how the field editor displays them, how core filters use their values, or both
    • some of which (like draft.of) are, again, not visible in standard edit mode — though most in this category are
    • some of which (like name) do have a “special” system purpose, but won’t really break anything if a user repurposes them as standard content fields on a normal tiddler.

Since all this makes the system/non-system distinction confusing, I think it’s best to use the same naming scheme for both and let users adapt it as they will.

Tangentially, I modified my own field editor a while back so that “system” fields1 sort before “non-system” fields and are additionally marked with bolded field names. I find this visual indicator makes it easier for me to recall which rarely-used fields do have a specific core function — so if I do reuse them, I’m doing it intentionally and with some foreknowledge of potential conflicts/side effects. This might be a useful “clue” to introduce if you want to provide some indication that “system” fields (by whichever definition) are special.

1 as defined by the following function (taking some cues from @Scott_Sauyet, IIRC):

\function system.fields()
[all[shadows]removeprefix[$:/language/Docs/Fields/]]
:filter[lookup[$:/config/EditTemplateFields/Visibility/]!match[hide]]
+[join[ ]]
\end
2 Likes
#13

Thanks very much Emily for your contribution.

I understand this sentiment, but here it may actualy be an advantage because “field tiddlers” can only be made programaticaly (without going to more detailed processes).

Perhaps they are not safe to modify?

Perhaps this is an argument to use the $:/fields/ prefix if we want such tiddlers only be modified using a custom view which includes the logic and suggestions a “field tiddler standard” may demand?

  • But I get your point.

Good argument and approach, thanks.

  • I can also make the nature of such fields clearer.

Interesting. However this information is sourced, I am looking at an enhanced field editor and think the suggestion is valid that we present information about known fields, that is standard system and extended core fields. That is where information is available and possibly relavant we present that information so a user and or designer is better informed. Perhaps even via an indicator icon.

  • I still plan to leave the “field editor cascade” intact however if a field definition tiddler exists it will use a lookup to that to see if there is another way given to edit such fields. I also plan to introduce a “field viewer cascade” and the ability to use one found in a “field definition tiddler”.

Something I did not elucidate enough above

I am looking at allowing each/a tiddler to be provided a filter that determins the prefix in use so custom solutions such as plugins can select an alternate prefix for field definition tiddlers, thus come packaged with the solution with the field definitions according to there own world view. Such definitions are only valid on the tiddlers with the matching prefix specified. Fields can mean different things on different tiddlers.

Thanks for this reference. I need to think this thrue, as I was planning to allow the field definition to create/set a visibility tiddler. This is a “state tiddler” and I am not so sure if this is the best way to identify if it is a system field. Or do we define a system field as one which is hidden from the editor?

Thanks again for your considered responce. It is very helpful to me. I hope you can give further feedback.

1 Like
#14

One technology I hope to develop is if a given field definition tiddler has a “field type” eg color field, If the field type is provided the field definition need not be fully defined, by pulling in the settings assisgned to that field type tiddler. This allows the field handling be defined using reusable types. eg color and background-color may share the same type.

  • For all the different possible fields we cound define most can be grouped into a much smaller set of field-types eg
    • color
    • number
    • Short text/long text/text area
    • Title
    • Title list
    • date stamp

etc…

If field-type can be selected and reused there will be less bytes required.

  • Changes will be reflected in all fields of the same type

Or such field types can just be a template source for a new field definition at creation.

  • Changes will NOT be reflected in all fields of the same type

Should field-types be template or reused a tiddler.

  • Such field-types could be distributed freely to all tiddlywiki users who wish to use custom fields, thus allowing a shared resource to evolve.
2 Likes
#15

This is my preference for the idea. Having a field name be a tid feels like a powerful way to make connections. I use the ability for any tid to tag any other tid as a way to organically grow structure, and feel the same would be true for fields. I dont really see tids that are tag names as “tag tids” they are just tids. Similarly, I wouldn’t expect I’d see “field tids” as, well, “field tids”, they’d just be tids. From what you’ve written, I’d just expect them to be tids with a few optional extra fields that control how the equivalent field works, but none of them required (eg: field-type: which might have “list” (as the “type” field has now), or “color” (as the “color” field has now), each resulting in a change of the field entry UI to suit, but without a field-type the default would be “short text”). But the same name being used as a normal tid is also possible, and therein lies the potential for organic and emergenct structure.

That’s not to say there isn’t a place for a $:/fields/ style namespace, but I would assume from the style of name, that it’d ne roughly analagous to the $:/tags/ namespace - in that it holds the system ones. But it’s an imperfect analogy since field names as shown/saved don’t have this prefix. (but maybe they could have a field caption field or something to hold the name as displayed?)

Relevant (I hope) tangent: I know the tid format was modeled after HTTP headers, and they were explicitly modeled after RFC822 - the text message format which has evolved since (RFC5322) mainly on the demands of email, and that evolution originally had a mechanism to ensure official headers were separate from user-defined ones (by promising no official header would begin X-), but later updates didn’t even mention it - the distinction between “official” and “user-defined” became irrelevant. I mention all this because in a way, I feel it’s relevant to here too - a tid can be as small as title: x …and other fields are optional. This is part of why I feel the distinction between system and user-defined doesn’t need to be super struct.

The rfc822 message format also evolved into handling the other thing I read in that issue 5805 thread - handling multiple multi-line data. Email solved it by mime attachment, which in the vast majority of cases is just a simple attachment, but at it’s core effectively defines a mechanism by which a full nested tree of attachments can exist (including other full messages). So on the one hand, the problem has been solved, but it’s been solved in a way that is effectively the issue that Jeremy raised, in that it turns one field into a full sub-tid!

(final note: I’m not suggesting tid format follow rfc5322 format - it’s already diverged in a few keys way, though the continued similarity it has made it convenient for the tid->Maildir proof of concept script I’ve drafted up to give me IMAP read-only access to the raw content!)

#16

It would impact search-ability as well as ease of creation/filtering, though. When I want to figure out how the core (or a third-party wiki) accomplishes something, I generally go to Advanced Search first — and similarly, it’s useful to be able to filter for certain types of system tiddlers in my own wikis. Introducing unicode characters like ▭ adds a lot of (IMO, unnecessary) friction. I couldn’t type that on the fly even if I wanted to; I had to copy-paste it from your post.

I understand and appreciate your point about developing tools to manage tiddlers of this type; I think it’s important to provide GUI interfaces whenever possible (as, in fact, the core does with the field editor). At the same time, I’d say I use the core field editor for less than 10% of the changes I make to fields; most of my work is done with $action-setfield and $edit-text. Now, I’m sure this isn’t at all representative of most TW users… but I do think it’s probably a more common workflow among the power-users and developers who would presumably be making and sharing field templates the most. So I think it’s important not to make it too onerous to search, filter for, or even modify these tiddlers, even if “standard” users will never interact with anything beyond the default GUI.

I think that’s one possible definition. I’m not sure whether it’s quite broad enough for your purposes… but on the other hand, if a given field isn’t displayed in the field editor, you probably don’t need a custom template for it, anyway. And conversely, if a user has deliberately modified the display state of a given system field, it’s reasonable to think they might want a custom display, too. So on reflection, it might work as-is.

If you wanted a more complete list of fields used by the core, I think you could omit the :filter run and simply use

[all[shadows]removeprefix[$:/language/Docs/Fields/]]

This could conceivably catch some plugin shadows, too… I’m not sure whether anyone is actually using the $:/language/Docs/Fields/ namespace in their plugins.

If you don’t want to catch any non-core tiddlers, you could use this instead:

[[$:/core]plugintiddlers[]removeprefix[$:/language/Docs/Fields/]]
#17

I understand where you’re coming from, and generally I’d agree; this is how I use tag tiddlers in my own wikis as well. My concern is that one of @TW_Tones’s goals is specifically to make it easier for users to share configurations… and if I have content in the “author” tiddler, and you have your “author” tiddler set up for a certain field configuration, I can’t import your “author” without losing my own content. Instead, I’d have to open it and manually copy all the relevant fields into my own version of the tiddler.

That’s not too difficult if there are only a few extra fields, of course, though I imagine it could get tedious on mobile. But I think the fact that you can typically “copy” a configuration just by dragging some plugins/system tiddlers — without altering any of your own content tiddlers — is a major selling point of TW. I think it’s risky not to follow that precedent — not to mention potentially confusing for users.

#18

Thanks for the feedback. Yes I see value in this kind of transparency fieldname=tiddlername and do think it could be a good source of innovation. As a result I think it importiant to at least allow this to be the case even if ultimatly its not the default. To clarify example approaches, they currently include

  • fieldname = name of field tiddler “fieldname”
  • fieldname = name of field tiddler “▭ fieldname”
  • fieldname = name of field tiddler “$:/fields/fieldname”
  • fieldname = name of field tiddler specified in the fiddler in which its used using a filter eg: [<fieldname>addprefix[#:/]]

However @nemo one example where this may not be helpful is an area we have not explored much yet and that is using fieldnames based on existing tiddler names, independant of field handling. For example the Tiddler “Tony Smith” may have a field named “Cathy Smith” with the value “sister”. Because there is a tiddler called “Cathy Smith”. You can see here we can search for all tiddlers containing “Cathy Smith” to discover all relationships. I will have a closer look at your references to standards in your reply.

  • So in this case there will be a lot of fieldnames, arguably up to the number of tiddlers, and we need not define every field, but could.
  • We can see here we may want to differentiate “field definition” tiddlers using a “namespace”.
  • Actualy not really. Consider the use of ▭ name it is easy to search for any full or partial string in name the difference is in the search results you will see ▭ name in the list and know what you are looking at. You may also see name which is “just a tiddler” not behaving as a field definition tiddler.
  • Similarly to tags tiddlers is[tag]you can use is.field[] later is[field]. also with all operator.

Arguably this unicode prefix method is more powerful and easier to search than tag tiddlers, and there is otherways not indicator in the search drop down for tag tiddlers/ tiddlers that are also tags.

I get what you are saying here, but I think you may be the exception. This solution should support your approach by providing an implied standard way to untroduce field edit code to replace the default field editor. especialy for the suit of different input types, ranges and validation.

This seems like a good definition or method to identify “core system fields”, and allows us to stil leverage the $:/config/EditTemplateFields/Visibility/ mechanisium :+1:

My uncertiantly now exists around language changes. Are these tiddlers simply overidden by other languages? Observing $:/language/Docs/Fields/name contains “The human readable name associated with a plugin tiddler” which is a system field, but users may very well define differently.

Some further thoughts

Thanks @nemo and @etardiff for your continuing feedback. I see here how I/we need to juggle two factors, one is the value of standards and guidence, the other innovation and creativity.

standards and guidence

For want of a better term “field handling” is already something anyone can do in tiddlywiki by making their own rules. I have gone down this path a number of times, and find lots of nessasary details bog down development of field handling, and tends to make the solution more and more bespoke and not nessasarily keeping things open for intergration with other solutions, plugins and tiddlers.

This is why I have ventured into this fairly complex project because if we can establish a robust yet flexible solution for frield handling on which users/designers can build advanced field handling we provide suggested “Standards and guidence” and enable cross solution intergration.

#19

Just for clarity, I think this could be a cascade “looking for the field definition tiddler”.

  • ordered bottom up
#20

Also on lurking over here Check for empty fields with a "generated" field name I need to add to the requirments a way to group fields into sets.

Even if a wiki built by a single user there will need to be logical groupings of field tiddlers, we could call these “forms” or “tiddley types” eg all the fields that belong to a “contact tiddler”, I am tempted to recomend tags eg each field definition that belongs to a “contact tiddler” would have the tag “contact tiddler” or “contact-fields”.

  • You can see here how field handling starts to quickly bleed into other “nomenclatures” like field-types, tiddler types, forms etc…
1 Like
#21

My point was that it would be significantly more difficult to search for ▭ — a character I can’t type or even Google efficiently without copy-pasting — or for [prefix[▭]] or similar filter constructions I might use if I was trying to list all the tiddlers of this type. $:/fields (or any of your other alternatives) is conversely very easy.

Personally I’m happy to agree to disagree about this. But if you’re hoping to ultimately get some or all of this work into the core, I just don’t see it being adopted if it relies on unicode titles, so it seems prudent to design with that in mind.

#22

Comments accepted.

Although just as we have filters to find tags I think we can have filters to find field tiddlers, field definitions., with or without a unicode character.

  • I would like make an arbitary prefix possible, if not a unicode one.
  • Similarly I would like to be able to add a new system namespace eg with arbitary prefix, so If I used ▭ then they remain hidden like $:/ tiddlers.

Now I need to find a pithy name for “field definition tiddler”, then build some filter operators to find them eg: [all[fields]] or [is[field]]. [all[field.definitions]] or [is[field.definition]].

  • perhaps “definition” is too general, and needs the field qualification?
  • of course suplimenting the current fields, has, get…

You can use these operators rather than the prefix to find field definitions just as you do when searching for tags with or without tiddlers system or otherwise [tags[]] etc…

#23

Notes on my Design strategy

I am trying to bring together knowledge from my own and others use of tiddlywiki from over more than a decade. In this case speficialy to handle fields, field-types and may extend into “tiddler-types” and forms, Hopefully this will converge on a good solution.

However: As an expierenced designer I know it is importiant to keep the initial design development phase open as long as practical, not to become locked into a particular path. I am always looking for solutions that;

  • Avoid forced compromises
  • Leverage existing standards or philosopies
  • Are as intuitive as possible

In this case I feel there may be more to be discovered here!

so I continue to encorage ideas here.

Note on missing tiddlers;

I have come to realise that the current missing tiddler mechanisium is less than optimal. The $:/language/MissingTiddler/Hint is only invoked at the lowest point in the existing cascads from within the “view template body”, making it hard to ultimatly have different “missing tiddler types”. Relavent here is “missing field (definition) tiddlers”, but the same could be said also for missing tag tiddlers, or missing alt-tag tiddlers.

  • I suspect a new cascade would make sense here to determin by filter which missing tiddler hint to use.
  • Realise this is a somewhat advanced subject that I know perhaps only myself, @etardiff and @Springer have focused on of late.
  • I am now going to study the Info > Advanced > Cascades to look into introducing additional ones for field definitions and missing tiddlers.