Separating user defined fields from, well, all the others

Commander is wonderful. But it’s also worth noting that it’s relatively easy to do these bulk changes on your own. Here is a button to rename "source" to "origin" across all non-system tiddlers:

<$button>Rename 'source' to 'origin'
  <$list filter=[!is[system]has:field[source]]>
    <$action-setfield origin={{!!source}} $timestamp="no" />      
    <$action-deletefield source $timestamp="no" />
  </$list>
</$button>

Note: The above has been edited to add the $timestamp attribute, as suggested by @Greyed. This avoids changing the modified date and therefore moving these to the top of the Recent tab.


Rename Source.json (411 Bytes)

Hi Scott,

I’m really glad that you found my post helpful. Thanks for letting me know. I’m also grateful to be reading your reply over breakfast with a coffee to hand because there’s a lot of learning in there for me to unpack. I’m a complete beginner when it comes to working under the hood of Tiddlywiki, or even polishing the chrome, to push the metaphor to the limit. Though I’ve been using TW for a good few years now.

The remove prefix operator is a particularly interesting discovery for starters. And the whole block of code under <table>, that creates the tabulated list of fields and their descriptions. I’ve been wanting to find out how to do that for a while. I presume it’s a macro.

I haven’t gone into those parts of the TW jungle yet, I’m just hovering at the edge of the clearing, listening to the noises. Looks like it’s time to put on my bush hat and head in. :cowboy_hat_face:

Regards, Chris

Whoah! That’s above and beyond! And great learning. Thank you Scott, that’s very good of you. Best regards, Chris

I assume you mean how removeprefix[...] does TWO actions:

  • filter out items that don’t have the specified prefix
  • remove the specified prefix from the items that are left

Although it is sometimes convenient to do both actions (as in your previously posted response), there are also times when you just want to remove the prefix while retaining all items, even if they don’t have that prefix. For this behavior, you should use trim:prefix[...] instead.

-e

2 Likes

Yes, exactly. I always prefer the simplicity of tools that do one thing, allowing me to combine them as I choose.

Oh, yes, thank you. I use trim often, but only as I would in JS, to remove leading/trailing whitespace. I guess I never looked at the docs to realize that it had other behavior as well. Very useful; thank you.

Yes, Your Honor, the defendent was found exceeding the posted limit of one extended metaphor per message… :slight_smile:

There are a lot of operators. It takes a long time to learn them all, or at least longer than I’ve been able to spend in the last few years.

And as seen in the conversation here, sometimes individual operators do multiple things, either by default, as in removeprefix or via configuration, as in trim.

There are various tools worth investigation. I assume that if you’ve been doing this a while, you’re at least passingly familiar with WikiText, but he others may be new to you.

It’s a combination of HTML markup and a ListWidget. If you’re familiar with HTML from another context, it works pretty well the same inside TW tiddlers. It’s difficult (at least for me) to get WikiText Tables to work with lists, so I usually end up with HTML for them.

How I work

Typically, I would start with a <<list-links>> macro to make sure I have the filter I want to use correct. Here I already tested the filter in the Advanced Search > Filter tab, but I still rechecked it here. So my first step would be:

<<list-links filter:"[all[shadows]prefix[$:/language/Docs/Fields/]]" >>

That displays a bulleted list of tiddler titles, and looks correct, so I mechanically change that to a <$list> widget like this:

<$list filter="[all[shadows]prefix[$:/language/Docs/Fields/]]" >
  <$link/><br/>
</$list>

Note that we now have opening and closing tags for the <$list> widget, and we’ve moved from the macro shortcut syntax of <<name ...params >> to the more HTML-like <name ...attributes> ...content </name>, where name uses the tiddlywiki convention of starting with $, so <$link ...>...</$link>. Also note the minor annoying need to switch from filter: to filter= as we switch from the macro shortcut syntax to a list widget. (This might be going away sometime soon.) Finally, we have some content that will be created for every value our filter creates. Here I use another placeholder, just to make sure my list widget is acting correctly. I include <$link />, which will create a link to the current title, followed by <br/> to add a line break between them.

When I’ve confirmed that the above is working correctly, I replace the content with the markup for my table row, and wrap the whole thing in <table> tags and add a header row. (I often skip this step, but it can help me avoid making too many changes at once.):

<table>
  <tr><th>Name</th><th>Description</th></tr>
<$list filter="[all[shadows]prefix[$:/language/Docs/Fields/]]" >
  <tr>
    <td>name will go here</td>
    <td>description will go here</td>
  </tr>
</$list>
</table>

I fill in with the actual content I want for my table cells, generally one at a time, saving and confirming after each change. First I want to start with the current tiddler and remove the prefix to leave just the name:

<td>{{{ [<currentTiddler>removeprefix[$:/language/Docs/Fields/]] }}}</td>

for name is almost fine, but I don’t want it to be a link, so I wrap it in a text widget

<td><$text text={{{ [<currentTiddler>removeprefix[$:/language/Docs/Fields/]] }}} /></td>

That works fine, and now I want the description:

<td><$text text={{{ [<currentTiddler>get[text]] }}} /></td>

Et Voila!, we have our table:

<table>
  <tr><th>Name</th><th>Description</th></tr>
<$list filter="[all[shadows]prefix[$:/language/Docs/Fields/]]" >
  <tr>
    <td><$text text={{{ [<currentTiddler>removeprefix[$:/language/Docs/Fields/]] }}} /></td>
    <td><$text text={{{ [<currentTiddler>get[text]] }}} /></td>
  </tr>
</$list>
</table>

This is my usual process, and most of it is mechanical and quick. Usually finding the exact filter expressions for the list – and sometimes for extracting the text cells – are the only time-consuming parts. This one took me maybe five minutes to create… and a lot longer to write up here.

Note that that bare-bones list made with <<list-links ...>> may last a good while. It’s perfect temporary filler for something I want to come back to later but don’t want to distract me now.

If you haven’t figured this out by now, then good news: this actually isn’t nearly as hard as you think it is! In fact, I figured out (b) in the course of trying to update Scott’s filter a couple posts ago by just poking around the core’s tiddlers a little.

  • Scott’s language-based filter in post #13 is fantastic to show all of the non-system fields, but it has a small flaw for your use case: it won’t include fields which are normally system fields that a user has added to their own tiddlers, like you said you’ve done with the source field.

Here’s how you can do (b):

Because we’re looking for how the core does the “field” dropdown in a tiddler’s “edit” mode, I did an Advanced Search for field edit, and clicked over to the Shadows tab where most of the core tiddlers will be. In that list is $:/config/EditMode/fieldname-filter – open it, and pop up its Info pane (in the More dropdown) and look at the Fields tab. (Or you can just edit the tiddler to view them.)

See how there’s two filters there, one in a field named first-search-filter and one in second-search-filter? Those are the two filters that are used to generate the two halves of the add-a-field’s dropdown menu. The first- one shows just the user fields, and the second- one shows just the system fields.

  • You can verify what each filter does by copying it into a $list in a new tiddler and seeing what it returns.

By default, the first-search-filter field’s contents are:

[!is[shadow]!is[system]fields[]search:title<userInput>sort[]] -created -creator -draft.of -draft.title -modified -modifier -tags -text -title -type
  • By the way, this also answers your question of “why is the system source field listed in the field dropdown’s user section now?” When you add a field which is normally a system field to your own tiddlers, it becomes considered a user field by the above filter – because it now has a non-shadow, non-system tiddler which includes that field, it no longer gets filtered out here!

We don’t actually need the part of the first filter run that says search:title<userInput>, because that’s what “trims down” the list of fields as you type into the add-a-field text box. So you could simply copy the above filter, remove that bit, and ta-da! you’ve got a nice, working list of user-created fields.

But! We can make it a little tidier by simply transcluding that field’s filter directly into our new filter:

[subfilter{$:/config/EditMode/fieldname-filter!!first-search-filter}]

Note the curly braces used for transclusion there. We also need to pass it to the $list as a subfilter for it to work.

  • Note that this method will keep the search operation, but I don’t think it should be too much of a performance hit because you probably won’t have a “userInput” variable defined in this tiddler anyways.

Regarding (a):

Scott’s provided code in post #14 is your best bet! I’d just like to add a little tip to it: if you need/want to keep the current last-modified timestamps on your edited tiddlers, both the $action-setfield and $action-deletefield widgets he uses in his code support the $timestamp="no" attribute (in TW version 5.3.4 or higher.)


…For fun (shh, I was on a roll), if you have more fields than just source to change, why not make a single button press change ALL of them at once? :smiley: Replace the “FIRST-FIELDNAME-OLD” and “-NEW” in the code with the actual first field’s old and new names, and so on for additional field names.

WARNING! POTENTIALLY DANGEROUS CODEmake a backup and double-check the output of the filters first by removing the $button widget plus the “fieldrenamingaction” calls, and replacing the latter with something like this to verify the components being used:

''Tiddler:''  <$link to=<<tidtochange>> /> ¦ ''Old field:'' <<fieldtochange>> ¦ ''New field:'' <<condition>> ¦ ''Contents:'' <$transclude $tiddler=<<tidtochange>> $field=<<fieldtochange>> />

(Also, I won’t lie: the below code is super clunky. If someone suggests an improvement to it or finds a typo I’ve made, listen to them instead! :sweat_smile:)

\procedure fieldrenamingactions()
<$action-setfield $tiddler=<<tidtochange>> $field=<<condition>> $value={{{ [<tidtochange>get<fieldtochange>] }}} />
<$action-deletefield $tiddler=<<tidtochange>> $field=<<fieldtochange>> />
\end

<$button>
Rename multiple fields

<$list filter="[subfilter{$:/config/EditMode/fieldname-filter!!first-search-filter}]" variable="fieldtochange">
<$list filter="[!is[system]!is[shadow]has:field<fieldtochange>]" variable="tidtochange">

<%if [<fieldtochange>match[FIRST-FIELDNAME-OLD]then[FIRST-FIELDNAME-NEW]] %>

<<fieldrenamingactions>>

<%elseif [<fieldtochange>match[SECOND-FIELDNAME-OLD]then[SECOND-FIELDNAME-NEW]] %>

<<fieldrenamingactions>>

<%elseif [...] %>

	(continue as before for each new field, to define the old field names and their new names)

<%endif%>
</$list>
</$list>
</$button>
  • The condition variable used in the procedure’s actions is automatically set by each conditional %if statement to be the output of its own filter, so in this case it will be each new fieldname you specified for that section of the %if.

To only change some of the user-fields, you can add %if / %elseif sections for only the old tid names you want to include; everything you don’t include should be ignored. Or, you can replace the initial $list’s filter that pulls ALL user-fields with [[a specific list]] [[of]] [[fieldnames]] +[enlist-input[]] . The former is less initial work, but the latter might be speedier if previewing the field changes makes your wiki choke or get laggy since it’s working with fewer starting tiddlers.

1 Like

(I’m adding this as a separate post, because it’s not actually quite what ChrisH’s particular case needs. It’s just a nice tweak I came up with, so I thought I’d share.)

This filter is amazingly creative, and solves a minor problem I’ve had for a while! But I found it to have one small flaw: it does not exclude fields used in plugins. So here’s a way to get the results without these plugin fields included:

[subfilter{$:/config/EditMode/fieldname-filter!!first-search-filter}] -[all[shadows]removeprefix[$:/language/Docs/Fields/]]

As noted in my post above, the $:/config/EditMode/fieldname-filter tiddler’s “first-search-filter” field contains the list of all user-created fields, which is used for the add-a-field dropdown list’s “User fields” section. By using this existing filter, we can take advantage of the core’s method which excludes plugins’ fields from the “User fields” section.

  • If we solely use the language method to filter, it would include any plugin field or core field that doesn’t have a language entry – for example, in my wiki, the “demo” field that the Relink plugin uses was included, as well as the “status” field that the core $:/Import tiddler uses.

By then adding your language-based method to exclude system fields, that also excludes originally-system fields like caption or code-body which might regularly be added by users.

Thus, the output of the above tweaked filter should be EXCLUSIVELY user-created fields – well, as long as nobody’s added their own language entry for them!

That’s a very nice addition, and I think there are times I’d want either one.

Often I don’t want to care about what’s in plugins. Mostly I would use this for when I want a list those fields I use in a particular tiddler that are not part of Tiddlywiki’s core. That Relink uses demo, AutoComplete uses rows and PlantUML uses documentation, would not stop me from using them for my own fields. But I’d probably always avoid type, or modified.

Other times, it’s possible that I’d want to scan the whole wiki, plugins included.

The big concern of course is that somebody adds these $:/language/Docs/Fields/* tiddlers for other fields, and ones we care about disappear from the list. I guess they shouldn’t. Jeremy has recently made it clear that he thinks of the $:/language namespace as reserved for the core. But it’s a concern. For my own uses, it would not be hard to whitelist back in a value that is otherwise restricted by these techniques, but it’s easy enough to imagine places where that is not possible.

Very good point. And in fact, I’m going to edit my post to include that.

Hi Scott,

Well, I don’t know what to say other than thank you very much for going to all the trouble and effort of writing such a comprehensive post. It’s very generous of you and I’m very grateful to you for it. The post is really helpful to me and I’ll try it all out tomorrow. It’ll also force me to try to get my head around working with HTML code, a subject that I know even less about than TiddlyWiki’s inner workings.

Best regards and thanks again,

Chris

1 Like

Hello Greyed,

It’s nice to make your aquaintance too. And thank you very much for this additional material! My reply to Scott just now clearly applies here with equal weight. It’s seems clear that the TW community is a friendly and generous crowd. I’m quite overwhelmed by everyone’s kindness, though you are all stacking me up a big study day tomorrow.

Best regards, Chris

I have explored fields deeply but will always use the fieldnames I want without prefix or suffix except if I want to groups the fields.

A smart approach is to use a field such as object-type (because type is in use) and use this when designing tiddlers with custom fields and thus you can list all fieldnames that belong to that object-type. You can call this self documenting.

  • There is rarely if ever a clash of fieldtypes if your solution has this other layer of naming, the object-type, just dont list or consider a custom field unless its in a known object-type.
  • Your solution will not have any problems due to others using the same fieldname else where. Other solutions rarly clash, although I avoid the system fields.
  • When using current tiddler you need only query the fields on the current tiddler. Include empty fields.

I will see if I can find my own field filters or lists I use in a function so I can filter fieldnames dynamicaly.

[Edit] Use relink plugin for fields as well

Hi Anthony,

Thanks for your advice, I appreciate it. But can you clarify for me one of the points that you made? It’s this phrase from your second paragraph.

I’m not sure how to interpret that. Do you mean that one should use the “object-type” field, for example"? I don’t think so because I can’t find such a field but thought I’d ask, just to be sure. Or are you proposing “object-type” as a generic naming convention for user defined fields. So that the “object” part is a field specific name, whereas the “type” part names a (possible) group of related fields?

I have used something like that format for a few of my own user defined fields, though my constructions take the form “type-object” not “object-type” but it serves the same function. For example: “game-date” to reference an arbitrary date in the game; “location-aft” and “location-fore” to reference locations on the game map that are immediately behind and just ahead of some character or group. You see what I mean.

I didn’t do that as a rule, however, because I preferred to use single nouns or verbs as names wherever possible, as you would, it seems.

Best regards, Chris

Imagine a wiki that tracks purchases for your online store. You might have tiddlers categories of Customer, Supplier, Item, LineItem, and Order. My own preference would be to add tags of Customer, Supplier, etc. to the appropriate tiddler. But some people like to keep their tag space very tidy. So instead of a tag, you could use a dedicated field, say, object-type:

title: Acme Manufacturing
tags: (whatever)
object-type: Supplier
...
title: Wile E Coyote
tags: (whatever)
object-type: Customer
...

While we might prefer to use type, that one is used by the core, and is quite important to the functioning of TW. We couldn’t well reuse it for something else.

Thanks Scott. That’s clear now. Regards, Chris

I am suggesting the use of a custom field such as creating a field “object-type” so each tiddler you create can be assigned an object-type eg contact, character, business, document. Then when you use additional fields they are added to the tiddlers with a given object-type.

  • As you qualify a fieldname (mobile-phone-number) as belonging to an object-type eg “contact” the same fieldname can be used in another object-type with a different meaning.

The reason I chose object-type is because I wanted to use type but it is in use, the word object implies a tiddler represents an object, a logical object in database terms.

The thing is when you add this logical grouping of object-type you write your lists and code to first filter on object-type and it will never accidently use the same fieldname from other tiddlers or plugins, and you are able to use what ever fieldnames you want.

Hi Anthony,

Thank you for taking the time to offer this clarification. I’m grateful. I’ll need to mull that over for a while to work out the ramifications but here’s what I understand you to be saying at first glance.

“object-type” is your suggestion for the actual name of a user-defined field that I could add to every non-core tiddler in the wiki, together with a value taken from a set the members of which specify particular "type"s (eg message, character, event, decision) of “object” (ie non-core, non-plugin tiddlers).

The field is then used early in any filter run to select only those tiddlers that either have that field or, more selectively, match a particular “object-type” value of that field.

The aim of this early filter is to ensure that any field name that is used later in the filter run that duplicates the name of a system field could only be attached to a non-core, non-plugin tiddler and so must be: (a) the user defined version of the field: and (b) only be carrying a user defined value for that version of the field or no value.

OK, :thinking: … I’m pretty sure I’ve got that now :crossed_fingers: and I’m even more sure that it would work as I already achieve something similar with tags, having names such as those in the example that I gave in parentheses, earlier.

The question that remains is whether it’s the right way to go? I have a nagging feeling that we’ve come adrift from the original problem, though with the definite advantage of having created some really useful TW learning opportunities for me.

My original problem was, how do I generate a reliable list of the user defined fields to be found in the wiki and only those fields. The main reason behind that was to use the list in a table which would: (a) list all the user defined fields (duh); and (b) provide a description and intended function for each. I think that question is now well answered. My thanks (in case I forget) to ansolutely everyone who’s chipped in.

For what it’s worth, I think that the second best solution for me is to rename the duplicated field or fields as that seems like the better practice in any case. I can then work on the most efficient way to get that list from a filter, from among the many ways that have been suggested.

So I’ll get on with that while also thinking about how I get to the “best” (OK, “better”) solution to my original problem, apart from deduplicating the names. (“Deduplicating”? Did I actually write that? Is deduplicating actually a real word or did I just make it up?)

So that is, how can I add my description and intended function to directly each user defined field so that the information populates the table that Scott created in his earlier post? See post #13 May 20, 2:28 AM.

I think there’s more than enough guidance available to me from Scott’s post to allow me to tease that out, so no one need feel obliged to write a “how to” for me though I’m grateful for the thoughts that were probably already forming and more than happy to get one if that’s what anyone actually wants to do.

Right, I’d better stop now. Things to do, places to go, people to see … 'Bye for now and thanks again, Chris

In reality perhaps you want to do this for the tiddlers you introduce not all tiddlers with custom fields so as an example you may use a filer to find all fields in all your object-types. The advantage being if you add a field to any tiddler with an object-type it will then appear in your list.

  • Not withstanding this you can use tiddlywiki.com, or an empty.html and find all the standard fieldnames, put these in a list, within a function and use this in filters to exclude the “standard fields” all else are custom fields, or from additional plugins or tiddlers. Exclude fields not defined in your object-type tiddlers and you can see.

Real word, desperatly importiant in IT, especialy in the old days of batch programing, but sneeks in in many places :nerd_face:

Thanks Anthony,

That’s a good point but in this case custom fields (i.e. user-defined fields) are only present on tiddlers that I’ve created. I haven’t added any fields to tiddlers that are part of the core or of any plugins. In essence I just want to be able to list those fields that I’ve created for the purpose of running the game together, ideally, with their descriptions just as a reminder of what they are and what I’m using them for.

Thanks. And consulting the online Oxford English Dictionary, I’ve been very surprised to discover how far back it goes.

The earliest known use of the noun deduplication is in the 1830s.

OED’s earliest evidence for deduplication is from 1835, in the writing of John Lindley, botanist and horticulturist.

deduplication is a borrowing from French. Etymons: French déduplication.

Incroyable, non?

Anyway, thanks again and best regards, Chris

Incroyable, Oui.

I think there is more to be said on this subject so when I get the chance I may add a little more here. For example

  • Documenting the difference between fieldnames as they appear on the add field dropdown, which shows shows user and system tags seperatly.
  • Including a “registry” in the form of lists, that group logical relationships between sets of fieldnames.
  • documenting how to hide fields from the field editor as modified and created already are
  • Identify is there are there places we should request some additional hackability