Feedback on documentation of filter operators

[vuk]

1. Spotted three filter operators that lack examples:

• function
• slugify
• order

Purely intuitively, first of them seems to be of significant importance

2. each seems to be my personal nemesis: after skimming through all other operators (with the exception of the above three without examples), I had a first feeling that I kind of superficially understand how they work (I’m well aware I may be wrong). But after reading over and over the description and examples for each operator, I still don’t get it at all

[telumire]

I made a PR for the function operator: Add example for the function operator by DesignThinkerer · Pull Request #8793 · TiddlyWiki/TiddlyWiki5

Try it here: TiddlyWiki v5.3.6 — a non-linear personal web notebook

Feedback is welcomed.

[CodaCoder]

@vuk

But after reading over and over the description and examples for each operator, I still don’t get it at all

I couldn’t agree more. The docs for each are at best ambiguous and at worst meaningless^[1].

Working from the top, here’s the phrasing I have trouble with:

1. each group: looking ahead to “input: a selection of titles” I have no idea what each group could possibly mean. So right at the beginning, we’re not off to a good start.

2. each:list-item: “The value…” [value? the value of what? a title?] “… is treated as a title list” [each title is treated as a title list??? ] “… Each title in the list…” Oh boy, now we have value, title, title list and each title in the list. In my head, I’m already dealing with a list derived from a filter supplied to the <$list> widget. Is this sentence talking about my list or this arbitrary title list it seems to have invented out of nowhere which I don’t yet understand?

3. each:value: “As long as the title is unique it is appended to the output whether or not the corresponding tiddler exists.” Unless I’m missing something, this sounds completely redundant. Isn’t that what filters do anyway? Why do I need this when “1 2 3 4 5” is a valid list where NONE of those exist as tiddlers. And if it isn’t, why isn’t that made clear in the wording?

At this point, I’m going to stop and leave you with this: I’ve only ever needed each[X]has[X] which works fine though intuitive it is not. Without searching my code for an example, I’d be hard pressed to explain why I needed it or used it.

---

[1] Meaningless in the sense they transfer very little meaningful “take away” information into my brain.

[vuk]

Disclaimer: I fully accept responsibility, I acknowledge this is a PEBCAK issue, despite me not being a native English speaker. My [hopefully] relatively impartial comparison rule is, as I wrote above, that this is pretty much the only piece of documentation about filter operators that gave me trouble. So far I don’t have any advices how to improve this particular situation here (since I don’t even understand this operator yet!). Perhaps other TiddlyWiki users remember their aha moment (after not getting it after the first read) about it and could share what made it click for them.

[CodaCoder]

Perhaps @Scott_Sauyet will finish the response he was typing a few minutes ago.

[Scott_Sauyet]

@telumire

I made a PR for the function operator:

Thank you for taking this on; this has been long needed!

But I’m not sure if this is capturing the right thing here. I can’t see the actual use of the function operator here, just of functions. (Never mind, I was blind!)

A simple case might be something like this:

\function double() [multiply[2]]

{{{ [enlist[1 2 3]function[double]] }}} <!-- yields: 2 4 6 -->
<$--              ^^^^^^^^                 -->

If the name of function we are using includes a ., then we can include it directly in our filter expression, and don’t need the function operator:

\function dou.ble() [multiply[2]]

{{{ [enlist[1 2 3]dou.ble[]] }}} <!-- yields: 2 4 6 -->

[Scott_Sauyet]

@CodaCoder

Perhaps @Scott_Sauyet will finish the response he was typing a few minutes ago.

Sure, but it’s not about each. I’ll try to look at each soon, but if it’s what I expect it to be (from my JS experience), then it’s a function I mostly avoid anyway.

[Scott_Sauyet]

Here’s my understanding of each:

If we have ten tiddlers tagged Demo, each with a my-field value, we can note if each one is the first with its specific value:

title	my-field	first?
A	foo
B	foo
C	bar
D	foo
E	bar
F	bar
G	baz
H	bar
I	foo
J	qux

then the filter [tag[Demo]each[my-field]] will return those firsts: A C G J.

But gleaning that from the documentation and/or the examples is tricky. And I haven’t even looked at the two suffixes.

EachDemo.json (1.2 KB)

[etardiff]

@vuk

each seems to be my personal nemesis: after skimming through all other operators (with the exception of the above three without examples), I had a first feeling that I kind of superficially understand how they work (I’m well aware I may be wrong). But after reading over and over the description and examples for each operator, I still don’t get it at all

Scott will have to weigh in with the JS comparison, but here’s my layperson’s understanding:

• each[field] returns each input tiddler (i.e., each tiddler produced by the preceding filter steps, or each tiddler in the set of all[tiddlers] if each is the first filter step) with a unique value of the field field.

  • IMO, this is the most confusing use. From the name alone, I would expect each to return each unique value of field, but this is not what it does; instead, it returns each tiddler that has a unique value. When multiple tiddlers have the same field value, it returns only the first title with that value.
  • each treats field values as literal strings, not title lists: field: A B C and field: C B A are unique field values, even though they contain the same list-items.
  • If you want the field values, not the titles of the tiddlers that have them, you need each[field]get[field].

• By contrast, each:list-item[field] does return field values (not tiddler titles), with a twist: each field value is treated as a title list, and list items that appear in more than one tiddler’s field only appear once in the output of each:list-item.

  • For instance:

    

  • As you can see above, each:list-item[field] is equivalent to get[field]enlist-input[]unique[]

• I’ve never actually used each:value[], but at a glance it seems to be equivalent to unique[].

  • Unlike the other two each variants, this one isn’t concerned with fields at all.

  • Also unlike each, each:value does not care whether an input value is a tiddler title or not. (“Vanilla” each automatically discards all inputs that don’t correspond to tiddler titles, since — by definition — a tiddler that does not exist does not have any fields.)

  • It cannot take a parameter (other than, technically, each:value[title]).

    • each:value[tags] = each[tags] → the :value suffix is ignored.
    • each:value[title] = each:value[] → probably for the same reason that [[missing tiddler]] :filter[{!!title}] returns [[missing tiddler]]; {{!!title}} and <<currentTiddler>> both yield the string output of the filter, whether or not that string corresponds to an actual tiddler with a title field.

  • Demo:
    

    
    
    vs.
    
    

  • Again, I don’t know why you’d use each:value[] when unique[] exists. EDIT: Because it’s (sometimes considerably) more efficient, as I discovered later in this thread.

[CodaCoder]

@Scott_Sauyet

But gleaning that from the documentation and/or the examples is tricky.

My point, exactly.     

[Scott_Sauyet]

@etardiff

Scott will have to weigh in with the JS comparison, but here’s my layperson’s understanding:

That was a bad guess on my part. This has nothing to do with each/forEach found in JS itself or certain libraries. While I see reasonably well now what it does, I have no real idea why it exists. Does anyone have real-world examples of using each?

[etardiff]

@Scott_Sauyet

Does anyone have real-world examples of using each?

I often use each[field]get[field] for prepopulating $select dropdowns with all the values I’ve previously used in a field.

• For instance, I have a number of tiddlers that summarize external sources, where the name of the source is stored in the source field.
• I can add a segment to my EditTemplate that lets me type a new value for the source field or quickly select from the list of sources I’ve used before:

@@.combobox
	<$edit-text field=source tag=input placeholder="type or select" />
	<$select field=source>
		<$list filter="[each[source]get[source]sort[]]">
			<option>{{!!title}}</option>
		</$list>
	</$select>
@@

(.combobox styling courtesy of @telumire)

In this case, I’m using the source field to store a single title only, so each[source]get[source] is appropriate. But elsewhere in my wiki, I also use an authors field which uses title-list format. For instance:

title: The Elements of Style
authors: [[Strunk, William Jr.]] [[White, E. B.]]

If I wanted a list of all the authors represented in my wiki, I’d use [each:list-item[authors]sort[]].

[vuk]

Filter operator that selects one tiddler for each unique value of the specified field.

Surprisingly, this is concise and at least I get the feeling that I understand[1] This is quoted straight from each.js.

With suffix “list”, selects all tiddlers that are values in a specified list field.

This follows immediately and… is a total black hole again for me!

I got curious and ran a git blame on that file. Very interesting findings:

1. Apparently this inline minidocumentation was added a couple of years after the original code.

2. Commit dates show that the list suffix functionality was added later as well (here Operators work inconsistently · Issue #3117 · TiddlyWiki/TiddlyWiki5 · GitHub)

---

1. Note: I mean I understand the “how” part, not the “why” (as in where would I use this) part as well.

[Scott_Sauyet]

@etardiff

@Scott_Sauyet

Does anyone have real-world examples of using each?

I often use each[field]get[field] for prepopulating $select dropdowns with all the values I’ve previously used in a field.

Interesting. Is there an advantage to this over [get[field]unique[]]?

@etardiff

(.combobox styling courtesy of @telumire)

Damn, that looks like it could be useful! Do you have to do any more than add that stylesheet and use your code? My initial attempt only shows the SELECT, without the text input.

ComboboxTest.json (2.1 KB)

[etardiff]

@Scott_Sauyet

Interesting. Is there an advantage to this over [get[field]unique[]]?

I’m actually not sure how they compare, performance-wise. My gut feeling is that each[field]get[field] may be faster.

… and some quick tests in my largest wiki (25360 total tiddlers, 1740 tags) support that:

Here’s the data from Maurycy’s Advanced Performance plugin. I ran each filter three times, and you can see that the each filter is consistently more efficient at scale—though the difference may be less noticeable in a smaller wiki.

Edit: Just for fun, I also compared [each:list-item[field]] with [get[field]enlist-input[]unique[]]:

And each:list-item was so much faster that I couldn’t get them both onscreen at the same time.

With unique[] vs. each:value[], the difference was less dramatic…

… but again, each wins out. So I suppose the reason to use it over other constructions that yield the same results is simply: efficiency! And I may need to go refactor some code now.

@Scott_Sauyet

Do you have to do any more than add that stylesheet and use your code? My initial attempt only shows the SELECT, without the text input.

I end up tweaking nearly all the code I borrow, so it’s very possible I’ve changed it in some incompatible way. Here’s the .combobox style from my own stylesheet:

.combobox {
--dropdown-button:20px;
 position: relative;
 display: inline-flex;
 padding-right: var(--dropdown-button);
	background-color: #fafaf9;
 border-radius: 5px;
	width: 10em;
}

.combobox input {
	background-color: #fafaf9;
	width: 10em;
}

.combobox select {
 position: absolute;
 /*the width of the select will be the width of the dropdown*/
 width: 100%;
 /*select is hard to style, so we clip the size we want and hide it, then show a pseudo-element above it*/
 opacity:0;
 clip-path: inset(0 0 0 calc(100% - var(--dropdown-button)));
 pointer-events:all;
 cursor:pointer;
}

In my wiki, that looks like this:

Tested on TW-com, the dropdown caret isn’t visible, but you can type into the input field, and clicking at the right end of the box brings up the dropdown. I’ve restyled the base input elements in my wiki, and I suspect that accounts for the difference; you may need to adjust the width/positioning to get things to line up properly.

[Mohammad]

I like these examples as they are easy to understand, and no other concept is required to know in advance. Those given in the PR are very difficult (they are good for advanced users)

[Mohammad]

Worth to be added to TW official docs.

[Scott_Sauyet]

@etardiff

Here’s the data from Maurycy’s Advanced Performance plugin.

I’m sorry. I didn’t mean to give you an assignment. But thank you very much for checking on this.

Clearly each is valuable. All the more reason to ensure that it’s well-documented!

@etardiff

@Scott_Sauyet

Do you have to do any more than add that stylesheet and use your code? My initial attempt only shows the SELECT, without the text input.

I end up tweaking nearly all the code I borrow, so it’s very possible I’ve changed it in some incompatible way. Here’s the .combobox style from my own stylesheet:

That works for me, subject the arrow-visibility caveat you mention. Thank you very much for sharing. I can see several similar uses of my own for such comboboxes.

[pmario]

@vuk

Purely intuitively, first of them seems to be of significant importance

2. each seems to be my personal nemesis [. . .]

There is a bug in the each[] filter implementation. I am having a closer look at this one at the moment.

I also think it is necessary to have a closer look, where the each[] operator was used the first time in the TW core. This points to the main usecase it was developed for.

So the each-operator help from TW v5.1.0 points us into the right direction. That’s the whole purpose this operator was developed for. So it basically is a helper operator to create “typed-lists”

Now the operator is more powerful, but it’s purpose is pretty much the same.

• It iterates through all tiddlers and
• Returns the first tiddler that has a new eg: type

[vuk]

There is a bug in the each[] filter implementation

Well, an [each[coror]] filter can’t produce a tiddler with a color field. But since the screenshot shows the correct version, this seems to be just an unfortunate typo that very likely got propagated from the issue title into the first post via copypaste.

I agree that the 5.1.0 documentation is less confusing.