Feedback on documentation of filter operators

[Scott_Sauyet]

@CodaCoder

Perhaps @Scott_Sauyet will finish the response he was typing a few minutes ago.

Sure, but it’s not about each. I’ll try to look at each soon, but if it’s what I expect it to be (from my JS experience), then it’s a function I mostly avoid anyway.

[Scott_Sauyet]

Here’s my understanding of each:

If we have ten tiddlers tagged Demo, each with a my-field value, we can note if each one is the first with its specific value:

title	my-field	first?
A	foo
B	foo
C	bar
D	foo
E	bar
F	bar
G	baz
H	bar
I	foo
J	qux

then the filter [tag[Demo]each[my-field]] will return those firsts: A C G J.

But gleaning that from the documentation and/or the examples is tricky. And I haven’t even looked at the two suffixes.

EachDemo.json (1.2 KB)

[etardiff]

@vuk

each seems to be my personal nemesis: after skimming through all other operators (with the exception of the above three without examples), I had a first feeling that I kind of superficially understand how they work (I’m well aware I may be wrong). But after reading over and over the description and examples for each operator, I still don’t get it at all

Scott will have to weigh in with the JS comparison, but here’s my layperson’s understanding:

• each[field] returns each input tiddler (i.e., each tiddler produced by the preceding filter steps, or each tiddler in the set of all[tiddlers] if each is the first filter step) with a unique value of the field field.

  • IMO, this is the most confusing use. From the name alone, I would expect each to return each unique value of field, but this is not what it does; instead, it returns each tiddler that has a unique value. When multiple tiddlers have the same field value, it returns only the first title with that value.
  • each treats field values as literal strings, not title lists: field: A B C and field: C B A are unique field values, even though they contain the same list-items.
  • If you want the field values, not the titles of the tiddlers that have them, you need each[field]get[field].

• By contrast, each:list-item[field] does return field values (not tiddler titles), with a twist: each field value is treated as a title list, and list items that appear in more than one tiddler’s field only appear once in the output of each:list-item.

  • For instance:

    

  • As you can see above, each:list-item[field] is equivalent to get[field]enlist-input[]unique[]

• I’ve never actually used each:value[], but at a glance it seems to be equivalent to unique[].

  • Unlike the other two each variants, this one isn’t concerned with fields at all.

  • Also unlike each, each:value does not care whether an input value is a tiddler title or not. (“Vanilla” each automatically discards all inputs that don’t correspond to tiddler titles, since — by definition — a tiddler that does not exist does not have any fields.)

  • It cannot take a parameter (other than, technically, each:value[title]).

    • each:value[tags] = each[tags] → the :value suffix is ignored.
    • each:value[title] = each:value[] → probably for the same reason that [[missing tiddler]] :filter[{!!title}] returns [[missing tiddler]]; {{!!title}} and <<currentTiddler>> both yield the string output of the filter, whether or not that string corresponds to an actual tiddler with a title field.

  • Demo:
    

    
    
    vs.
    
    

  • Again, I don’t know why you’d use each:value[] when unique[] exists. EDIT: Because it’s (sometimes considerably) more efficient, as I discovered later in this thread.

[CodaCoder]

@Scott_Sauyet

But gleaning that from the documentation and/or the examples is tricky.

My point, exactly.     

[Scott_Sauyet]

@etardiff

Scott will have to weigh in with the JS comparison, but here’s my layperson’s understanding:

That was a bad guess on my part. This has nothing to do with each/forEach found in JS itself or certain libraries. While I see reasonably well now what it does, I have no real idea why it exists. Does anyone have real-world examples of using each?

[etardiff]

@Scott_Sauyet

Does anyone have real-world examples of using each?

I often use each[field]get[field] for prepopulating $select dropdowns with all the values I’ve previously used in a field.

• For instance, I have a number of tiddlers that summarize external sources, where the name of the source is stored in the source field.
• I can add a segment to my EditTemplate that lets me type a new value for the source field or quickly select from the list of sources I’ve used before:

@@.combobox
	<$edit-text field=source tag=input placeholder="type or select" />
	<$select field=source>
		<$list filter="[each[source]get[source]sort[]]">
			<option>{{!!title}}</option>
		</$list>
	</$select>
@@

(.combobox styling courtesy of @telumire)

In this case, I’m using the source field to store a single title only, so each[source]get[source] is appropriate. But elsewhere in my wiki, I also use an authors field which uses title-list format. For instance:

title: The Elements of Style
authors: [[Strunk, William Jr.]] [[White, E. B.]]

If I wanted a list of all the authors represented in my wiki, I’d use [each:list-item[authors]sort[]].

[vuk]

Filter operator that selects one tiddler for each unique value of the specified field.

Surprisingly, this is concise and at least I get the feeling that I understand[1] This is quoted straight from each.js.

With suffix “list”, selects all tiddlers that are values in a specified list field.

This follows immediately and… is a total black hole again for me!

I got curious and ran a git blame on that file. Very interesting findings:

1. Apparently this inline minidocumentation was added a couple of years after the original code.

2. Commit dates show that the list suffix functionality was added later as well (here Operators work inconsistently · Issue #3117 · TiddlyWiki/TiddlyWiki5 · GitHub)

---

1. Note: I mean I understand the “how” part, not the “why” (as in where would I use this) part as well.

[Scott_Sauyet]

@etardiff

@Scott_Sauyet

Does anyone have real-world examples of using each?

I often use each[field]get[field] for prepopulating $select dropdowns with all the values I’ve previously used in a field.

Interesting. Is there an advantage to this over [get[field]unique[]]?

@etardiff

(.combobox styling courtesy of @telumire)

Damn, that looks like it could be useful! Do you have to do any more than add that stylesheet and use your code? My initial attempt only shows the SELECT, without the text input.

ComboboxTest.json (2.1 KB)

[etardiff]

@Scott_Sauyet

Interesting. Is there an advantage to this over [get[field]unique[]]?

I’m actually not sure how they compare, performance-wise. My gut feeling is that each[field]get[field] may be faster.

… and some quick tests in my largest wiki (25360 total tiddlers, 1740 tags) support that:

Here’s the data from Maurycy’s Advanced Performance plugin. I ran each filter three times, and you can see that the each filter is consistently more efficient at scale—though the difference may be less noticeable in a smaller wiki.

Edit: Just for fun, I also compared [each:list-item[field]] with [get[field]enlist-input[]unique[]]:

And each:list-item was so much faster that I couldn’t get them both onscreen at the same time.

With unique[] vs. each:value[], the difference was less dramatic…

… but again, each wins out. So I suppose the reason to use it over other constructions that yield the same results is simply: efficiency! And I may need to go refactor some code now.

@Scott_Sauyet

Do you have to do any more than add that stylesheet and use your code? My initial attempt only shows the SELECT, without the text input.

I end up tweaking nearly all the code I borrow, so it’s very possible I’ve changed it in some incompatible way. Here’s the .combobox style from my own stylesheet:

.combobox {
--dropdown-button:20px;
 position: relative;
 display: inline-flex;
 padding-right: var(--dropdown-button);
	background-color: #fafaf9;
 border-radius: 5px;
	width: 10em;
}

.combobox input {
	background-color: #fafaf9;
	width: 10em;
}

.combobox select {
 position: absolute;
 /*the width of the select will be the width of the dropdown*/
 width: 100%;
 /*select is hard to style, so we clip the size we want and hide it, then show a pseudo-element above it*/
 opacity:0;
 clip-path: inset(0 0 0 calc(100% - var(--dropdown-button)));
 pointer-events:all;
 cursor:pointer;
}

In my wiki, that looks like this:

Tested on TW-com, the dropdown caret isn’t visible, but you can type into the input field, and clicking at the right end of the box brings up the dropdown. I’ve restyled the base input elements in my wiki, and I suspect that accounts for the difference; you may need to adjust the width/positioning to get things to line up properly.

[Mohammad]

I like these examples as they are easy to understand, and no other concept is required to know in advance. Those given in the PR are very difficult (they are good for advanced users)

[Mohammad]

Worth to be added to TW official docs.

[Scott_Sauyet]

@etardiff

Here’s the data from Maurycy’s Advanced Performance plugin.

I’m sorry. I didn’t mean to give you an assignment. But thank you very much for checking on this.

Clearly each is valuable. All the more reason to ensure that it’s well-documented!

@etardiff

@Scott_Sauyet

Do you have to do any more than add that stylesheet and use your code? My initial attempt only shows the SELECT, without the text input.

I end up tweaking nearly all the code I borrow, so it’s very possible I’ve changed it in some incompatible way. Here’s the .combobox style from my own stylesheet:

That works for me, subject the arrow-visibility caveat you mention. Thank you very much for sharing. I can see several similar uses of my own for such comboboxes.

[pmario]

@vuk

Purely intuitively, first of them seems to be of significant importance

2. each seems to be my personal nemesis [. . .]

There is a bug in the each[] filter implementation. I am having a closer look at this one at the moment.

I also think it is necessary to have a closer look, where the each[] operator was used the first time in the TW core. This points to the main usecase it was developed for.

So the each-operator help from TW v5.1.0 points us into the right direction. That’s the whole purpose this operator was developed for. So it basically is a helper operator to create “typed-lists”

Now the operator is more powerful, but it’s purpose is pretty much the same.

• It iterates through all tiddlers and
• Returns the first tiddler that has a new eg: type

[vuk]

There is a bug in the each[] filter implementation

Well, an [each[coror]] filter can’t produce a tiddler with a color field. But since the screenshot shows the correct version, this seems to be just an unfortunate typo that very likely got propagated from the issue title into the first post via copypaste.

I agree that the 5.1.0 documentation is less confusing.

[pmario]

Geeks Warning !

@vuk

So far I don’t have any advices how to improve this particular situation here (since I don’t even understand this operator yet!). Perhaps other TiddlyWiki users remember their aha moment (after not getting it after the first read) about it and could share what made it click for them.

For users that can read JavaScript or C-like languages and are willing to dig into the TW test code, it is always worth to have a closer look at the TW test edition

In this particular case: each test coverage is at: TiddlyWiki5/editions/test/tiddlers/tests/test-filters.js at 028c80782d105beb90f5d58a7f22e865c7e8c6f4 · TiddlyWiki/TiddlyWiki5 · GitHub

But it is not very detailed atm. There is an improved version in the new PR – which may help advanced users.

Many of the test in the test-edition are part of the TW documentation. But not all of them.

---

This should be no excuse for missing documentation, but it can help to improve the documentation for those who are able and willing to dig the “rabbit hole”.

-m

[pmario]

@etardiff

It cannot take a parameter (other than, technically, each:value[title]).

• each:value[tags] = each[tags] → the :value suffix is ignored.
• each:value[title] = each:value[] → probably for the same reason that [[missing tiddler]] :filter[{!!title}] returns [[missing tiddler]]; {{!!title}} and <<currentTiddler>> both yield the string output of the filter, whether or not that string corresponds to an actual tiddler with a title field.

You are right. That’s an inconsistency which has potential for more confusion. – It may or may not be fixed. – At the moment I am not sure if it should be fixed.

The implementation of the each-operator is a bit tricky at the moment. It contains nested if()s that are hard to read and debug. From my point of view it should be rewritten.

@etardiff

I’m actually not sure how they compare, performance-wise. My gut feeling is that each[field]get[field] may be faster.

… and some quick tests in my largest wiki (25360 total tiddlers, 1740 tags) support that:

That’s a very interesting catch, but understandable. Running each[] does all it’s work in 1 run. So if used in the right way it can be an improvement for slow filters that need unique lists.

The unique[] filter needs to go through a potentially large list 2 times. We already know that it can be a performance killer.

[etardiff]

@pmario

At the moment I am not sure if it should be fixed.

IMO, it doesn’t need to be. It’s a little odd that each:value[title] is the one exception. But it is consistent with both

• each[] = each[title] (and the current docs, where title is explicitly called out as the default parameter for each)
• and the way TW uses title as equivalent to “the input string”, generally speaking: [title[missing title]] returns “missing title”, even though [field:title[missing title]] does not.

I think it’s a pretty obscure exception, anyway. I didn’t remember having seen each:value in the wild before (in fact, I had forgotten it existed at all before this thread prompted me to read the docs more closely), and a Google search for “each:value[” tiddlywiki just now only turned up two or three real results, including this thread.

@pmario

Running each[] does all it’s work in 1 run. So if used in the right way it can be an improvement for slow filters that need unique lists.

This was my intuition too, but I’m glad to have some numbers to support it—particularly because I also find unique[] much more semantically obvious, and I’ve never seen each:value[] explicitly recommended instead, even though it would be a more efficient 1-to-1 substitution.

If each does get rewritten, I hope the better performance can be preserved!

[CodaCoder]

@pmario Thanks for the diligence. It’ very much appreciated.

Having spent some time digging into the links you provided (thank you!), here’s my take on what I’ve seen. And this I think speaks to the issues that confound readers of the documentation:

The core source documentation, and the user documentation (for each[] in this case) bring about a problem often seen when terms are not clearly defined. It seems that the root cause is the core source documentation, whereby its notion of what things are called and what things are, are not set out clearly. In the case of each[], that same issue seems to have been copied across to the user documentation.

Here are the terms that seem to be sometimes synonyms, and sometimes not:

tiddler → title → value

tiddler is an object with properties { <data> }
title is a string "my title"
value is whatever you say it is in the coding context you are working. It could be a tiddler, or a title, or an array element/member of a list (a list whose properties are unknown outside of the core code’s context). It really could be anything with any possible JavaScript type.

Sadly, value is transplanted from the core source documentation into the user documentation as though the context where it made sense would come right along with it. Bang. Without it, users have no clue (other than guessing) what value truly refers to.

Muddling up tiddler and title happens across user documentation on tw .com. Old-timers like you and I can generally mush them together and deduce what is actually meant. In most cases, we do it without thinking about it. But that’s not what happens in a newbie’s head – which is unfortunate, to say the least

And if this point is lost on anyone, consider any “list of tiddlers” you might think you have in your wikitext/code – it is in actual fact a “list of (tiddler) titles”.

Aside: Where “group” came from, I have no idea. I still don’t get what it means, what the author’s intention was, what it’s referring to or… anything.

Yes, I know PRs for docs are always welcome. But if i don’t understand it, I can’t offer any help.

^Z

[telumire]

I created the combobox stylesheet back in 2021, tiddlywiki has changed how the core svg is handled now, so the demo will no longer work. Here’s an updated version that works with the latest version: combobox

[pmario]

@pmario

But it is not very detailed atm. There is an improved version in the new PR – which may help advanced users.

I did close the PR since the exact issue has been discussed several years ago. I did miss that one. As so often backwards compatibility is the argument here.

---

There is one thing we can be certain. Whenever we do have a problem to explain a functionality in a consistent and logical way, there is a 90% chance that the underlaying concept has issues and “backwards compatibility” adds to our technical dept.