Feedback on documentation of filter operators

[CodaCoder]

@Scott_Sauyet

But gleaning that from the documentation and/or the examples is tricky.

My point, exactly.     

[Scott_Sauyet]

@etardiff

Scott will have to weigh in with the JS comparison, but here’s my layperson’s understanding:

That was a bad guess on my part. This has nothing to do with each/forEach found in JS itself or certain libraries. While I see reasonably well now what it does, I have no real idea why it exists. Does anyone have real-world examples of using each?

[etardiff]

@Scott_Sauyet

Does anyone have real-world examples of using each?

I often use each[field]get[field] for prepopulating $select dropdowns with all the values I’ve previously used in a field.

• For instance, I have a number of tiddlers that summarize external sources, where the name of the source is stored in the source field.
• I can add a segment to my EditTemplate that lets me type a new value for the source field or quickly select from the list of sources I’ve used before:

@@.combobox
	<$edit-text field=source tag=input placeholder="type or select" />
	<$select field=source>
		<$list filter="[each[source]get[source]sort[]]">
			<option>{{!!title}}</option>
		</$list>
	</$select>
@@

(.combobox styling courtesy of @telumire)

In this case, I’m using the source field to store a single title only, so each[source]get[source] is appropriate. But elsewhere in my wiki, I also use an authors field which uses title-list format. For instance:

title: The Elements of Style
authors: [[Strunk, William Jr.]] [[White, E. B.]]

If I wanted a list of all the authors represented in my wiki, I’d use [each:list-item[authors]sort[]].

[vuk]

Filter operator that selects one tiddler for each unique value of the specified field.

Surprisingly, this is concise and at least I get the feeling that I understand[1] This is quoted straight from each.js.

With suffix “list”, selects all tiddlers that are values in a specified list field.

This follows immediately and… is a total black hole again for me!

I got curious and ran a git blame on that file. Very interesting findings:

1. Apparently this inline minidocumentation was added a couple of years after the original code.

2. Commit dates show that the list suffix functionality was added later as well (here Operators work inconsistently · Issue #3117 · TiddlyWiki/TiddlyWiki5 · GitHub)

---

1. Note: I mean I understand the “how” part, not the “why” (as in where would I use this) part as well.

[Scott_Sauyet]

@etardiff

@Scott_Sauyet

Does anyone have real-world examples of using each?

I often use each[field]get[field] for prepopulating $select dropdowns with all the values I’ve previously used in a field.

Interesting. Is there an advantage to this over [get[field]unique[]]?

@etardiff

(.combobox styling courtesy of @telumire)

Damn, that looks like it could be useful! Do you have to do any more than add that stylesheet and use your code? My initial attempt only shows the SELECT, without the text input.

ComboboxTest.json (2.1 KB)

[etardiff]

@Scott_Sauyet

Interesting. Is there an advantage to this over [get[field]unique[]]?

I’m actually not sure how they compare, performance-wise. My gut feeling is that each[field]get[field] may be faster.

… and some quick tests in my largest wiki (25360 total tiddlers, 1740 tags) support that:

Here’s the data from Maurycy’s Advanced Performance plugin. I ran each filter three times, and you can see that the each filter is consistently more efficient at scale—though the difference may be less noticeable in a smaller wiki.

Edit: Just for fun, I also compared [each:list-item[field]] with [get[field]enlist-input[]unique[]]:

And each:list-item was so much faster that I couldn’t get them both onscreen at the same time.

With unique[] vs. each:value[], the difference was less dramatic…

… but again, each wins out. So I suppose the reason to use it over other constructions that yield the same results is simply: efficiency! And I may need to go refactor some code now.

@Scott_Sauyet

Do you have to do any more than add that stylesheet and use your code? My initial attempt only shows the SELECT, without the text input.

I end up tweaking nearly all the code I borrow, so it’s very possible I’ve changed it in some incompatible way. Here’s the .combobox style from my own stylesheet:

.combobox {
--dropdown-button:20px;
 position: relative;
 display: inline-flex;
 padding-right: var(--dropdown-button);
	background-color: #fafaf9;
 border-radius: 5px;
	width: 10em;
}

.combobox input {
	background-color: #fafaf9;
	width: 10em;
}

.combobox select {
 position: absolute;
 /*the width of the select will be the width of the dropdown*/
 width: 100%;
 /*select is hard to style, so we clip the size we want and hide it, then show a pseudo-element above it*/
 opacity:0;
 clip-path: inset(0 0 0 calc(100% - var(--dropdown-button)));
 pointer-events:all;
 cursor:pointer;
}

In my wiki, that looks like this:

Tested on TW-com, the dropdown caret isn’t visible, but you can type into the input field, and clicking at the right end of the box brings up the dropdown. I’ve restyled the base input elements in my wiki, and I suspect that accounts for the difference; you may need to adjust the width/positioning to get things to line up properly.

[Mohammad]

I like these examples as they are easy to understand, and no other concept is required to know in advance. Those given in the PR are very difficult (they are good for advanced users)

[Mohammad]

Worth to be added to TW official docs.

[Scott_Sauyet]

@etardiff

Here’s the data from Maurycy’s Advanced Performance plugin.

I’m sorry. I didn’t mean to give you an assignment. But thank you very much for checking on this.

Clearly each is valuable. All the more reason to ensure that it’s well-documented!

@etardiff

@Scott_Sauyet

Do you have to do any more than add that stylesheet and use your code? My initial attempt only shows the SELECT, without the text input.

I end up tweaking nearly all the code I borrow, so it’s very possible I’ve changed it in some incompatible way. Here’s the .combobox style from my own stylesheet:

That works for me, subject the arrow-visibility caveat you mention. Thank you very much for sharing. I can see several similar uses of my own for such comboboxes.

[pmario]

@vuk

Purely intuitively, first of them seems to be of significant importance

2. each seems to be my personal nemesis [. . .]

There is a bug in the each[] filter implementation. I am having a closer look at this one at the moment.

I also think it is necessary to have a closer look, where the each[] operator was used the first time in the TW core. This points to the main usecase it was developed for.

So the each-operator help from TW v5.1.0 points us into the right direction. That’s the whole purpose this operator was developed for. So it basically is a helper operator to create “typed-lists”

Now the operator is more powerful, but it’s purpose is pretty much the same.

• It iterates through all tiddlers and
• Returns the first tiddler that has a new eg: type

[vuk]

There is a bug in the each[] filter implementation

Well, an [each[coror]] filter can’t produce a tiddler with a color field. But since the screenshot shows the correct version, this seems to be just an unfortunate typo that very likely got propagated from the issue title into the first post via copypaste.

I agree that the 5.1.0 documentation is less confusing.

[pmario]

Geeks Warning !

@vuk

So far I don’t have any advices how to improve this particular situation here (since I don’t even understand this operator yet!). Perhaps other TiddlyWiki users remember their aha moment (after not getting it after the first read) about it and could share what made it click for them.

For users that can read JavaScript or C-like languages and are willing to dig into the TW test code, it is always worth to have a closer look at the TW test edition

In this particular case: each test coverage is at: TiddlyWiki5/editions/test/tiddlers/tests/test-filters.js at 028c80782d105beb90f5d58a7f22e865c7e8c6f4 · TiddlyWiki/TiddlyWiki5 · GitHub

But it is not very detailed atm. There is an improved version in the new PR – which may help advanced users.

Many of the test in the test-edition are part of the TW documentation. But not all of them.

---

This should be no excuse for missing documentation, but it can help to improve the documentation for those who are able and willing to dig the “rabbit hole”.

-m

[pmario]

@etardiff

It cannot take a parameter (other than, technically, each:value[title]).

• each:value[tags] = each[tags] → the :value suffix is ignored.
• each:value[title] = each:value[] → probably for the same reason that [[missing tiddler]] :filter[{!!title}] returns [[missing tiddler]]; {{!!title}} and <<currentTiddler>> both yield the string output of the filter, whether or not that string corresponds to an actual tiddler with a title field.

You are right. That’s an inconsistency which has potential for more confusion. – It may or may not be fixed. – At the moment I am not sure if it should be fixed.

The implementation of the each-operator is a bit tricky at the moment. It contains nested if()s that are hard to read and debug. From my point of view it should be rewritten.

@etardiff

I’m actually not sure how they compare, performance-wise. My gut feeling is that each[field]get[field] may be faster.

… and some quick tests in my largest wiki (25360 total tiddlers, 1740 tags) support that:

That’s a very interesting catch, but understandable. Running each[] does all it’s work in 1 run. So if used in the right way it can be an improvement for slow filters that need unique lists.

The unique[] filter needs to go through a potentially large list 2 times. We already know that it can be a performance killer.

[etardiff]

@pmario

At the moment I am not sure if it should be fixed.

IMO, it doesn’t need to be. It’s a little odd that each:value[title] is the one exception. But it is consistent with both

• each[] = each[title] (and the current docs, where title is explicitly called out as the default parameter for each)
• and the way TW uses title as equivalent to “the input string”, generally speaking: [title[missing title]] returns “missing title”, even though [field:title[missing title]] does not.

I think it’s a pretty obscure exception, anyway. I didn’t remember having seen each:value in the wild before (in fact, I had forgotten it existed at all before this thread prompted me to read the docs more closely), and a Google search for “each:value[” tiddlywiki just now only turned up two or three real results, including this thread.

@pmario

Running each[] does all it’s work in 1 run. So if used in the right way it can be an improvement for slow filters that need unique lists.

This was my intuition too, but I’m glad to have some numbers to support it—particularly because I also find unique[] much more semantically obvious, and I’ve never seen each:value[] explicitly recommended instead, even though it would be a more efficient 1-to-1 substitution.

If each does get rewritten, I hope the better performance can be preserved!

[CodaCoder]

@pmario Thanks for the diligence. It’ very much appreciated.

Having spent some time digging into the links you provided (thank you!), here’s my take on what I’ve seen. And this I think speaks to the issues that confound readers of the documentation:

The core source documentation, and the user documentation (for each[] in this case) bring about a problem often seen when terms are not clearly defined. It seems that the root cause is the core source documentation, whereby its notion of what things are called and what things are, are not set out clearly. In the case of each[], that same issue seems to have been copied across to the user documentation.

Here are the terms that seem to be sometimes synonyms, and sometimes not:

tiddler → title → value

tiddler is an object with properties { <data> }
title is a string "my title"
value is whatever you say it is in the coding context you are working. It could be a tiddler, or a title, or an array element/member of a list (a list whose properties are unknown outside of the core code’s context). It really could be anything with any possible JavaScript type.

Sadly, value is transplanted from the core source documentation into the user documentation as though the context where it made sense would come right along with it. Bang. Without it, users have no clue (other than guessing) what value truly refers to.

Muddling up tiddler and title happens across user documentation on tw .com. Old-timers like you and I can generally mush them together and deduce what is actually meant. In most cases, we do it without thinking about it. But that’s not what happens in a newbie’s head – which is unfortunate, to say the least

And if this point is lost on anyone, consider any “list of tiddlers” you might think you have in your wikitext/code – it is in actual fact a “list of (tiddler) titles”.

Aside: Where “group” came from, I have no idea. I still don’t get what it means, what the author’s intention was, what it’s referring to or… anything.

Yes, I know PRs for docs are always welcome. But if i don’t understand it, I can’t offer any help.

^Z

[telumire]

I created the combobox stylesheet back in 2021, tiddlywiki has changed how the core svg is handled now, so the demo will no longer work. Here’s an updated version that works with the latest version: combobox

[pmario]

@pmario

But it is not very detailed atm. There is an improved version in the new PR – which may help advanced users.

I did close the PR since the exact issue has been discussed several years ago. I did miss that one. As so often backwards compatibility is the argument here.

---

There is one thing we can be certain. Whenever we do have a problem to explain a functionality in a consistent and logical way, there is a 90% chance that the underlaying concept has issues and “backwards compatibility” adds to our technical dept.

[TW_Tones]

@pmario

Whenever we do have a problem to explain a functionality in a consistent and logical way

In my own involvement with tiddlywiki, I think these issues are often the result of the documentation being minimalistic when a new feature was released. At the time, to those who introduce something it makes sense and is consistent. The problem is in time the unwritten meanings are forgotten.

• If the each operator is an example, I have always being comfortable with it, in part because of what I was trying to do when I first encountered it.
• Its initial design was very focused on field values each[fieldname] and it is not generic, the keyword each was then expanded with each:list-item and each:values, It is worth reading examples and following the link to GroupedLists because this is almost the why of its existence.

To start with we should expand on changes in the releases and expand the documentation a little. If a little history helps then it may be worth while eg;

The each operator was introduced in its each[fieldname] for finding unique values of fieldnames, and returning a/one title containing that value. Thus a common way to use is it to retrive the list of values of a fieldname each[fieldname]get[fieldname].

• Since each is a generic word there was value expanding the each operator through the operator suffixes list-item or value

However the documentation of these operator suffixes are simply too brief. I would make a documentation change but to be responcible I would need to confirm through tests my understanding and that will take some time.

Can I suggest the participants in this thread add just a little to the documentation to improve it?

[Scott_Sauyet]

I wonder if this is helpful. I have a start to the updating the documentation for each especially including some examples, written using the testcase widget. I have also slightly modified the descriptions in the each Operator description tiddler.

You can see this by downloading the following and dragging it onto tiddlywik.com:

EachDocs.json (6.7 KB)

I would like it if these examples could all be shown inside a tabbed interface, but when I tried in several ways, they always showed all the separate parts of the CompoundTiddler in one. I’ll try to figure that out later, but first I want to know if people see this as a useful approach.

I would also love to be using the .operator-example mechanism with its dynamic “Try It” button, but I don’t like that this puts a temporary tiddler inside the CompoundTiddler. That confuses things. But I haven’t spend much time on it.

So, is this worth pursuing?

[CodaCoder]

@Scott_Sauyet

I want to know if people see this as a useful approach.

Trust yourself. That’s of enormous benefit.