Understanding the filter run prefix concept

#1

Surprisingly, https://tiddlywiki.com/#Filter%20Run%20Prefix does not contain a formal definition of the concept.

What is the relationship between a filter run prefix and a filter operator suffix present in a filter step? The latter has a definition (albeit hidden in another tiddler: https://tiddlywiki.com/#Filter%20Step ):

The suffix is additional text, often the name of a field, that extends the meaning of certain operators.
#2

Just to get the fundamentals straight: what is a prefix in “filter run prefix” ?

There are different kinds of filter runs. The filter run prefix tells us what kind of filter run we are about to do.

Great. But that might not make sense if we don’t clearly understand what a filter run is. Are we cool with what that is ?

#3

what a filter run is

The analogy in my mental model is from physics. A filter is the molecule. A filter run is the atom. And the filter step is the elementary particle. Albeit this analogy is not “clean”. TiddlyWiki structures are unidimensional.

#4

I think I am going to have a hard time discussing filter runs with the analogy of molecules and atoms.

For me, a filter run is a process. The process receives an input (a list of items), possibly filters out items that do not meet (or do meet) a particular set of criteria and/or alters the items somehow (actually transforming the items and/or sorting them), and produces an output (a list of items based on the input items.)

To describe filter run concepts via a physics analogy, we are going to need somebody else chiming in.

#5

Very little, except that a filter run that has a filter run prefix can contain an operator with a filter operator suffix. For example:

[tag[HelloThere]] :intersection[search:title:casesensitive[TiddlyWiki]]

Example 1

image

Here we have two filter runs, [tag[HelloThere]] and :intersection[search:title:casesensitive[TiddlyWiki]]. The :intersection is a filter run prefix: it precedes an entire filter run and modifies the relationship between filter run #1 and filter run #2.

For comparison, here’s the results of these same two filter runs without any filter run prefixes:

Example 2

image
image480Ă—721 20.8 KB

If you’ve ever studied set theory in math or logic, I think it’s a good way to conceptualize some basic filter runs.

  • When you include multiple unprefixed filter runs in the same filter, the filter’s output is the union of the results of each individual filter run. (This is technically identical to using the :or filter run prefix on each filter run after the first—but, to be honest, I’ve never seen a real-life example of :or.)
  • Using the :intersection filter run prefix on the second filter run gives you the intersection of filter run #1 and filter run #2. This is the six-result list we saw above.
  • The results of a multi-run filter are deduplicated, by default—so while the six results in Example 1 appear among the results of both individual filter runs, they don’t appear twice in the Example 2 results. To prevent deduplication, we could use the filter run prefix :all (or its shortcut =):

Example 3

image
image828Ă—446 14.1 KB

There are several additional filter run prefixes, and not all of them map so neatly onto set theory concepts, but I hope this illustrates the general idea: Filter run prefixes modify the relationship between filter runs.

Let’s set the filter run prefix aside for now and break down the content of the second filter run.

[search:title:casesensitive[TiddlyWiki]]
  • filter step: search:title:casesensitive[TiddlyWiki]
  • filter operator: search
  • filter parameter: [TiddlyWiki]
    • parameter type: literal string (as indicated by the [] around the parameter)
  • filter operator suffix(es): :title:casesensitive

A filter step = one filter operator (with optional suffixes) and 1+ parameters, as indicated by the operator’s documentation.

  • search takes a single parameter; search-replace takes two.
  • A few filter operators such as unique and enlist-input do not take parameters at all: they’re used in the forms unique[] and enlist-input[]. Any content that appears in what would be the parameter slot is simply ignored.

I used search for this example specifically because it is one of the filter operators that does use (optional) operator suffixes, as indicated in the docs:

The search operator uses an extended syntax that permits multiple fields and flags to be passed:

[search:<field list>:<flag list>[<parameter>]]

field list: a comma delimited list of field names to restrict the search

  • defaults to tags, text and title if blank
  • an asterisk * instead of the field list causes the search to be performed across all fields available on each tiddler
  • preceding the list with a minus sign - reverses the order so that the search is performed on all fields except the listed fields

flag list: a comma delimited list of flags (defaults to words if blank)
parameter: filter parameter

(Personally, I find the use of <angle brackets> in operator syntax diagrams somewhat confusing: none of these are variables in the TW sense, and you should not be using angle brackets in this way in real filter steps. But this is the convention the docs use, so it’s best to get used to it. IMHO, a visual designation like color-coding might be clearer if we could do it in an accessible way.)

In any case, in my example [search:title:casesensitive[TiddlyWiki]]…

  • the field list is title (If I omitted the title list, as in search::casesensitive[TiddlyWiki] or search[TiddlyWiki], the default title list title,tags,text would be used instead.)
  • The flag list is casesensitive. (search is case-insensitive by default, so search[TiddlyWiki] and search[tiddlywiki] would produce the same results.)

Filter operator suffixes modify the default behavior of filter operators. Some operator suffixes (like those used by search and field) include the names of tiddler fields; others, like compare and prefix, do not.

  • You can get a full list of filter operator suffixes by pasting this filter into the Filter tab of Advanced Search: [has[op-suffix]]

And here’s one final twist: A filter run prefix can take its own suffix, as we see with :map and :sort. For instance, I used the suffixed filter run prefix :map:flat in the multi-run filter I wrote for you the other day (as well as the “basic” filter run prefix :else/~ in the associated function).

4 Likes
#6

Filter run prefixes modify the relationship between filter runs.

This is exactly the missing thing I was complaining in OP, thank you! Trying to come up with a definition of a concept when getting introduced to it through implementation details and usage examples is seldom easy (for me at least). The statement in bold quoted above fills the gap, just like the other one about filter operator suffixes.

I used the suffixed filter run prefix :map:flat in the multi-run filter I wrote for you the other day

Believe it or not, but this is the actual reason why I posted this topic. I made myself a note (in TiddlyWiki of course!) to review this. Started with https://tiddlywiki.com/#Map%20Filter%20Run%20Prefix , went down the rabbit hole to more general concepts, until I got stuck.

3 Likes