In need for a cascade:all[] Filter Run Prefix

#1

The :cascade[] filter run prefix is super efficient for selecting tiddlers that match any of the filters given in the operand. However, I sometimes need to select tiddlers that match all the operand filters. I didn’t find an efficient solution to this problem yet, although I’m sure it is possible to write a recursive solution. Here is my use case below.

Say we’d like to maintain a recipes database in TiddlyWiki. Among the set of tools we want is a query builder that would help the database manager to create named queries from a list of conditions, without prior knowledge of the TiddlyWiki filter language. The list of conditions that make up a query may look like this to the database manager:

Capture d’écran 2024-03-08 à 17.24.50
Capture d’écran 2024-03-08 à 17.24.501542×328 53.3 KB

Let’s also say that we store the information for each condition in a dedicated tiddler. Then, its subfilter field may get updated with a corresponding filter whenever the user modifies a condition, using default values if needed:

  • subfilter value for condition 1: [get[title]] :and[search:title<user.input>]
  • subfilter value for condition 2: [get[Origin region]] :else[[World]] :intersection[list[World!!regions]]
  • subfilter value for condition 3: [get[Preparation time]] :and[compare:integer:lt[1000]]
  • subfilter value for condition 4: [apply.query[No meat]]
  • subfilter value for condition 5: etc...

(Note that a previously defined query can also serve as a condition in another query.) The list of all the condition filters from a given query is easily built with this helper function:

\function condition.subfilters(queryName) 
    [<queryName>listed[]tag[conditions]]
:map[get[subfilter]]
\end

If the query is meant to be an OR query, we expect it to return a result if any of the query filters matches an input item. In order to apply this query to all the recipes in the database, we would use such a construct:

[all[tiddlers]tag[recipes]!is[draft]] :filter[suchAs.any<selected.query>]

Since our suchAs.any[] function has to return a result if any of the filters from the given query matches the current tiddler, calling on the :cascade[] filter run prefix is a suited wikitext solution. This works like a charm:

\function suchAs.any(queryName) 
        [all[]] 
:cascade[condition.subfilters<queryName>] 
 :filter[!is[blank]]
\end

However, if we’d want to build an AND query, we’d rather need a suchAs.all[] function that applies all of the condition filters from the selected query to the input list, likeso:

[all[tiddlers]tag[books]!is[draft]] :filter[suchAs.all<selected.query>]

This suchAs.all[] function would need to pipe every filters of the query in turn and return a result only if they all match the input tiddler. Something functionally equivalent to:

\function suchAs.all(queryName) 
       [all[]] 
:filter[condition.subfilters<queryName>first[]] 
:filter[condition.subfilters<queryName>nth[2]] 
:filter[condition.subfilters<queryName>nth[3]] 
:filter[etc.]
:filter[condition.subfilters<queryName>last[]] 
\end

This is where I’m stumbling.

This problem looks like it is begging for a recursive solution in wikitext, but it would have to recurse on both the input list and the filter list, which, after many attempts, seems beyond my reach. Can you suggest a method for such intricated recursions?

I did find rather convoluted and slow workarounds, but nowhere near as straightforward nor effective than a cascade:all[] filter run prefix. It would apply in turn each filter defined in its operand to its input list, until it finds a filter that doesn’t match the input tiddler, in which case it would return the empty string, or the last filter result in case of success. If such filter run prefix existed, I could define my suchAs.all[] function as simply as:

\function suchAs.all(queryName) 
            [all[]] 
:cascade:all[condition.subfilters<queryName>] 
     :filter[!is[blank]]
\end

Needless to say, programming such a filter run prefix in JavaScript isn’t up my street either!

#2

@xcazin you are presenting a somewhat challenging idea here for using existing filters from a cascade another way. I will need to work through this in more detail later, but my initial idea would be a different approach.

  • Custom Functions and custom filter operators work not unlike a cascade because they tend to return the first (non blank) value.
  • If all your filters were defined as custom operators (including . in their names), they could be used to replace both the cascade filters and appended together to construct another filter.
    • Filter operators can be used with or without the subfilter operator
    • They can use other values and parameters, if necessary.

I am keen to clarify, develop and if necessary to introduce tools to help with the manipulation of filter definitions such as to build “filter engines” to handle queries as you are doing.

  • I suppose what I am suggesting is to define your filters in a way they can easily be combined and used in different ways.
#3

I may not following correctly where you are heading, but I could indeed explore that route by using for instance .AND[1], .AND[2], … .AND{!!count} with an .AND function taking as its parameter the sequence number of each condition in the query.

I could then recurse on this sequence number rather than on the filter list itself, which could ease the coding. Is this what you had in mind?

#4

My idea was;

Where you have the filter defined in a separate tiddler, instead put in a function, then rather than use the subfilter operator use the function as a custom operator instead.

  • But I see you are using some functions already.

Without working through your whole solution, which may take me some time, unless you can give us a simple example we can run on tiddlywiki.com ? Here are just a few possible approaches;

  • Use the new <$ if %> and Conditional Shortcut Syntax so structure with an if/else and else if rather than a cascade.
  • To ask if this, AND that, AND the other you just need to string the filters together, you can also ask not eg [operator[value]operator[value]operator[value]] unless all are true there is no result, including if you use not ! conditions [operator[value]!custom.operator[value]!operator[value]]
  • So in your case you want to “join” a number of filters together inside the one outer [...] this can be done with the subfilter operator OR using custom. Functions.

Note I still have not worked throught the bigger picture.

#5

I think this is a reasonable proposal. We need to review the name “all”, I think it may be a bit misleading because it doesn’t necessarily run “all” of the filters, it stops when it gets to the first failure.

It might help to think up the suffix that would be used to indicate the current behaviour. One possibility would be “:cascade:first” on the basis that it returns the first successful result.

Or, maybe the current behaviour might be called “:cascade:success” on the basis that it returns the first successful result, and your proposal be “:cascade:fail” on the basis that it returns the first failed result.

#6

Then in reference to greedy and lazy regex quantifiers, may I suggest :cascade:greedy[] whereas the current cascade behaviour tends to be :cascade:lazy[]?

#7

Actually, greedy is not quite appropriate, since it would imply that the operation returns the last successful result.

:cascade:insist[] would probably be better as it conveys that the operation will always try to apply the next filter until failure or exhaustion.

#8

cascade:first-of[] … cascade:last-of[] leaving space for cascade:some-of[] should it ever appear necessary.

#9

From @jeremyruston and @CodaCoder suggestions, I’m wondering if I caracterised the expected behaviour of this operation correctly.

I get the impression that your name proposals describe situations where the operation would return the last matching filter result, whether or not it was the very last one.

I’m actually looking for an operation that aims to apply all the filters defined in its operand to its input list, so that it can help decide if all criteria of a query are met (like in my suchAs.all() function above).

Per similarity with the :cascade[] filter run prefix, it probably should return the empty string upon failure, that is as soon as its input list is exhausted. Upon success (that is if the input list is not exhausted after having been applied every filters in the operand), the return value would arguably be the value returned by the very last filter.

#10

You can certainly ignore my input. Something that was said spawned my thought applying to the general case. Cascades confuse me. In fact, getting my head around someone else’s filters usually involves me having to use @Yaisog’s debug filter (pure gold). :confounded:

1 Like
#11
  • This is what I thought, I think coming from the cascade approach is influencing your perspective on your filter design.
  • What I think your looking for is the way filters work “out of the the box” where “all criteria of a query is met”, [operator[value]operator[value]!operator[value]]
1 Like
#12

@TW_Tones your approach leads to a third option, besides wikitext recursion and a new :cascade[]-like filter run prefix: build the query filter string out of the condition filter strings. As you noted earlier, the filters that describe a condition are usually more complex than operator[value], so if we want to simply concatenate them, we would need to load them through functions.

Which means that in order to produce such a [function.1[]function.2[]function.3[]...] filter, we should maintain dynamic function definitions out of the database administrator actions, instead of piain filters. That’s certainly feasible, but this looks cumbersome to me.

#13

When I use this term, I am referring to custom operators that contain within them the filters needed to generate a larger filter. perhaps custom.operator[] is more appropriate. Custom operators can be strung together.

Also;

:filter[condition.subfilters<queryName>first[]] 
:filter[condition.subfilters<queryName>nth[2]] 
:filter[condition.subfilters<queryName>nth[3]] 
:filter[condition.subfilters<queryName>last[]]

If we know how many items here we can use something like

[range[1],[4]]

condition.subfilters<queryName>nth<range-variable>]

I will let you know when I have more…

#14

Hi everyone,

Here is an update on my quest for converting in real time a set of choices entered via the TiddlyWiki user interface into filter-based queries.

I was missing an operator that would test whether all filters given in the operand apply to its input tiddlers, hence the title of the post.

I initially overlooked the :reduce[] filter run prefix because it would only return its result after testing all tiddlers on all filters, even when one of the filters wouldn’t match. When testing several thousand tiddlers on a few filters, the performance was mediocre especially on refresh when one of the tiddlers or one of the filters changed.

Then I finally realised that I could take advantage of the initial value provided by the reduce operator (not the filter run prefix). Now I could define my suchAs.all() function that way:

\function suchAs.all(queryName)
 [all[]] :filter[condition.subfilters<queryName>apply.all.to<currentTiddler>]
\end

provided that I gave it the following helpers based on the reduce operator:

<!-- apply.all.to reduces the filter list to the tested tiddler <t> if they all pass the test, 
     or the empty string [] if at least one fails -->
\function apply.all.to(t) 
  [all[]] :and[reduce<while_applies>,<t>] :and[!is[blank]]
\end

<!-- <accumulator> is initialised with the tiddler <t> being tested ; -->
<!-- If all the filters pass, this initial value survives and <t> is returned ; -->
<!-- When one of the filters doesn't match, <accumulator> gets forced to [], -->
<!-- so that the next filters are not tried, which cause the reduce operator to returns the empty string []. -->
\procedure while_applies() [<accumulator>!is[blank]filter<currentTiddler>else[]]

This solution proves to be very efficient, so a new operator doesn’t seem necessary after all! Thanks to all of you who scratched their head on this problem with me :pray:

#15

Nice work and clever use of reduce.

I had trouble understanding how to use the apply.all.to and I came up with this simple working example to help me:

<$list filter="cat bat can cap cave" variable=input>
{{{"[length[]match[3]]" "[prefix[ca]]" :and[apply.all.to<input>]}}}
</$list>

results in cat can cap in the output as the only inputs which meet both conditions of having 3 letters and starting with ca.

The above uses the “trick” of surrounding the input filters with quotes in order to prevent them from being evaluated as filters.

See it in action on the share site.

1 Like
#16

Wow

Thanks @btheado This may solve issues I have raised elsewhere, when trying to document and reuse filters, or construct filters.

When used with join it actually is an approach to concatenation and avoids substitution or backtick attributes.

  • We may need to document this.

What a trick :clap: :nerd_face: 

\function construct.filter() "[" "length[]match[3]" "prefix[ca]" "]" +[join[]]

<<construct.filter>>

Returns: [length[]match[3]prefix[ca]]

  • This may even permit a rewrite of the solution to this topic?
  • Note you can even include evaluated filters for variables, fields etc…

[Edited]
If you use <<construct.filter>> as the filter to a list widget it operates as a filter. So it can be viewed or used.

We can say;

  • Did you know you can use literals in functions?
  • Functions to prepare strings, tooltips and more.
  • Functions to generate plain text strings
1 Like
How To: Fun with functions beyond TiddlyWiki 5.3.0 release
Plea for a new :as filter run prefix
#17

Thank you so much @btheado it’s a very valuable trick! I was indeed looking for an Occam’s razor way to demo apply.all.to, and you came up with the perfect one!

1 Like
#18

Great, it’s been a very interesting thread.

That is indeed very useful, and I don’t think anyone has put those pieces together before.

1 Like