Is it the Time to Have Pipeline in TiddlyWiki Filter Language?

#1

Now with the release of 5.3.0, TiddlyWiki will be more powerfull than ever. Now TiddlyWiki has parameterized transclusions and a decent set of filter operators, filter run prefixes, custom filters and widgets. Is it the time to be able to use Less Complex filter expressions for More Complex tasks? I mean it should be possible to have shorter and simpler and more understandable filter expressions for complex tasks.

One of my wishes is to have pipeline (see Powershell Pipeline). While pipeline in TiddlyWiki is implemented as :map filter run prefix, but it can be improved and can have shorthand operator, or some new one can be developed on top of :map.

Short description

Combining filter expressions into pipelines in the TiddlyWiki

Long description

A pipeline is a series of filter expressions connected by pipeline operators (|) (ASCII 124). Each pipeline operator sends the results of the preceding filter expression to the next filter expression.

The output of the first filter expression can be sent for processing as input to the second filter expression. And that output can be sent to yet another filter expression. The result is a complex filter expression chain or pipeline that is composed of a series of simple filter runs/expressions.

Filter Question: APA Style Title Case Capitalization
#2

The :map filter run prefix pretty much does what we need from pipeline. A shorthand can be added as below using | character.

filter expression-1 | filter expression-2 | filter expression-3

Example

<$list filter=" filter expression-1 | filter expression-2 | filter expression-3 ">
...
</$list>

Note that, the :map gives access the input title through currentTiddler. This variable enables you to build pipeline.

#3

I think it would be good to do something like this and I actualy built one that converted each filter into a series of nested lists. Comming from a background in windows batch programing, incedently each thing you pipe into was called a filter.

The truth is, its is now possible to achive and approximation of this with map and filter/filter runs.

However I feel there is more room for innovation including placing internediate results into variables a bit like the counter.

Certainly 5.3.x will provide opportunities but it does raise the question, “when will we implement these in functions and custom widgets and when should it move to the core?”

#4

Hi @Mohammad you described the pipeline as follows:

Of course, that is how ordinary filter runs work: the output of each filter operator in the run is passed as the input to the next filter operator.

The subfilter operator can be used to make a pipeline out of complete filter runs:

[subfilter<expression1>subfilter<expression2>subfilter<expression3>]

Would your needs be met by making the vertical bar be a shortcut syntax for the map filter run prefix? So,

[all[tiddlers+shadows]tag[foo]] | [has[bar]]

Would be equivalent to:

[all[tiddlers+shadows]tag[foo]] :map[has[bar]]

Is that correct?

1 Like
#5

Hi @jeremyruston
Thank you for your input.

That’s true. Also in pipeline one needs to have access to the input variable. Fortunately, TiddlyWiki supports this in the :map filter run prefix.

I think the key point here is having access to currentTiddler variable.

#6

@Mohammad on reflecting on your post and @jeremyruston’s response, I have realised even before 5.3.x and definatly once we have 5.3.x there are a number of ways to address the pipeline, or series of filters idea, with existing mechanisiums including as I mentioned;

And as a result I think I have answered my own question;

The answer being “not yet”, because we still have a lot of oportunities already, that we can explore first. Once we explore these posibilities we will be better informed and be in a better position to answer your question;

Is it the Time to Have Pipeline in TiddlyWiki Filter Language?

Again my answer would be "not yet"

So what am I talking about? well this is still a set of ideas I/we can explore but they are largely intuitions at the moment, here is some hints, which I will explore further;
  • As mentioned; Using map filter run(s), filter filter run(s) or others.
  • Previously mentioned use of nested list widgets, which allows intermetiate variables to be available, an often neglected approach, when we have a habit of craming solutions into single filters.
  • I have created a filter operator synonym for subfilter of Ć’ which makes @jeremyruston’s subfilter easier to read/write.
    • [Ć’<expression1>Ć’<expression2>Ć’<expression3>]
    • Or perhaps a pipe synonym | eg; [|<expression1>|<expression2>|<expression3>] note the leading |
  • Using set, vars or let widgets to provide “subfilters” to a combined filter.
    • This is effectivly a series of filters with each step stored in a variable.
    • Potentialy avoiding the need to nest list widgets as an example.
  • using the let widget’s evaluation order with the values derived from “filtered transclusions”.
  • Make use of SetMultipleVariablesWidget and possibly ActionSetMultipleFieldsWidget
    • These allow a pair of filters $names(s) and $value(s) to be used in one statement and return N results.
    • We could use integers or “list variables” for the variable/fieldname names.
  • Did you know | is also valid macro name? <<|>> in filters <|>
    • So in 5.3.0 .| or |. can also be a custom filter operator.

Extreamly speculative;

  • If a filter could output keyword=value pairs we can then use elsewhere.
    • Possibly a variation of the SetMultipleVariablesWidget
#7

This is very clever! I did not think we can use several subfilter in series like this. I will add some practical examples later using this approach.

1 Like
#8

Thank you @TW_Tones for your inout! The main idea here, we need to process the inputs in several stage in series. Each stage includes several filter expressions (which it has several filter steps). subfilter seems do the job as long as we can work with internal variable name and map when explicitly need currentTiddler var. I will add some practical examples later.

1 Like
#9

We need a way to hand-off more than one result (or channel). For instance you might have one filter that gets the name of a product, and another that gets the name of the company that makes the product. Today we can’t distinguish the two because the output is just the title of a tiddler or <currentTiddler>. What if had a an operator like "store<OUTPUT variable name>"

e.g.

[[filter that finds product]store[product]] [[filter that finds company]store[company]] +[[ is made by ]addprefix<product>addsuffix<company>]]

By doing it this way, you don’t need two nested list filters – everything can be done in one filter run expression.

3 Likes
#10

i think this would be very useful! i remember learning TW filter syntax and wishing i could nest filter operators, and i know i am not alone in that. an operator like store[] is a lot more intuitive than setting variables externally to the filter run!

#11

This would be so helpful! I’m running against this limitation so often right now.

#12

The ability to set and retrieve variables within a filter expression has been discussed a fair bit amongst core developers going back quite some time. The challenge is coming up with a consistent, clean and backwards compatible implementation. With parametrized transclusions in place, hopefully we can find the time to work on an implementation after the release of TW 5.3.0. In the meantime, any PRs for this would be very welcome!

2 Likes
#13

Hi @Mark_S

Absolutely, yes, I am very keen to introduce the ability to define variables within a filter expression, and something like the “store” operator syntax you suggest would work well.

One question is what precisely is stored in the variable and in what format. We would want to be able store not just a single result, but also the option of storing an entire list of results. Perhaps an optional suffix that could be “list” or “single”.

The obvious choice for storing a list of results would be to use the familiar core stringified list format, where list entries are separated by whitespace, and any entries containing whitespace are wrapped in double square brackets.

However, the trouble is that the stringified list format doesn’t work for certain strings. For example, it is not possible to express strings that contain the substring ]]. That does seem problematic given how common the string ]] is in TiddlyWiki.

We could perhaps fix the stringified list problem by introducing some new escaping scheme, but that would not be backwards compatible. If we’re going to break backwards compatibility, it may be better just to adopt an existing format, such as a string array in JSON format. For example:

\define json-list() ["elephant", "entry with space", "]]"]

To retrieve the contents of such an array one can use [<json-list>jsonget[]].

I’m also interested in being able to define local functions within filters, which would of course just be a special type of variable assignment.

2 Likes
#14

Maybe a special variable type for filter use that stores titlelists/arrays and then is retrieved by a special function (e.g. “recall”). My imagination is failing me here a bit. This is a use case I can think of:

[[long filter expression for historical weather tiddlers]store:list[almanac]][[recall[almanac]subfilter<filter-for-highest-temperature>][[recall[almanac]subfilter<filter-for-lowest-temperature>]] +[join[|]]

It’s beginning to remind me of the Perl obfuscated code contests they used to hold :slightly_smiling_face:

#15

I know my responce previously verbose but it is a comprehencive reply to this subject because its being a long standing interest of mine. I urge you too look closer because it contains a number of approaches to the logical need for pipelining filters, based on current code and hints at a solution in 5.3.x

A pipeline in filters may be desirable but it is importiant to note there is a community bias to cramming things into single filters when there are posibly more performant and easier methods to use and read the code outside a filter.

If you look at nested lists, vars and setmultiple variables you can see such logical structures are already available so in 5.3.x we may construct a custom widget that delivers a superior solution without adding more complexity to filters.

#16

I reminds me of similar too, because this can already be done in let and nested list widgets, the complexity is all just to get them into single filters. For example the C language and its variants are not as broadly used, because the coding functions are compressed down into single character switches, sure you can write very concise code, but by so doing you increase the prerequisite knowledge to use it.

Also when we drive more into filters, when existing solutions exist, we imply to other users, they must master an “increasingly complex filter” syntax, when in fact they do not need to.

  • I am in favor of improving filter functionality, just not at the cost of usability or neglecting existing code patterns.
  • Sometimes I do speculate on what we could do, but otherwise I only assert things to be true given a deep level of understanding.
  • For some reason many people ignore some of my arguments, not even contesting them. And although the contents of my responses may be verbose they are only so to avoid confusion, yet they are often more concise, than some of the arguments I respond to.

Just speaking what I perceive to be true.