Introduce Multi-valued Variables and Let Filter Run Prefix - Examples

[TW_Tones]

I am confident new lines are permitted in a filter especially between filter runs.

However this is not so common for myself as I tend to break complex filters into functions

[Mohammad]

Yes, you can store intermediate results into new variables and use them later inside you filter runs.
See edited OP for examples.

[Mohammad]

The original post is a wiki now.
Few examples have been added.

Please add more examples.

[Yaisog]

Examples with subtract[] or divide[] are better than those with add[], since for the latter you can use sum[] in most cases (e.g. Example 1), which has no equivalent for subtraction or division.

[Mohammad]

Thank you @Yaisog. I revised the Example.
By the way have you noticed the new :let ignores error in filter? For example:

[Tested on TiddlyWiki Pre-release — 5.4.0-prerelease]

<$let diff= {{{ [tag[Learning]get[text]length[]sum[]]   :let[[smL]] 
                [tag[HelloThere]get[text]length[]sum[]] :let[[smH]]
		[<smL>subtract<smH>]
			    	}}} >

The difference is: <<diff>>
</$let>

Produces: The difference is: 49437

Now, intentionally remove a bracked from second filter expression, e.g. instead of [tag[HelloThere]get[text]length[]sum[]] :let[[smH]] use tag[HelloThere]get[text]length[]sum[]] :let[[smH]] still it works, I expect a filter error here. Look at the below one:

<$let sample= {{{ tag[Learning]get[text]length[]sum[]]   :let[[smL]] 
				  [<smL>addsuffix[/somthing]]
			    	}}} >

<<sample>>
</$let>

Produces tag/somthing

[TW_Tones]

I am playing with the new filter “run let”, and the [(varname)] form to to “get a measure” of what this enables, and which of the current challenges it stands to overcome.

• This is the “multi-valued variable” part not yet illustrated above.

@Mohammad

By the way have you noticed the new :let ignores error in filter? For example:

I think this is because of;

The :let filter run prefix always clears the current result list.

I would hope we can illustrate what it enables for a more general audience.

[Yaisog]

@Mohammad

By the way have you noticed the new :let ignores error in filter?

I wonder if that’s intentional. Filters are already hard to debug with only the means provided by the core, not getting an error message makes this harder. I’d think it’s less predictable what you would actually get from a faulty filter. Maybe @saqimtiaz or @jeremyruston care to comment.

[pmario]

Please create an issue at GitHub, otherwise it will be forgotten here.

[Yaisog]

@Mohammad

Now, intentionally remove a bracked from second filter expression

If you put this in a tiddler in 5.3.8 (not having a leading bracket):

{{{ tag[Learning]get[text]length[]sum[]] }}}

you get

tag
0

There is no error, just unexpected output. So maybe :let doesn’t ignore errors after all…

[Mohammad]

Yes, you are right. I checked the example on TiddlyWiki Pre-release — 5.4.0-prerelease

<$list filter="[tag[Learning]get[text]length[]sum[]]    
               tag[HelloThere]get[text]length[]sum[]]" variable=diff>
							 
The difference is: <<diff>>
</$list>

Produces:

The difference is: 59333

The difference is: tag

So it seems TW here ignore the filter error too.

[Yaisog]

Hi @Mohammad, here is a real-world example for why MVVs and :let are so fantastic:

\function .transfer-function(tapsData,sampleRate:1e9)
	[[{"x":$1$,"y":$2$}]] =>[[data-point-template]]
	[<tapsData>get[text]enlist-input:raw[]] =>[[filterTaps]]
	[(filterTaps)count[]log[2]ceil[]] :map[[2]power<currentTiddler>] [[2048]] +[maxall[]] =>[[outputLength]]
	[(filterTaps)fft[magnitude],<outputLength>] =>[[transferMagnitude]]
	[<outputLength>divide[2]negate[]] =>[[negativeSamples]]
	[<outputLength>divide[2]subtract[1]] =>[[positiveSamples]]
	[range<negativeSamples>,<positiveSamples>divide<outputLength>multiply<sampleRate>] =>[[fValues]]
	[(transferMagnitude)last<negativeSamples>] =[(transferMagnitude)first<negativeSamples>] +[log[10]multiply[20]search-replace[Infinity],[1000]] =>[[HValues]]
	[<data-point-template>substitute(fValues),(HValues)join[,]]
\end

It’s a function that computes the filter transfer function for an FIR filter whose (possibly complex-valued) taps are saved in the tapsData tiddler. It outputs a JSON array (minus the outer brackets) of objects, one for each frequency point, ready for plotting with e.g. Observable Plot.

It takes the taps, calculates the next highest power of 2 of their count, computes and shifts the FFT, collects data for both axes and then puts it all together.

This could have been done (and initially was) with procedures, functions, lots of format:titlelist[]join[] and enlist[], but is now all within a single function that’s easy to copy-paste without missing half of the definitions. Marvellous!

Unfortunately, it can’t be used as example on tiddlywiki.com, because it uses a custom fft[] operator and a modified substitute[] operator that can handle MVV parameters (generate a templated output item for each item in the parameter lists). But man, are MVVs and => nice additions.

Final output looks like this (uses a custom Observable Plot widget):

[Mohammad]

Wow! This is amazing! I can’t believe such a complex operation can be done with just one function.
Thank you!

[TW_Tones]

@Yaisog

Unfortunately, it can’t be used as example on tiddlywiki.com, because it uses a custom fft[] operator and a modified substitute[] operator that can handle MVV parameters (generate a templated output item for each item in the parameter lists)

Thanks for this excellent example of what you can do. I just thought it worth pointing out even if few filter operators handle the “multivalue variable” that fact is filters are already good at handling lists. So as long as a list can be stored let and retrieved [(varlist)] then 90% of the possibilities are realised.

Although areas for enhancement that come to mind are;

• Enabling parameters to such functions, that generate lists [(func.name tagname)] however this can be achieved another way, just as a function.
• Provide a shortform that can return the list (for a quick peep) <(varlist)> or <<(varlist)>>

I could see there there would be value the community brainstorming filters and syntaxes that we may be able to add to the Multi-values variables handling.

Some play

\function input.list() [all[shadows+tiddlers]tag[$:/tags/ViewTemplate]]
\function 2nd.func() [(input.list)addprefix[tag: ]format:titlelist[]join[ ]]

{{{ [(input.list)] }}}

{{{ [2nd.func[]] }}}

<<input.list>>

<<2nd.func>>

{{{ [<2nd.func>enlist-input[]] }}}

[stobot]

One of the things that I think let will make cleaner is when you need to compare the values that result from multiple :or situations. Using map only keeps the result for one more filter run, so if you need an :or in a later comparison, then you have to start using helper functions.

In TiddlyWiki, there are 
<$text text={{{ [prefix[A]] [prefix[B]] +[count[]] =>AB 
                [prefix[X]] [prefix[Y]] +[count[]] =>XY 
				[<AB>compare:number:gt<XY>then[more]else[fewer]] }}}/> 
tiddlers that start with A or B than there are that start with X or Y. 

BONUS: If you use a font with coding ligatures (In modern Windows, try “Cascadia Code”) then the shorthand operator is an actual arrow which looks great!

I might suggest that Cascadia Code be added somewhere in the list of default fonts for editing ahead of Consolas - I’m not sure how often this is revisited, but Cascadia Code is now the default font for Windows Terminal. I’m using it as my primary TiddlyWiki font these days.

[Springer]

@stobot

Cascadia Code is now the default font for Windows Terminal

It’s also available as a google webfont: https://fonts.google.com/specimen/Cascadia+Code

In case any of y’all haven’t seen my site detailing use of google fonts for TiddlyWiki (tips, tricks, draggable WYSIWYG per-font tiddlers once you grab the setup tiddlers), here’s a little plug: Google fonts — for TiddlyWiki

EDIT TO ADD: It’s not easy to find the list of ligatures supported by Cascadia Code. I did find it here. Pasting for reference:

[Yaisog]

Here is another real-world example that would be much more complex without :let:

\function .csv-column(tiddler,n,removeBlank:"yes")
	"^[\d,.\s-]+$" =>[[numberRegex]]
	[<tiddler>get[text]trim[]splitregexp[\n]] =>[[lines]]
	[<lines>first[]!search:title:regexp<numberRegex>] :then[<lines>butfirst[]] :else[<lines>] =>[[lines]]
	[<lines>] :map[split[,]nth<n>] =>[[items]]
	[<removeBlank>match[yes]] :then[<items>!is[blank]] :else[<items>]
\end

This gets column number n from tiddlers with CSV data.
One of the neat things here is that I can now define regular expressions that contain square brackets, curly brackets and all that inline via a variable instead of having to declare a procedure outside that I might forget when copy/pasting.
It is also easy to remove header lines that don’t contain numbers (line 3) or selectively remove empty items (line 5), all within a single function!

Update 2026-01-04: Changed the line split regex from \s to \n as discussed below.

[Scott_Sauyet]

I can’t test at the moment, but is this line correct?

@Yaisog

	[<tiddler>get[text]trim[]splitregexp[\s]] =>[[lines]]

It really looks as though it should be \n instead of \s.

[Yaisog]

Good catch! This is a remnant from an earlier version where the data could also be tab- or space-separated. \s whitespace does include newlines, though, and does work.

[Scott_Sauyet]

@Yaisog

\s whitespace does include newlines, though, and does work.

Sure, but it would get wonky with data like this:

London, 12, 13.5
New York, 17, 12.2
Paris, 9, 16.7

Even if you removed the spaces between column values, the space in New York would presumably throw it off.

[Yaisog]

You’re right and I already changed it. The data this was made for was just numbers without spaces (for plotting) so I never had any problems. As an example, this should be more robust, of course.