Expanding MVV compatibility: a feature wishlist

[etardiff]

In a similar vein to V5.4.0 prerelease - Clearifications about MVVs using with various operators… I thought I’d start a new conversation for post-5.4.0 possibilities.

As an avid user of +[format:titlelist[]join[ ]] + enlist<var> constructions, I’ve been excited about the potential of multi-valued variables since they were first teased, and this week I’ve finally found some time to start converting a lot of my old functions, filters, and $let assignments to use the new syntax. This has enabled me to convert a lot of code like this…

\function concepts() [tag[Concepts]] +[format:titlelist[]join[ ]]

<$let variables={{{ [tag[Variables]] +[format:titlelist[]join[ ]] }}}>

{{{ [enlist<concepts>] [enlist<variables>] }}}

into the following simpler alternatives:

\function concepts() [tag[Concepts]]

<$let variables={{{ [tag[Variables]] }}}>

{{{ [(concepts)] [(variables)] }}}

Which is great! Except that I keep running into instances like this…

<$let variables={{{ [tag[Variables]] +[format:titlelist[]join[ ]] }}}>

{{{ [[A]match[A]then<variables>enlist-input[]] }}}

At present, the only filter operators that accept MVVs as parameters (as per the documentation) are title and function. This means that in a scenario like the one above, I can’t actually redefine <<variables>> as an MVV, because I can’t replace <variables>enlist-input[] with (variables). then doesn’t support MVVs, so then(variables) will return only the first title in the list, not the entire list I need.

In this very simplified example, I can replace then with :then as a workaround:

<$let variables={{{ [tag[Variables]] }}}>

{{{ [[A]match[A]] :then[(variables)] }}}

But this isn’t always a viable solution, especially in more complex multi-run filters.

This leads me to a question for our devs: Is it possible to extend additional filter operators to accommodate MVV parameters?

And a discussion question for other users: Which other filter operators would you like to use with MVVs?

My personal wishlist:

• then, as discussed above
• other operators that can take space-separated lists, like fields:exclude/fields:include or search::words (where each value in the MVV would constitute a “word” — I expanded on this in a later post)
• sortby could take a MVV in place of a title list
• perhaps contains could be similarly extended so that contains:tags(var) would return tiddlers containing all the MVV values in their tag field?

@etardiff

• either a getvariable suffix like getvariable:mvv or a separate operator getmvv. In either case, I’d expect the new form to output the full list of titles defined by the MVV, e.g.

\function test() [tag[HelloThere]]

{{{ [[test]getvariable[]] }}} --> A Gentle Guide to Tiddlywiki
{{{ [[test]getvariable:mvv[]] }}} --> A Gentle Guide to TiddlyWiki Discover TiddlyWiki ...

@etardiff

I imagine this is an unlikely dream as the conditional shortcut syntax automatically applies limit="1" to the underlying $list widget… but it’d be really cool if it also assigned a MVV ((condition)) containing the full list of output values.

[TW_Tones]

@etardiff

fields:exclude/fields:include or search:words (where each value in the MVV would constitute a “word”)

I am just wondering if MVV’s different values are better described as “titles” rather than “words” however I have not being able to explore the true nature of MVV’s yet to properly understand them deeply. How are they internally delimited?

[etardiff]

@TW_Tones

I am just wondering if MVV’s different values are better described as “titles”

Yes, probably! And generally speaking I’d tend to opt for “strings” rather than “titles”, which might imply that values correspond to extant tiddlers — though obviously this isn’t really true in filter logic, and doesn’t completely describe the way we use constructions like {{!!title}}.

In this case, I used “words” in scare quotes because I was thinking about search::words (i.e. the default form of search) in particular. With the current filter syntax, we can do something like this:

{{{ [search:title[filter operator]] }}}

or

<$let terms="filter operator">

{{{ [search:title<terms>] }}}

The default words mode treats its parameter as a space-separated list of tokens which must all be present in each result, so both of these examples will find tiddlers whose titles contain both “filter” and “operator” — but not necessarily in that order.

I’m imagining an expanded version of search which could accept a MVV in place of a standard variable, something like this:

<$let terms={{{ [[filter operator]] variable }}}>

((terms))

{{{ [search:text(terms)] }}} = {{{ [search:text:literal[filter operator]search:text[variable]] }}}

I’d expect this to return all tiddlers that contained the strings “filter operator” and “variable”. This would allow us to search for multiple tokens which could themselves include whitespace.

[etardiff]

Today’s wishlist items:

• either a getvariable suffix like getvariable:mvv or a separate operator getmvv. In either case, I’d expect the new form to output the full list of titles defined by the MVV, e.g.

\function test() [tag[HelloThere]]

{{{ [[test]getvariable[]] }}} --> A Gentle Guide to Tiddlywiki

{{{ [[test]getvariable:mvv[]] }}} --> A Gentle Guide to TiddlyWiki Discover TiddlyWiki Some of the things you can do with TiddlyWiki Ten reasons to switch to TiddlyWiki What happened to the original TiddlyWiki? Donations Funding TiddlyWiki Open Collective

[etardiff]

I imagine this is an unlikely dream as the conditional shortcut syntax automatically applies limit="1" to the underlying $list widget… but it’d be really cool if it also assigned a MVV ((condition)) containing the full list of output values. I’m imagining something like this:

<% if [tag[HelloThere]] %>

<<condition>> --> A Gentle Guide to Tiddlywiki

NOT CURRENTLY FUNCTIONAL:
((condition)) --> A Gentle Guide to TiddlyWiki, Discover TiddlyWiki, Some of the things you can do with TiddlyWiki, Ten reasons to switch to TiddlyWiki, What happened to the original TiddlyWiki?, Donations, Funding TiddlyWiki, Open Collective

<% endif %>

This would enable us to use the full set of titles in the conditional’s content a bit more conveniently than the current workaround, which still requires format:titlelist[]join[ ]:

<% if [tag[HelloThere]] +[format:titlelist[]join[ ]] %>

{{{ [enlist<condition>]first[]] }}}  --> A Gentle Guide to Tiddlywiki

{{{ [enlist<condition>] }}} --> A Gentle Guide to TiddlyWiki Discover TiddlyWiki Some of the things you can do with TiddlyWiki Ten reasons to switch to TiddlyWiki What happened to the original TiddlyWiki? Donations Funding TiddlyWiki Open Collective

<% endif %>

[TW_Tones]

I am starting to delve into MVV’s so as to deeply understand them and contribute to this discussion;

Here are some apparent facts, please edit if untrue, code snipits use on Tiddlywiki.com

MVV’s are comma delimited values ?

<$let items={{{ [all[tiddlers]tag[Welcome]sort[]] }}}>

((items))

</$let>

Community, Find Out More, GettingStarted, HelloThere, Quick Start, Testimonials and Reviews, TiddlyWiki on the Web

Now add this inside the above let widget and !sort[]

<<items>> 

TiddlyWiki on the Web

((items||:))

TiddlyWiki on the Web:Testimonials and Reviews:Quick Start:HelloThere:GettingStarted:Find Out More:Community

An attempt to see the native representation still only returns the first value;

<$text text=((items))/>

TiddlyWiki on the Web

I have used ChatGPT to see if it can help document ans analyse the use of MVV’s. I recently had Great success in a complex problem area by getting the LLM to provide test code, paste the result in and had the LLM determine what was or was not working.

It has helped but not really publishable, and ChatGPT remains not very good at TiddlyWiki script, although it is very good at writing JS modals if you give it an example.

The key gaps in MVV’s that seem true are;

The filtered attribute or back ticked attributes can not handle MVV’s

• We need an additional method to handle (var) in backticks

The payload=<<source>> passes MVV in many cases

first filter result only is returned

payload=`${ [(source)] }$`

There seem to be more inconsistencies but there are sufficient already to make it somewhat unclear.

[etardiff]

@TW_Tones

MVV’s are comma delimited values ?

I think it’s probably more accurate to say that MVVs can be rendered as comma-delimited values. Looking at the “parse tree” preview mode on TW-com, we can compare the way <<test>> is rendered…

vs. the way ((test)) is rendered:

This gives us a clue as to what’s happening behind the scenes: ((test)) is effectively equivalent to <$text text={{{ [(test)join[, ]] }}} />.

In fact, the parser rule mvvdisplayinline highlights a interesting footnote from the introductory Multi-Valued Variables tiddler:

Wiki rule for inline display of multi-valued variables and filter results.

Variable display: ((varname)) or ((varname||separator))
Filter display: (((filter))) or (((filter||separator)))

The default separator is ", " (comma space).

That is, it’s the (( )) or ((( ))) syntax that renders comma-delimited lists. ((( [tag[HelloThere]] ))) will be rendered as <$text text={{{ [tag[HelloThere]] +[join[, ]] }}} />.

Similarly, you can use || + an alternative delimiter, as in ((test|| and )) or ((( [tag[HelloThere]] || and ))), to switch the parameter used for join; these would be rendered like <$text text={{{ [(test)join[ and ]] }}} /> and <$text text={{{ [tag[HelloThere]] +[join[ and ]] }}} />, respectively. (I note that, unlike a {{{ [tag[HelloThere]] || TransclusionTemplate }}}, the spaces following || are considered part of the parameter; ((test|| and )) and ((test||and)) are not interchangable.)

Personally, I might use ((test)) (with a defined MVV “test”) for filter-testing purposes as a quick alternative to {{{ [(test)] }}}. But I’m most interested in the potential applications of MVVs in filters.

@TW_Tones

The filtered attribute or back ticked attributes can not handle MVV’s

• We need an additional method to handle (var) in backticks

I also ran into this issue the other day. I suspect the issue underlying issue may not be backtick substitution… it’s that most widgets simply don’t know what to do with MVVs as attribute values. You can see the same thing happening with <$text text={{{ [(test)] }}} />, which likewise returns only the first result… though to be fair, I’m not sure what else I would expect it to do.

[pmario]

@TW_Tones

MVV’s are comma delimited values ?

No. Internally MVVs are arrays of values. The existing title-lists is a space separated list of tiddler titles. That’s why we need [[title with spaces]] to separate them.

So the difference is internal. If you use [tag[HelloThere]] =>ht-list the ht-list variable internally is handled as an array. Not a title list.

If you try this in AdvancedSearch, you will get back the list

[tag[HelloThere]] =>ht-list [(ht-list)]

If you try the enlist operator with it, you get a list of the words of the first title.

[tag[HelloThere]] =>ht-list [enlist<ht-list>]

The main advantage of MVVs is that you can reuse the variable later in a filter string. So it is like an inline “stack”. eg:

The following example gives you the tags of all tiddlers tagged: HelloThere

[tag[HelloThere]] =>ht-list [(ht-list)tags[]]

HelloThere and About. But what if in your filter you need the original list back.
You can use a function to recreate it, which will need to rerun the ´[tag[HelloThere]]´ filter or you can use the variable that is stored an the “named stack”

[tag[HelloThere]] =>ht-list [(ht-list)tags[]] =>ht-tags [(ht-list)]

Now both lists are on the internal “stack” for later use

Hope that makes sense.

[TW_Tones]

And this array is a JavaScript array with {“a”,“b”} and will it be escaped if we use " and , in the items?

[pmario]

Yes a JS array ["a", "b"] and yes string escaping will be handled by JS, so users do not need to care about it. What you put in, you will get back.

[btheado]

@etardiff

At present, the only filter operators that accept MVVs as parameters (as per the documentation) are title and function. This means that in a scenario like the one above, I can’t actually redefine <<variables>> as an MVV, because I can’t replace <variables>enlist-input[] with (variables). then doesn’t support MVVs, so then(variables) will return only the first title in the list, not the entire list I need.

Since functions work with mvv parameters you can construct a generalized, short-hand helper for passing mvv as parameters to operators which don’t support it. This helper can take care of the format:titlelist[]join[ ] and enlist-input[] for you.

Here is an implementation. It’s first parameter is the name of a function and the second parameter would be the mvv:

\function mvv.op(func param)
  [all[]] =>input
  [(param)format:titlelist[]join[ ]] =>param
  [(input)function<func>,<param>enlist-input[]]
\end

However, built-in operators can’t be called as functions, so you need some wrapper functions like this:

\function op.then(param) :all[then<param>]
\function op.fields.exclude(param) :all[fields:exclude<param>]
\function op.fields.include(param) :all[fields:include<param>]

Then you can use mvv.op like the following. Here is the then example you provided:

{{{
a b c [[d e f]] =>variables
[[A]match[A]mvv.op[op.then],(variables)]
}}}

and the other two operators:

{{{ 
text created =>fields
[[HelloThere]mvv.op[op.fields.include],(fields)]
}}}

{{{ 
text created =>fields
[[HelloThere]mvv.op[op.fields.exclude],(fields)]
}}}

Not as readable as if mvv were supported natively by these operators, but maybe still an improvement?

I didn’t think it through whether the same will work for sortby, contains and search::words. If format:titlelist[]join[ ] and enlist-input[] is needed, then they should work.

Edit: see and play with the above examples on the share site.

[etardiff]

@btheado

Since functions work with mvv parameters you can construct a generalized, short-hand helper for passing mvv as parameters to operators which don’t support it.

This is a very clever use of functions! Unfortunately, I don’t think it’d save much in terms of either character count or code transparency… and since it’s still using format:titlelist[]join[ ]] + enlist behind the scenes, I wouldn’t expect improved performance, either. IMO, there’s not much reason to update working code unless the new alternative is more convenient.

[btheado]

@etardiff

I don’t think it’d save much in terms of either character count or code transparency… and since it’s still using format:titlelist[]join[ ]] + enlist behind the scenes, I wouldn’t expect improved performance, either.

Sure. You found an alternate approach for then then[] which doesn’t require format+enlist. Looks like an alternate approach for fields can also be implemented:

\function mvv.then(v) [all[]] :then[(v)]

\function mvv.fields.include(fields) :all[fields[]] :intersection[(fields)]
\function mvv.fields.exclude(fields) :all[fields[]] :except[(fields)]

{{{
a b c [[d e f]] =>variables
[[A]match[A]mvv.then(variables)]
}}}

---

{{{ 
text created =>fields
[[HelloThere]mvv.fields.include(fields)]
}}}

---

{{{ 
text created =>fields
[[HelloThere]mvv.fields.exclude(fields)]
}}}

I suspect these are the easy cases and I also suspect you will remain unconvinced. Just sharing for completeness.

Updated share site link for the above.

[etardiff]

@btheado

You found an alternate approach for then then[] which doesn’t require format+enlist.

Yes — but my broader point was that, rather refactoring extant code to use MVVs with helper functions, it makes more sense to me to continue using the <<variable>> patterns already supported by core operators until those operators have native MVV support.

That said, I appreciate your demo and your contributions!

[pmario]

@etardiff

… it makes more sense to me to continue using the <<variable>> patterns already supported by core operators until those operators have native MVV support.

That said, I appreciate your demo and your contributions!

MVVs are still in early stage. Not every operator supports them. It would be good to get feedback, what users actually need, to simplify their complex filters. So we can improve them.

Our filter runs can get more and more complex in one line. Which from my point of view should not be the goal. Using combinations with functions and filter runs is much more debuggable. So a good combination will probably be easier to understand and maintain.

[etardiff]

@pmario

It would be good to get feedback, what users actually need, to simplify their complex filters. So we can improve them.

Absolutely! I’d hoped this thread might be a helpful repository for this sort of feedback. I’ve been editing my own thoughts into the OP as they occur to me, and I’d like to add others’ needs as well… should they share them.

Relatedly: I realize 5.4.0 came with a lot of changes; it’s very possible that most people simply haven’t had time to digest them yet. But do you think the topic might get more attention in Discussion? I’d put it in Developers as I assumed it might be a bit technical for casual users, but I’m happy to move it if you think it’s appropriate.

[btheado]

@etardiff

Today’s wishlist items:

• either a getvariable suffix like getvariable:mvv or a separate operator getmvv. In either case, I’d expect the new form to output the full list of titles defined by the MVV

I’m very much in favor of this.

[TW_Tones]

I am wondering is MVV’s deduplicate titles. How we stop it doing so for example a collected list of numbers etc…

Or cause it to deduplicate.

Another thought is transforming a list in an MVV to another form or order than using that independently or against the original set.

Do MVV’s work as expected in the $parameters widget, or could the param variable name be used to set an MVV?

I notice there is no pragma form of the MVV as there is procedure/define, the question is perhaps can we move away from our original titles lists into the MVV world and fully inhabit that?

[etardiff]

@TW_Tones

I am wondering if MVV’s deduplicate titles.

Yes, this happens by default.

I’m not currently aware of any mechanism to prevent deduplication in an MVV.

@TW_Tones

Another thought is transforming a list in an MVV to another form or order then using that independently or against the original set.

Yes, MVVs make this very convenient!

@TW_Tones

Do MVVs work as expected in the $parameters widget

Good question! I hadn’t tested this one before, but while it’s not explicitly called out in the documentation, $parameters does seem to be among the variable-setting widgets that work with MVVS:

Generally, all the methods for setting variables implicitly set multi-valued variables, with the exception of the Set Widget.

Conveniently, this also works with parameters declared as part of a pragma:

<<proc>> shows the default value of the variable/MVV param, which I’ve defined as an MVV in the parameter list. <<proc param=((alpha))>> defines param using another MVV.

• Since ((alpha)) is a variable, it requires the new param=((alpha)) syntax; <<proc "((alpha))">> does not work, for the same reasons that <<proc "<<myVar>>">> wouldn’t.
• On the other hand, <<proc param=<<alpha>> >> does work just like <<proc param=((alpha))>>. This isn’t explicitly called out in the MVV documentation, but it makes sense to me that it would work the same way as $let…

The $let widget also allows the complete list of return values from a function to be assigned to a multi-valued variable. For example: <$let varname=<<my.func>> >

However, I’m trying to get in the habit of using ((alpha)) whenever I invoke \function alpha() in a variable definition, even when <<alpha>> would technically produce the same results. IMO, it’s useful to have the visual clue as to whether I’ve written a given function to be used as an MVV or as a title list.

@TW_Tones

I notice there is no pragma form of the MVV as there is procedure/define

No, \function fills this niche. All functions automatically define an MVV (though if a given function uses +[format:titlelist[]join[ ]] to generate a title list, the corresponding MVV will only have a single (title list) value.

@TW_Tones

can we move away from our original titles lists into the MVV world and fully inhabit that?

I hope so! That’s why I’m trying to compile a list of gaps that would need to be filled to realize this goal.

[tw-FRed]

@etardiff

@TW_Tones

I am wondering if MVV’s deduplicate titles.

Yes, this happens by default.

I don’t think so:

\function test() 1 2 3
\function a() 1 2
\function b() 2 3
\function c() [(a)] [(b)]
\function d() =[(a)] =[(b)]

`((test))=` ((test))

`((c))=`((c))

`((d))=`((d))

results in:

So I think deduplication is a result of c() filter syntax, not MVV, as opposed to d() where we prevent deduplication with = prefixes.

Hope this helps,

Fred