Introducing Multi-valued Variables

#1

Over the weekend I’ve been working on a new PR that implements an apparently fundamental change to TiddlyWiki’s internal design: variables may now contain not just a single value, but a list of zero or more items.

In practice, this change is considerably less radical than one might think: for full backwards compatibility, all the existing usages of variables in the core just use the first item in the list.

Nothing actually changes apart from a handful of new features that expose the true nature of variables:

  • A new :let filter run prefix that assigns the current result list to a variable whose name is taken from the first value returned by the filter run itself. The variable can be used in the rest of the filter expression (and any subfilters that it may invoke)
  • Extended the existing <$let> widget to assign multi-valued variables. This is backwards compatible because the existing ways of accessing these variables will still only see a single text value
  • A new varlist[name] that retrieves the value of a variable as a list. It works like the all[] operator in that it replaces the input list with the new list. If the variable only contains a single value then a list of one item is returned

There is a preview build available.

There are further improvements that could be made. An example is that it gives us a way to reduce our dependence on title lists in many situations. Title lists are brittle and cannot properly express some items. For example, it is not possible to store a string like ]] in a title list.

The <$draggable> sets a variable called actionTiddler containing a title list of the dragged titles that is accessible within the associated actions. We could now provide an alternative variable that made the list of dragged titles available as a list variable.

What Problems Does This Change Solve?

There are a few common scenarios that these new capabilities can make much simpler.

Nested Let Widgets

A common technique today is to break up complex filters using a series of assignments. In this example we use an intermediate variable “category” to compute a value that is then used as a parameter to “jsonget”:

<$let
  category={{{ [tf.makecat<currentTiddler>,[8]] }}}
  subcategory={{{ [{schema}jsonget[root],<category>] }}}
>
...
</$let>

This approach (or a similar one) is needed to get around a fundamental limitation that it is not directly possible to use a the result of a filter as the parameter to a subsequent filter operator. We are limited to parameters that are transclusions, or function/macro calls with fixed parameters.

With the changes here we can use a single filter to achieve the same result:

<$let
  subcategory={{{ [tf.makecat<currentTiddler>,[8]] :let[[category]] [{schema}jsonget[root],<category>] }}}
>
...
</$let>

All-in-one Filters

Generally, the advantage of being able to shoehorn a complex, multistep filter into a single filter expression is to be able to use them in the situations that only cater for a single filter. Cascades are an example of this. The example that drove the development of this PR was the new colour handling improvements which redefines palette entries as filters, instead of wikitext as at present.

Background

I’ve been thinking about the ability to assign variables within a filter for a long time; my first notes on the subject are from 2021. It was only after finally implementing the feature on Friday that it became clear that there would be value in being able to store a list in a variable.

As it has turned out, the work has been remarkably straightforward, largely because of the way that the internal implementation of function calls actually treats them as a parameterised variable that returns a list instead of a single item.

Open Questions

I am interested in any feedback, and welcome any questions or thoughts. There are a couple of open questions that have come to my mind:

  • The names :let and varlist[] are poor, and we need to come up with better terminology that is more distinct from existing usages
  • Do we need further terminology to make the documentation clearer? For example, a term for a variable that contains a list versus a variable that contains a single item. The terminology needs to make clear that these are exactly the same type of variable, but with different values; they are not different types of variable
8 Likes
#2

I’m excited to see progress here, particularly helping with “all-in-one” filters is greatly appreciated.

I might be not understanding the syntax, but on the demo site I’m entering something simple like: :let:test[prefix[b]] [varlist[test]] and expecting that the list of tiddlers with the prefix of ‘b’ would return but nothing does… Additionally doing :let:test[prefix[b]] [varlist<test>] actually throws a RSOE for me.

I’m sure it’s too late now, but I was really hoping the sytnax for this would end up being something like [prefix[b]set[test]...]. The thought was that you could store the incoming titles AND let it also keep going to do further transformations, but then in subsequent filter runs you’d have both the <currentTiddler> but also the stored list/value. The syntax would also at least look a little more familiar and you could conceivably use a variable as the name of the variable to set to, meaning [prefix[b]set<varname>...]

#3

This would be so much nicer than appending +[format:titlelist[]join[ ]] to. every. single. filtered. variable. assignment (when working with lists).

Could we extend enlist<name> instead of varlist[name] or is it important for name to possibly be a soft parameter? Maybe both, depending on the type of variable?

Is <$letlist> then similar to <$set>, except without the explicit name and filter attributes?

2 Likes
#4

Spot on. I end up using that idiom quite often. And it’s annoying.

#5

Very exciting. I can’t test vercel apps in my locked-down corporate environment, but I look forward to investigating this evening.

If I’m not mistaken, that would have solved a problem I brought up some time back.

#6

The current list of items is used as the input list for the evaluation of the filter run. Your example could be written as:

:let:test[all[tiddlers]prefix[b]] [varlist[test]]

Alternatively, this is equivalent:

[all[tiddlers]] :let:test[prefix[b]] [varlist[test]]

Ouch, thank you. That is fixed in this commit.

That’s an interesting idea. I think the implementation would be more difficult, and I do wonder whether having a hidden side effect of a filter operator might make it harder to read as opposed to the more distinctive syntax of the filter run prefix. I’ll give it some more thought.

The let filter run prefix currently clears the result list. The thinking was that in most cases one would want to start again, and it was easy to retrieve the result list by accessing the variable.

I did consider making provision for the variable name to be a computed value but struggled to see situations where it would be useful, and there isn’t an obvious syntax to use.

#7

Reusing “enlist” was my first thought. However, with the current code, the “enlist” operator only sees the value of the variable with the given name, and doesn’t have a direct way to be able to tell the name of that variable.

I would think of it as more like the <$let> widget because of the way that <$letlist> allows multiple variables to be set via separate attributes.

#8

I was thinking <$set> because it also assigns a list of filter results to the variable which can then be enlisted. I guess internally that is just a titlelist, not a “true” list like a JS array (which I assume this PR is)?

1 Like
#9

This is very exciting, @jeremyruston! I’m looking forward to removing a lot of +[format:titlelist[]join[ ]], but I can also see this enabling filter searches that weren’t previously possible in Advanced Search, in addition to the :cascade applications you mentioned.

Like @stobot, I would have assumed that all[tiddlers] would be implicit in the :let run if it was the first run in the filter — so I’d expect :let:test[all[tiddlers]prefix[b]] and :let:test[prefix[b]] to be equivalent as the first run, just as [all[tiddlers]prefix[b]] = [prefix[b]]. I think I’m pretty comfortable with filter syntax, but this would definitely trip me up; I use all[shadows] or all[shadows+tiddlers] on occasion, but I can’t remember the last time I used all[tiddlers] in this position.

1 Like
#10

This is a difficult trade off. As you suggest, the expected semantics of the input list for the filter run would be that it would be the same as the input list outside the filter run. However, the cost of this option is that it makes it impossible for a variable assignment to use previously computed values such as:

=[<getsometiddlers>] =[<getmoretiddlers>] :let:myvar[all[]]

The current semantics are definitely unexpected, but I don’t see any other way for the let filter run prefix to access the existing input list. It would be great to have all[outer] or something but the code doesn’t easily permit that.

The ability to process the input list with a let filter run prefix seems important. If we adopted the expected semantics there would be no way to code the above example as a simple filter; we’d be back where we started with the need for intermediate variables.

Conceivably we could add some options to the let filter run prefix to allow the behaviour to be manually specified:

  • :let:varname:all[...] would initialise the input list for the filter run with the equivalent of all[]
  • :let:varname:input[...] would initialise the input list for the filter run with the input list accumulated before the filter run
#11

So :let would be better understood as similar to :filter in terms of its expected input? I think that would make a certain amount of intuitive sense… except that it’s not entirely consistent with :filter, either. As a comparison:

:white_check_mark: {{{ [prefix[b]] }}}
:white_check_mark: {{{ [all[tiddlers]prefix[b]] }}}

:x: {{{ :filter[prefix[b]] }}}
:x: {{{ :filter[all[tiddlers]prefix[b]] }}}
:white_check_mark: {{{ [all[tiddlers]] :filter[prefix[b]] }}}

:x: {{{ :let:test[prefix[b]] [varlist[test]] }}}
:white_check_mark: {{{ :let:test[all[tiddlers]prefix[b]] [varlist[test]] }}}
:white_check_mark: {{{ [all[tiddlers]] :let:test[prefix[b]] [varlist[test]] }}}

It makes sense to me that :filter and :map can’t be used as the initial filter run because they need an explicit input filter; they don’t assume [all[tiddlers]] as the default input. I think the confusing aspect is that :let can evidently be used as the first run of a filter, which leads me to erroneously assume that it will behave more like an unprefixed run.

This seems like a reasonable solution, though you might consider making :input or :all the default value. Personally, I think I’d be more likely to define a variable with :let at the beginning of the filter, so it feels more natural to me to have :let:varname:all = :let:varname, with :let:varname:input as the marked case. But I could see an argument either way.

#12

varlist reminds me of the getvariable operator, with the distinction that the latter operates on its parameter and ignores its input. Perhaps we should keep that in mind when thinking of appropriate nomenclature in the interest of consistency.

What would be the intended behaviour when accessing a variable-list using getvariable[]? I would expect it to be the first member of the list.

a b c :let:alphabet[all[]] [getvariable[alphabet]]

#13

My preferred name choices would be:

  • :let:x[...] → :collect:x[...]
  • varlist[x] → spread[x]

The names have complementary meanings, and spread is also the name of JavaScript’s ... operator.

For the <$letlist> widget, I couldn’t find a better name. Besides, there’s already a variety of widgets to set variables in multiple, slightly different ways. Adding yet another widget type to that mix feels somehow wrong.

It might not be technically possible, but what about extending <$let> and filtered attributes, so that <$letlist x="filter"> could be written as <$let x={{{ :collect[filter] }}}> or <$let x={{{ :let[filter] }}}>, passing the special list value from the filter run to the attribute using a hidden default variable? This would be more orthogonal and composable and could prove useful in other situations, too.

Anyways, I think even without any further changes this already is a huge improvement.

#14

To play devil’s advocate, as someone without any coding background beyond TW, I actually really like that :let and $letlist are clearly related to $let, a widget I already understand and use frequently. One recurring complaint I see from new and intermediate users is that TW has so many widgets and filter operators that the names are difficult to remember. collect and spread may be memorable to those with some JS background; to me, though, they’re brand new terms without any inherent connection to variable handling. I think it would be a mistake to introduce names that differ significantly from the terms we already use when we could instead build on users’ understanding of extant tools that perform related functions.

Also, from a purely practical standpoint, :let is shorter. :stuck_out_tongue:

Of the three, I think varlist is the weakest name (and this is not helped by the fact that, like @Yaisog, I intuitively want to use enlist<name> to enlist any variable I’ve saved as name, regardless of the widget/function/filter I used to do it). But varlist does at least evoke a list of variables, so it’s already doing significantly better than spread. If anything, I’d anticipate some confusion about whether it’s varlist or varslist, as older wikis may still be using the $vars widget.

I’d had some similar thoughts myself. It would be convenient to be able to mix single-value and multi-value variable definitions within a single widget, as you can currently do if you’re defining “multi-value” title lists with name={{{ a b c +[format:titlelist[]join[ ]] }}}. Otherwise, I think I’d often need both $let and $letlist, just as I currently find myself using $let and $set. But I assume this is unlikely to happen, as it would increase the complexity and likelihood of errors when using $let.

2 Likes
#15

I have to admit this took me off guard.

One recurring complaint I see from new and intermediate users is that TW has so many widgets and filter operators that the names are difficult to remember.

This is correct, my feeling is that this is done for providing mechanisms to fill the gaps related to basic tools offered by “traditional” programming languages. For example the $list widget fills the gap of a missing explicit loop construction like for. The <%if> shortcut makes a step towards “programming” by offering an alternative to using then and else filter operators - a syntax that is clearly more “readable” (for me at least) than a piece of code in the middle of a long complex filter etc.

Even this whole thread and the pull request behind it triggers this controversial feeling. Getting better tools for lists handling is great, this is tangential to my older question Working with lists in TiddlyWiki . Yet this is one more thing to learn and start using. Plus the need to refactor old codebases to lower down technical debt. Plus the need to refactor old knowledge sources to recommend new “best practices”.

I kind of understand this is a tradeoof though. Wikitext is trying to fill the gap between “traditional” programming languages and pure markup/templating languages, by being both, and this is not easy at all.

#17

what about using varlist for all 3 purposes. varlist sets a variable which contains a list. so :varlist:test creates a variable named test which contains a list, <$varlist> sets a variable which contains a list and then use <$let-varlist >to set multiple variables that contain lists. to me this would help make code readable as whenever you see varlist, you know that the variable set by that is a variable that contains a list.

#18

Thinking through my historic pain points and where I’ve discussed the need for some kind of storing variable (this thread for example), the way it felt is that only having a single variable (currentTiddler) within the filter was the limit I was trying to overcome.

I’m realizing that I actually had something even more ambitious in mind. This veers off this topic a little bit, but list has always been 1-dimensional in nature - it exposes a single variable, and so other computed values need to be re-calculated within the loop of the widget. If we are filtering real tiddlers then we have syntax shortcuts and efficiencies using the {!!field} notation meaning I can sort by or filter on a field value, but we can’t do that by computed values that are also like properties of the <<currentTiddler>> “object” / record.

And then what if we let these pseudo-fields map within the widget? We kind of get to something more like a SQL situation where the output has “rows” and “columns”. Most of the time I use <$list> it’s within a <table> so this feels very natural.

Today example: Producing a list of tiddlers sorted by their length with both name and computed value put a filter and sort just to show comparison to pseudo-code below.

<$list filter="[prefix[A]] :filter[get[text]length[]compare:number:gt[20]] :sort:number[get[text]length[]]">
<<currentTiddler>>: <$text text={{{ [<currentTiddler>get[text]length[]] }}}/><br>
</$list>

One of the things I’m trying to illustrate here is that get[text]length[] is run not only multiple times within the filter, but then for each entry within the list widget. If that piece is “expensive” / “slow” then this is not ideal.

Pseudo-code below: Storing a computed value in the filter, and then spitting it out of the list widget along with the <<currentTiddler>> variable. I’ll use the word store instead of let to acknowledge that these really do different things.

<$list filter="[prefix[A]] :store:length[get[text]length[]] :filter[<length>compare:number:gt[20]] :and[sort<length>]">
<<currentTiddler>>: <<length>><br>
</$list>

What I’m trying to show in the above is using in-filter notation to store the computed values somewhat as properties of the currentTiddler variable, so that they can be used like they are. It might even be interesting if the syntax was <!!length> in the filter and <<!!length>> within the widget just to illustrate the near-field-like usage.

Maybe this is entirely separate from “multi-valued-variables” but I think there’s some overlap. The above actually doesn’t require variables to be multi-value, so maybe they can be both done with different names.

1 Like
#19

Thank you for the feedback and encouragement.

I have just implemented one minor change that I think goes a little way to reducing confusion. Originally, the <$letlist> widget was intended to be used with string attributes that were interpreted as a filter:

<$letlist
  subcategory=":let:category[tf.makecat<currentTiddler>,[8]] [{schema}jsonget[root],<category>]"
>
...
</$letlist>

Now, I’ve updated things so that the attribute value uses the familiar {{{}}} syntax to specify the filter:

<$letlist
  subcategory={{{ :let:category[tf.makecat<currentTiddler>,[8]] [{schema}jsonget[root],<category>] }}}
>
...
</$letlist>

It’s a small change, but I think it reduces confusion, and makes the <$letlist> widget be a much more precise analogue of the <$let> widget, with the only difference being the use of multi-valued variables.

As a bonus, this change also allows the <$letlist> widget to be used to assign the full result list of a function call directly to a multivalued variable:

\function myfunc() [all[tiddlers]sort[]]

<$letlist varname=<<myfunc>>>
<$text text={{{ [varlist[varname]] +[join[-]] }}}/>
</$letlist>

I will address the other feedback as soon as I can.

1 Like
#20

Thanks @saqimtiaz I think we could drop the varlistoperator and instead overload the getvariable operator. It would be backwards compatible because there’s no way for existing code to produce a multi-valued variable. I’ll look into it further, but it does seem desirable to reduce the number of novel entities introduced by this change.

#21

We may yet get there; the next step for TiddlyWiki might well be to support lists of lists, which would allow us to model both tables and other interesting structures.

Interesting idea. It may be possible to find an approach that would let us do this. We’d have to take a slightly different approach for filtered attributes versus situations like the <$list> widget where the widget itself computes the filter. I’ll give it some thought.