Feedback requested on proposed new filter operator: "braid"

#1

I’m thinking of adding a new filter operator called “braid” that would braid two lists together, one item at a time. E.g., the output of [braid[1 2 3],[a b c]] would be 1 a 2 b 3 c. This would not make it into the upcoming 5.3.6 release, since Jeremy Ruston has asked that all new features for 5.3.6 be submitted by the end of October, and I won’t have time to work on my “braid” operator idea until November at the earliest. But I hope to get it into 5.3.7.

If you think that a “braid” filter operator might be useful to you, I have a question for you: what would you expect the output of [braid[1 2 3 4 5 6 7],[a b c]] to be?

This is not a quiz where there’s one right answer. I can think of several possible answers, any of which would be “right” in some circumstances and “wrong” in others. What I want to know is what your gut feel would be: what answer would feel most correct to your intuition? My intuition would lead me to answer that question one way, but if there’s a clear majority of answers that point to a different answer, then I would want to take that into account as I design the “braid” filter operator.

So please let me know what you would expect the output of [braid[1 2 3 4 5 6 7],[a b c]] to be, and that will help this operator to be as intuitive as possible once I get it working.

#2

I would expect one of these three possibilities, from the most to least expected:

  1. 1 a 2 b 3 c 4 a 5 b 6 c 7 a
  2. 1 a 2 b 3 c 4 5 6 7
  3. 1 a 2 b 3 c

This proposed operator certainly seems interesting. However I can’t imagine any real life situation in TW where one would use it. Do you have any particular uses in mind?

I imagine it could be used in a list widget of two interlaced element types (e.g. text1 image1 text2 image2), but assembling the filter expression in the form of [braid<how_do_I_insert_a_filter_expression_here>,<another_filter_expression>] wouldn’t make it easier than doing something like this:

<$list filter=<filter_expression_one> counter=counter>
    <<currentTiddler>>
    {{{ [<filter_expression_two>nth<counter>] }}}
</$list>

Though repeating the filter expression two for every $list counter would make it (marginally?) less efficient.

Btw, wouldn’t the name interlace be an alternative to braid worthy of considering?

#3

i’d expect 1 a 2 b 3 c 4 a 5 b 6 c 7 a.

have you written this operator up for use in your wiki? what have you been using it for? i imagine it may be useful in generating key-value pairs for data tiddlers or similar but it does seem like a niche operator. i’m interested to hear what applications it may have that would warrant its ease of use for everyone :slight_smile:

#4

Interesting. Braid’s original name was interleave.

#5

just had an additional thought - this operator could (theoretically) be extended like this:
braid[1 2 3],[a b c],[q w e] >> 1 a q 2 b w 3 c e

is it possible to extend operators more generally for any number of input lists? would that be useful for the use cases you are seeing for braid?

#6

I was wondering about step as a third (or final) operand:

[1 2 3 4],[a b c d],[2] → 1 2 a b 3 4 c d

[1 2 3 4],[a b c d],[w x y z],[2] → 1 2 a b w x 3 4 c d y z

Not sure if that’s a “step” too far :stuck_out_tongue_winking_eye:

#7

There has been discussion of making braid a filter run prefix.

#8

This was the first option that occurred to me, though I can see the case for the others @vilc suggested. I wonder whether it would be too complicated to allow a suffix to toggle between these behaviors?

#9

Indeed. short, long and repeat are being proposed. What @rmunn is asking for, is which should be the default behavior.

#10

The use case that made me want this was when I wanted to use TW to keep track of characters in a role-playing game. Characters have attributes like Strength 3, Agility 2 and so on. Instead of having fields named strength, agility, etc, on each character, I wanted to go with a more generic approach that didn’t require my character-view tiddler to “know” about the details of the game’s stats. (That way I could reuse the same wiki code for a different game where the attribute is called Dexterity, not Agility). So I wanted to have one field called “attributes” containing Strength Agility Sturdiness and a different field called “attribute-values” containing 3 2 4. And then merge those two fields together for display into a result that looked like “Strengh 3, Agility 2, Sturdiness 4”. (Inserting the commas would have made the filter a little more complicated, but quite doable).

One thing I’d been considering is allowing the inputs to braid to be filter expressions, possibly specified with a :filter suffix. E.g,. [braid:filter<filter-expression-one>,<filter-expression-two>] where presumably you’ve defined those filter expressions with <$set> or <$let> earlier. I was thinking about introducing the filter operator first, then adding a :filter suffix later. From your reaction, that might be something people want from the start, so I think I’ll include a :filter suffix right away.

Sure. My original idea was “interleave”, but “interlace” is better that “interleave”. I thought “braid” was even better due to being shorter, but I’m certainly open to suggestions. If there’s widespread agreement on one particular name, great: it means that name most clearly communicates what the filter operator does.

#11

As you mentioned, I was basically using it for key-value pairs, where the keys were stored in one field and the values in another one. My specific data was characters in a role-playing game, who all have Strength, Agility, Sturdiness, and so on. One reason to do it this way is because in my use case, the keys are going to be the same (all characters have Strength, all have Agility) and they’re always in the same order (the game has a standard order for listing a character’s attributes: Strength is always first, Agility is always second, and so on). And so instead of every character tiddler having to store the fields strength: 2, agility: 3, sturdiness: 4 and so on, I could just have each character tiddler only stores the values. Then my tiddler for displaying characters would look something like this:

Attributes: {{{ [braid{DefaultAttributes!!list},{!!attributeValues}] }}}

(That’s a simplified version of the filter because the real one would also insert a comma after each value). Since my TW file would be tracking hundreds of characters (I want to be able to easily look up the strength, agility, etc. of any enemy the players might face), that would translate to a file potentially one or two megabytes smaller, because the field names strength, agility, etc. don’t need to be repeated in every single tiddler. It probably, though I haven’t measured this, makes each tiddler that much faster to load as well.

So saving space and increasing load speed for data-heavy TW files are two reasons why this might be more broadly useful than my specific use case.

Yes indeed, I do plan to allow any number of lists. And it would indeed work as your comment suggests, taking one item at a time from each list in turn, then when all input lists have contributed one item, take a second item from each list in turn. And so on until all the lists have run out. (Or, if the “repeat shorter lists that run out of items” behavior — the one that would produce 1 a 2 b 3 c 4 a 5 b 6 c 7 a — is chosen, then the rule would be “until the longest list(s) have run out”).

1 Like
#12

One use case I had was inserting a comma in between the key-value pairs. In my earlier replies I didn’t show that because it would have made the example harder to read, but using “repeat” mode would have made this possible:

[braid[Strength Agility Sturdiness],[3 2 4],[,]

Producing “Strength 3 , Agility 2 , Sturdiness 4 ,”. And then it’s pretty easy to remove the final comma, and replace the text " ," (space-comma) in the output with “,” (comma with no preceding space) in order to display a nicely-formatted list.

However, if “step” is allowed, it’s impossible to know whether a final input list with one item is supposed to mean “step” or if it’s supposed to mean “braid this single item in between the others, repeating it until the others lists are exhausted”. So as a practical matter, I don’t think [braid[1 2 3 4],[a b c d],[2]] would be possible to implement. Because someone might have wanted 1 a 2 2 b 2 3 c 2 4 d 2. I don’t know why someone would want that, but when you’re designing software for general use, people always have a way of surprising you with use cases you would never have considered.

It might be possible to have a step suffix, though. Since multiple suffixes are allowed in TW filter operators, it would even be possible to say that the step suffix must be followed by another suffix that is a a number, e.g. :step:2 or :step:3.

But I don’t actually think there’s a use case for interleaving lists two or three items at a time, so I don’t think anyone will actually want this. So I think my initial implementation will not do this, and I’ll add it in a later version of TW if enough people actually ask for a :step:N (where N is a number) suffix to be added.

#13

True. My initial thought was last position, single integer, means “step”. That still may not be enough, though it is pretty clear if documented as such.

That’s even clearer. Though I personally am not a fan of lengthy suffixes – if that’s the only sure-fire way, then so be it.

I’m always torn between YAGNI and “build it and they will come”. But KISS usually wins out.

1 Like
#14

The filter run prefix would be unable to handle braiding more than two lists together, because of the way TW internals work: each run prefix gets the input so far and its parameter, but doesn’t know about other run prefixes. So 1 2 3 :braid[enlist[a b c]] would produce 1 a 2 b 3 c as desired. But if you then did 1 2 3 :braid[enlist[a b c]] :braid[enlist[x y z]], you might expect to get 1 a x 2 b y 3 c z, but what you’d actually get is the list 1 a 2 b 3 c braided with x y z, which depending on the default mode would produce either 1 x a y 2 z b x 3 y c z, or else 1 x a y 2 z b 3 c.

Though I’ve just come up with a reason to use that step parameter you suggested! Because if step applied only to the input list (the elements produced so far), then you could do step:2 on the second braid and it would take 1 a and add x, then take 2 b and add y, and so on, and you’d get the 1 a x 2 b y 3 c z result that you would expect.

Hmmmmm. I wonder if I can write up documentation to explain that sufficiently clearly…

#15

Cool.

Lots of examples. I think it’s fine to start with a b c / 1 2 3, but pretty soon your game approach would work much better (or anything with actual words paired with numbers) – I was going “snow blind” trying to follow some of the earlier posts…

#16

@rmunn I think you have a very valid and generalizable problem to solve, but I’m not quite sure that the filter is the best way to solve it. Filter would be necessary if you intended to process the 1 a 2 b 3 c using further filter steps. But in this situation, where the “braided” output is only intended to be displayed, I think using existing mechanisms inc. $list widget would suit better, e.g.

<$list filter="[enlist[DefaultAttributes]]" variable="attribute" counter="counter" join=", ">
    <<attribute>>
    <$text text={{{ [<currentTiddler>get[attributeValues]nth<counter>] }}}/>
</$list>

IMO this would be easier to maintain even if the braid filter existed. You have the advantage of the joiner being taken care of, you can style the attributes and attribute values differently if you like, you can even easily convert it to a table with first column for attributes and second for values.

So I think the problem is a good and universal one. At the same time, the braid filter might also be a universally useful thing. But I’m not sure if such filter would be a good way to approach the problem at hand. Maybe we should wait until another problem comes up, for which the braid filter would be more appropriate and necessary? Especially that the discussed syntax for it is already starting to become quite convoluted.

#17

Images + captions  

#18

Yes, my original problem is better served by a list with join=", ", though I’m pretty sure that I had the idea for braid before join was an option. (In fact, come to think of it, didn’t I write the join attribute? … Yes, I did, but I actually had to go and check.)

But I still think I’d like braid to be an available filter option. There are many list-manipulation functions that are “standard” across many different programming languages. Some of them, like zip (which takes the lists 1 2 3 and a b c and produces a list of pairs, i.e. (1,a) (2,b) (3,c)), don’t make much sense in TiddlyWiki, because title lists don’t have the concept of “pairs”: they can only contain strings. Others, like filter and map, we have now. But interleave is one we don’t have, and I’d like TW to have as complete a set of list-manipulation functions as possible. Because the more available functions you have, the more clever solutions you can come up with to problems that crop up.

… Okay, so of the three possible ways to handle different-sized lists, we’ve had two votes for “repeat”, i.e. “repeat the shorter list until the longer one is exhausted” and one vote for “long”, i.e. “use each list fully, so when the shorter one is exhausted the rest of the results are just the longer one with no more braiding”. Nobody has voted for the default to be “short”, i.e. “stop as soon as the shortest list is exhausted, throwing away the rest of the items from any longer lists”. I’d like to get a few more people responding, to see if others agree with this.

For anyone who doesn’t want to scroll up and read the whole thread, this is what I’m asking for. If you saw a filter written like this: [braid[1 2 3 4 5 6 7],[a b c]], which one of the following three results would you expect it to produce? (I’ve labelled them for convenience in answering).

  • “short”: 1 a 2 b 3 c
  • “long”: 1 a 2 b 3 c 4 5 6 7
  • “repeat”: 1 a 2 b 3 c 4 a 5 b 6 c 7 a
1 Like
#19

I think I’d prefer repeat, but expect long.

And by the way, I assume the longer list can be in either position, correct?
[braid[1 2 3],[a b c d e f g]]

1 Like
#20

Sorry, meant to reply earlier.

My vote:

  1. short
  2. long
  3. repeat

But in my head this is still zip, even if that’s not really useful in TW. My view might be colored by that.

1 Like