Only partially removing duplicates in list

#1

Coniders this code:

[enlist:raw[a b a c d a b x]] [[c]]

c will be de-duplicated and the result will be a b a d a b x c.

If I want to keep both c I have to prepend the second filter run with = as in

[enlist:raw[a b a c d a b x]] =[[c]]

Now we can also observe that

[enlist:raw[a b a c d a b x]] [[a]]`

gives us b a c d a b x a and not b c d b x a as we could have expected. Is that a bug or just something that needs to be well documented?

We have the same things happening with the toggle filter op:

[enlist:raw[a b a c d a b x]] +[toggle[a]]

produces b a c d a b x] and not b c d b x] and here the official doc is nowhere near telling us this way of working. But its true that my example is not the use case intended at the creation of the “toggle” filter op.

Syntax question about keeping duplicates within a field
#2

Hi @jypre

Quoting the Dominant Append documentation, “if a title would be duplicated, the earlier copy of that title is discarded” (emphasis mine).

I’m not a native English speaker, but I can understand this sentence as “if a title would be duplicated, the first copy of that title is discarded”.

In any case, the doc might need clarification on this point.

Fred

#3

And perhaps more appropriate;

“if a title would be duplicated, the last copy of that title is retained”.

This is because unless you go out of the way to do otherwise, each additional title is naturaly appended to the list of titles.

  • If you don’t want this behaviour have a look at the Order Operators but specificaly Extended Listops Filters
  • I have not tested these in detail, in all cases, but the extended listops filters usualy work in their own right even without the action listops widget

for example on 5.3.x try this;

\function list.items() this that the other

{{{ [list.items[]append[end]prepend[start]] }}}

Or using the orignianl examples;

{{{ [enlist:raw[a b a c d a b x]prepend[c]] }}}
  • In this case the prepend is applied in the same filter run as enlist:raw
  • It is documented, you have just missed what RAW means.
  • Of course the documentation could be less frugal

If you want the deduplicated outcome, do not ask for the duplicate outcome!

" dedupe (the default) to remove duplicates, raw to leave duplicates untouched"

Use; without raw

{{{ [enlist[a b a c d a b x]] [[a]] }}}

Gives you

b
c
d
x
a

As desired.

For fun look at this;

{{{ [enlist[a b a c d a b x]prepend[a]] }}}

VS

{{{ [enlist[a b a c d a b x]prepend[a]unique[]] }}}
#4

I will bdd a bit more complexity, to show you some intirestiong but confusing side effect. First, the code and then thfe output.

<$let items= "[[one time]] [[d]] a bit">

{{{ [enlist[a b a c d a b x]] [[a]] }}}

---

{{{ [enlist[a b a c d a b x]] =[[a]] }}}

---

{{{ [enlist[a b a c d a b x]] =[enlist<items>] }}}

---

{{{ [enlist[a b a c d a b x]prepend<items>]  }}}
</$let>

produces

b
c
d
x
a
a
b
c
d
x
a
a
b
c
d
x
one time
d
a
bit
one time
d
a
bit
a
b
c
d
x

first case: why has the first, second and third “a” disappeared? It does not seems to align with what I was told here.

second case: the “=” has preserved the first "a« but the second and third have disappeared nonetheless. Weird!

Third case: same as second case for the “a”’. The “d”, biing only two, are not affected, which is what was awaited.

Fourth case: again similar to second case for “a”. and same effects than third case.

The effect of “=” is really strange if there is more than one element of any added elements. This should be clarified in the rule. Besides, are we sure that this effect is really what was intended?

#5

Sorry for the short answer: you used enlist[] instead of enlist:raw[].
Open tiddlywiki.com and try every step of your filters in Advanced Search, you’ll see what happens and why…

Fred

#6

We are adding [[a]], by default it is appended, and the list is deduplicated.

This is subtle and anoying. the first part of the filter [enlist[a b a c d a b x]] gets deduplicated leaving a b c d x then we append =[[a]] and given the = appends a without removing its duplicate.

In this case the = applies to all “items”, [enlist[a b a c d a b x]] gets deduplicated then the items are appended without removing any duplicates.

In this case [enlist[a b a c d a b x]] gets deduplicated then we immediatly in the same filter run prepend, we are in effect prepending the whole filter run, and further deduplication does not occur. However you could ask it to do so with;

{{{ [enlist[a b a c d a b x]prepend<items>unique[]]  }}}

Giving;
one time
d
a
bit
b
c
x

  • Notice the deduplication preference is to retain the first occurance!
  • The default append process to retain the last occurance.
    • This in part could relate to the reordering drag drop and reordering tools, if you dont know where it belongs put it at the end.

I hope this helps?

#7

@TW_Tones good point! i wast too tired to think about it. My missing trying also the enlist:raw is just a testimony of that.

#8

@TW_Tones Thanks a lot. Yes it’s now very clear to me.

The only sad point is that I had to come here to beg for an explanation. Really, I shall find time to offer a more elaborated doc to make it clear in the docs. No harm intended to current doc authors: it’s not a straightforward solution, I don’t really see how it should be done. I’m just convinced it should be done.

#9

Don’t be too hard on yourself, is a challenge for everyone at first, in part because the default behaviour is what most people need to start with, however when your demands get a little more detailed, you have to go back and reexamine the assumptions you made before.

As much as you want, the community is good at helping, and the best way to learn is to teach so its a virtuos circle.

#10

@TW_Tones You’re damn right. I try to answer from time to time, when I am lucky enough to have a possibility to help someone not helped yet. And indeed helping fellows helps yourself.