Speed of the !tag[] filter operator vs tag[]

Developers
#1

The performance instrumation gave me surprisingly long times for some !tag filter runs, so I did a direct comparison (on a wiki with 8,000+ total tiddlers):

[!tag[$:/tags/Journal]count[]] 	95.00ms
[tag[$:/tags/Journal]count[]] 	1.60ms

Naϊvely, I would expect both to require approximately the same amount of run time and not a factor of ≈ 50 more, but maybe the tag[] optimizations in the core cannot be applied to !tag[]? It would be a pity, as I use the latter much more often than the former…

Have a nice day
Yaisog

PS: Both filters return roughly the same number of tiddlers.
PPS: I might need some filter rewriting…

#2

The first filter output every tiddlers except tagged tiddlers, then count them while the second only count tagged tiddlers so maybe it’s a normal behavior ?

EDIT: Should have read your PS, sorry! New theory: tags are indexed/stored somehow, but the inverse require to do a filter operation on the whole wiki so you dont get the optimization ? Would be interesting to try with [tagging[]]

EDIT2: There is a difference !

  • [all[tiddlers+shadows]tag[$:/tags/Stylesheet]] => 20.00ms
  • [[$:/tags/Stylesheet]tagging[]] => 21.00ms
  • [all[tiddlers+shadows]!tag[$:/tags/Stylesheet]] => 458.00ms
  • [all[tiddlers+shadows]]-[[$:/tags/Stylesheet]tagging[]] => 510.00ms
  • [all[tiddlers+shadows]] :filter[!tag[$:/tags/Stylesheet]] =>504.00ms

Seems like the tagging[] operator is a tiny bit faster, but I tried that on a fresh tiddlywiki so to be confirmed

EDIT3: Nevermind, I redid my tests by refreshing the whole wiki before each test and the results are now the reverse, tag seems faster on first load, but maybe tagging[] has a slight advantage on subsequent loads … or it’s something else

#3

Alright, since I don’t wanna just complain about things, here are some more measurements and a possible improvement by using a different filter with the same result…

I checked these filters:

\define tagger() [tag[$:/tags/Journal]]

{{{ [tag[$:/tags/Journal]count[]] }}} <!-- tagged -->
{{{ [!tag[$:/tags/Journal]count[]] }}} <!-- untagged -->
{{{ [!subfilter<tagger>count[]] }}} <!-- untagged -->
{{{ [!filter<tagger>count[]] }}} <!-- untagged -->
{{{ [all[]] -[tag[$:/tags/Journal]] +[count[]] }}} <!-- untagged -->
{{{ [all[]count[]] }}} <!-- all -->

Here are the measurements (these were taken on a faster machine than the OP, but on the same wiki):

[!filter<tagger>count[]] 	72.70ms
[!subfilter<tagger>count[]] 	43.40ms
[!tag[$:/tags/Journal]count[]] 	41.70ms
[all[]] -[tag[$:/tags/Journal]] +[count[]] 	7.60ms
[all[]count[]] 	0.80ms	10	8.00ms	8.00ms	2.00ms	1.00ms
[tag[$:/tags/Journal]count[]] 	0.60ms	10	6.00ms	6.00ms	3.00ms	0.00ms

So, !filter and !subfilter are surprisingly slow, even though they use the quick tag[] syntax, even slower than !tag[]. An improvement seems to be the [all[]] -[tag[foo]] syntax, even though it still takes more than 10× more time than tag[].
Interestingly, [all[]] is slower than [tag[]], despite basically doing nothing.

PS: A disadvantage of -[tag[foo]] is that it always receives all non-shadow tiddlers as input, while +[!tag[foo]]’ would only operate on the previous input. Not sure that that makes a difference time- or result-wise, though…

3 Likes
#4

I would also not expect their to be such a big difference. Both tag[] and !tag[]` use the same indexing infrastructure. I assume that the timings are averaged over multiple runs?

#5

Correct, average timing over 10 runs (everything is inside a ListWidget with [range[10]] filter).

#6

Hi @Yaisog did you get any further investigating this by any chance?

#7

I’m sorry, no. I changed some time-critical !tag[] in my filters to the -[!tag[]] alternative that speeds it up somewhat, but to really figure out what’s going on, I’m lacking the “core knowledge”…