I have:
1000 shadows of which 10 are tagged “foo”
100 tiddlers of which 10 are tagged “foo”
(All tiddlers+shadows are identical except for the tags.)
Which of these filters is faster? And what’s the reasoning?
[all[tiddlers+shadows]tag[foo]is[tiddler]]
[all[tiddlers+shadows]is[tiddler]tag[foo]]
Any difference if the filter is split into two runs with a + injected?
Thanx!
The first filter will be faster. The tag operator is indexed and has higher performance. Having that first means there are fewer tiddlers that need the slow is[tiddler] check.
See https://tiddlywiki.com/#Performance
As a general rule, the quicker you narrow down the number of tiddlers in the filter run, the better the performance.
Filter operators are basically JS functions, so number of tiddler titles passing through them affects speed. And first few operators will reduce the number of titles.
IMO it should be [all[shadows+tiddlers]tag[foo]is[tiddler]] where shadows come first. Otherwise shadows will take precedence. If shadows are overwritten they become tiddlers. … So user modified tiddlers should always win
That’s a very interesting nuance. On tiddlywiki.com I note there are a few instances with tiddlers+shadows both among the docs and in shadow tiddlers. Maybe the all[] op should automatically order these operands behind the scene?
(…anyone remembers if there are other ops that take this double operand?)
One expects the is[tiddler] ignores all shadows isn’t it?
EDITED: [all[shadows+tiddlers]tag[foo]is[tiddler]] can return the overridden shadow tiddlers
The ordering is significant; [all[tiddlers+shadows]] lists the real tiddlers first and then the shadows, while [all[shadows+tiddlers]] lists the shadows and then the real tiddlers.
Maybe the OP was just an example regarding the filter speed and no real-life application, but
[all[shadows+tiddlers]is[tiddler]count[]]
and
[all[tiddlers]count[]]
give me the same number, even with an overridden shadow in the wiki, as I would expect.
So why not just use
[all[tiddlers]tag[foo]]
Am I missing something?
Have a nice day
Yaisog
Yeah, you’re missing the shadows. My guess is that you manually tagged them, thus overwriting them and creating system tids instead. In my hypothetical OP they were, so to say, part of the core or a plugin. I guess a way to test it would be to tag some tiddlers with a tag that a “good number” of actual shadow tids use.
EDIT: I mean, the resulting tiddlers are the same in the two filters in the OP but the question only concerned the speed as a result from the ordering of the filter operators.
Sorry if I seem blockheaded, but on a vanilla tiddlywiki.com both of these filters return zero results, as expected:
{{{ [all[shadows+tiddlers]tag[$:/tags/Filter]is[tiddler]] }}}
{{{ [all[tiddlers]tag[$:/tags/Filter]] }}}
If I modify e.g. $:/core/Filters/Missing to override its shadow, both of the above filters return the modified tiddler, still no difference.
I guess I’m still missing something.
What is not clear? Assuming you’re correc in that they first both return zero results - OK, so the tag[$:/tags/Filter] disqualifies everything. And when you overwrite a shadow tid, this creates a tiddler (note: a system tiddler is still a tiddler) so, yeah, both filters return that tiddler. There is still no shadow tiddler tagged such, because it would have to be created as a shadow tid with that tag for it to be tagged such. You can’t change a shadow tid without it creating a copy which is a system tid, i.e the shadow still doesn’t have the tag even if you try to apply it (…unless you use manipulate the actual json).
Just to be accurate, there are, in fact, 14 shadow tiddlers that are tagged with $:/tags/Filter:
$:/core/Filters/AllTags
$:/core/Filters/AllTiddlers
$:/core/Filters/Drafts
$:/core/Filters/Missing
$:/core/Filters/Orphans
$:/core/Filters/OverriddenShadowTiddlers
$:/core/Filters/RecentSystemTiddlers
$:/core/Filters/RecentTiddlers
$:/core/Filters/SessionTiddlers
$:/core/Filters/ShadowTiddlers
$:/core/Filters/StoryList
$:/core/Filters/SystemTags
$:/core/Filters/SystemTiddlers
$:/core/Filters/TypedTiddlers
The reason that @Yaisog’s filters return zero results is because the first filter uses is[tiddler] which excludes shadows, and the second filter use [all[tiddlers] which also excludes shadows.
{{{ [all[shadows+tiddlers]tag[$:/tags/Filter]is[tiddler]] }}}
The is[tiddler] asks if any of the tiddlers tagged is a system or normal tiddler. If there is no result the answer is no. That’s expected, because all of them are shadows. If you edit one of them in a “test wiki” you will get a result
If you ask are there any shadow or tiddlers tagged $:/tags/Filter like so
{{{ [all[shadows+tiddlers]tag[$:/tags/Filter]] }}}
There is answer with some results
Thanks for all your answers. What I am taking home from this is that
[all[shadows+tiddlers]is[tiddler]]
is in all cases equal to
[all[tiddlers]]
which I had assumed from the beginning, but was led to doubt by the discussion here.
This answers my question in #8 with “no”.
Sorry if my way of questioning may have been confusing, as it had no relation the the OP. I was just bugged that the example given in the OP was a filter that is unnecessarily complex and thus slow, which seemed to be the opposite of what was intended after all.
Have a nice day
Yaisog
A rule of thumb for filters is, to reduce the set of tiddlers that is worked with as fast as possible. … So at the beginning of the filter expression
So if you start with [tag[foo]] which is a shortcut for [all[tiddlers]tag[foo]] is generally a good thing. Filtering with tags is highly optimized.
Similar to tags there can be a check if a field has any value eg: [has[this-field]] also reduces the number of tiddlers to work with very fast.
If you need to check if a field has a value of “x” you can use [this-field[x]] … which is very close tagging with a single tag. It’s a shortcut for [field:this-field[x]]
… just some examples
I’m curious; what filter comparison would you (or anyone) suggest as a better example for asking how the order of the operations affect which is faster? (Or slower, for that matter.)
I guess [all[tiddlers]tag[foo]is[tiddler]] vs [all[tiddlers]is[tiddler]tag[foo]] would be possible but it feels like the “all tiddlers that are tiddlers?” kind of confuses the question.
I don’t know if the is[tiddler] is particularly important to you.
If not, you could compare
[all[tiddlers]tag[foo]sort[]] vs. [all[tiddlers]sort[]tag[foo]]
If so, you could compare the filter run
+[tag[foo]is[tiddler]] vs. +[is[tiddler]tag[foo]]
for arbitrary preceding filter runs which yield an input list of tiddlers.
Would at least have distracted me less.
PS: You got the answer that you sought with your initial post, so your examples couldn’t have been too bad… 
I put subfilter to the front to bring echarts loading speed from 3s to 0.17s refactor: speed up using subfilter to narrow down tiddler pool · tiddly-gittly/tw-echarts@7742b9a · GitHub
before
[all[tiddlers]sameday:created[${date}]][all[tiddlers]sameday:modified[${date}]] +${subfilter} +[sort[]]
after
/** Use subfilter to narrow down tiddler pool before the array.map on dates */
const tiddlerSourceIterator = $tw.wiki.makeTiddlerIterator(
$tw.wiki.filterTiddlers(subfilter),
);
then pass to
[sameday:created[${date}]] [sameday:modified[${date}]]
i’m surprised at the huge speed difference adding !is[system] seems to make in tests on filters (with over 3000 results) in my main wiki just now. is this because the system tiddlers, though few in number, are large/complex and therefore take longer to parse? if this difference is true, would it make sense to add it to most filters designed to produce many results?