What exactly are the benefits of the Indexed Filter Operators?

#1

I know on Performance, it talks about being strategic with the use of the indexed filter operators such as

  • [all[tiddlers]
  • [all[shadows]
  • [all[tiddlers+shadows]
  • [all[shadows+tiddlers]

I still don’t quite understand why using the indexed filter operators provides such a performance improvement. Maybe I’m missing something but is there an example that demonstrates this somehow?

#2

As I understand it such operators “search” the whole wiki and identify a potentialy large list of tiddler titles. These operators are commonly used so rather than search the whole wiki each time they use the index, or cached list. Users never see these indexes only a quicker responce.

#3

The Performance tiddler you mention is from Oct. 2019, so most of the info that is given there is already outdated. It’s true, that there had been some specialized filter improvements at that time. … In the meantime almost all filters, that are executed somewhere are cached.

So most filter results can be used from a “pre-calculated” cache. The cache is created, when the filter is used the first time. … But … most caches have to be cleared, when the tiddler store changes. So filters have to be calculated more than once.

It is still important to follow 1 simple rule.

  1. In a filter run, reduce the number of tiddlers, the rest of the filter has to work with, as soon as possible.

That means eg:

  • use tags … so [tag[something]..xyz..] will only pass tiddlers tagged “something” to the xyz part of the rest of the filter.
  • [all[current]] … is a bit misleading. In reality the internal code only uses the <<currentTiddler>> variable, instead of touching the “store” at all
    -[my-field[something]] … should be as fast as tags, to start with

and so on.

So instead of search operators or regexp search it has some advantage, if tiddlers get some “intrinsic” structure like tags or fields …

hope that helps
mario

#4

When did we start caching filter results? As far as I know we only have an outstanding PR for caching the compiled filter functions but even with that each filter has to be executed each time it is run.

What part of that tiddler do you find to be outdated?

#5

If you look at the code for the indexers, e.g. $:/core/modules/indexers/tag-indexer.js, you’ll see that the index cache is only built for the set of tiddlers identified in the OP (the [all[…]] filters correspond to the various subindexers):

TagIndexer.prototype.init = function() {
	this.subIndexers = [
		new TagSubIndexer(this,"each"),
		new TagSubIndexer(this,"eachShadow"),
		new TagSubIndexer(this,"eachTiddlerPlusShadows"),
		new TagSubIndexer(this,"eachShadowPlusTiddlers")
	];
	$tw.utils.each(this.subIndexers,function(subIndexer) {
		subIndexer.addIndexMethod();
	});
};

The function getTiddlersWithTag used e.g. in the tag filter and defined in $:/core/modules/wiki.js makes use of these indexers:

exports.getTiddlersWithTag = function(tag) {
	// Try to use the indexer
	var self = this,
		tagIndexer = this.getIndexer("TagIndexer"),
		results = tagIndexer && tagIndexer.subIndexers[3].lookup(tag);
	if(!results) {
		// If not available, perform a manual scan
		results = this.getGlobalCache("taglist-" + tag,function() {
			var tagmap = self.getTagMap();
			return self.sortByList(tagmap[tag],tag);
		});
	}
	return results;
};

So, when you use the constructs described in the Performance tiddler, your filter will use the indexers and be much faster, especially for large wikis.
Have a nice day
Yaisog

2 Likes