Many of my TiddlyWiki projects work with very large numbers of tiddlers that must be filtered quickly through custom search interfaces. Some only have a few thousand items, and others have tens of thousands of items. For collections these large, refresh performance becomes a major concern when the lists of these items (i.e. search results) must update.
I made a post a while ago about a specific performance trick I learned, which was to avoid using the filter Operator when it isn’t necessary. I’ve since discovered more tricks for achieving high performance that I haven’t seen documented or discussed elsewhere, and they’re surprisingly simple.
The biggest one is that search:tags[] is significantly faster than tag[]. I do not have the technical expertise to know why it’s faster, but it’s easily the most useful one for how simple it is. Even when using limit[] to cut down the number of displayed results, (such as for pagination), the difference can be astronomical when working with large lists, especially those that must be filtered down by multiple tags.
For an example of how ridiculous this optimization can be: in my largest searchable collection project BlinkieWiki, it accumulated some major issues with lag when searching by blinkies and by specific colors, since there were a large number of tiddlers in these categories and the filter expression to process search results previously used tag[]. But simply by changing these to search:tags[], the laggiest possible search query (blue blinkies, took 13-15 seconds to process navigating to the next page!) was cut down to around 550 milliseconds. Whew!
A related trick is to use as few filter steps of search:tags[] as possible. The fewer the steps, the quicker the refresh speed (generally speaking). You can combine similar statements into one filter step if they are of the same kind:
- Match all tags:
search:tags:words[a b c]>search:tags[a]search:tags[b]search:tags[c] - Match no tags:
!search:tags:some[a b c]>!search:tags[a]!search:tags[b]!search:tags[c] - Match any tags:
search:tags:some[a b c]>[search:tags[a]] :or[search:tags[b]] :or[search:tags[c]]>
The difference is more noticeable when the number of tags that must be filtered is large in addition to the input list being large.
This effect also applies to search:tags[]; the larger the collection, the bigger the difference is between the optimal and suboptimal arrangements. It makes these tricks invaluable for very large wikis to function at reasonable speeds and to stay functional as they scale upward by orders of magnitude.
Lastly, although not that special of a tip, be wary of using redundant or unnecessary filter steps with searching by tags. I discovered a stray unnecessary tag[img] (which wholly overlaps with all tiddlers tagged as “webgraphic”) in a dropdown present in the search interface tiddler. Removing tag[img] in that filter expression eliminated half of the lag caused by that interface.
I’m always in pursuit of pushing the performance speed of TiddlyWiki to its limit, partly out of necessity and partly as the next horizon of my skills to improve on. I’m curious if anyone else has any performance tips in regards to processing large lists. I’ve slowly been developing my first-ever plugin for standardizing the creation of searchable collections, and it takes advantage of as many tricks as possible to keep the refresh speeds snappy.
