How to optimize filter performance for a large number of tiddlers?

,
#1

I’m exploring the possibility of using TiddlyWiki as an academic database to manage journals, papers, books and scholars information, I’am trying to write a plugin in which there is a page to search all papers by using filter, I find that it became sluggish after importing around 1,000 tiddlers.

In my design, I wanted scholars, journals, and years to be mutually filterable. My wikitext code achieves this, but it appears to be very inefficient, here is a part of my code

<$select tiddler=<<state>> field='year' default='Year'>
<option disabled>Year</option>
<option>any</option>
<$list filter='[subfilter<author-filter-variable>subfilter<journal-filter-variable>get[year]unique[]!sort[]]' variable="year">
<option><<year>></option>
</$list>
</$select>

and there are similar codes filtering scholars and journals. Then the following wikitext show all the papers filterd.

<ol>
<$list filter="[subfilter<author-filter-variable>subfilter<journal-filter-variable>subfilter<year-filter-variable>subfilter<search-filter>!nsort[year]limit[50]]">
<li>{{||$:/plugins/shenzhy/academic-lib/templates/ref-styles/paper}}</li>
</$list>
</ol>

So it seems that it’s not a good design, or there is any method to optimize filter performance, for example, having a button that triggers the filter only when pressed, this might improve performance since it’s not real-time?

#2

In your first filter you have unique[] which is known to be performance intensive.

You use a lot of subfilters, without showing the variables / subfilters. So it is impossible for us to test something. It would be nice if you would provide an minimal setup, so we do not need to do reverse engineering, which uses a lot of time.

1000 tiddlers are not really a large number.

#3

Thank you for your patient reply.

The subfilters, for example, author-filter-variable is defined by the following wikitext

<$let state="$:/state/shenzhy/academic-lib/index/paper" all-filter="[prefix[$:/academic-lib/papers]has[topics]]" author={{{ [<state>get[author]!is[blank]else[any]] }}} author-filter="[regexp:authors<author>]" author-filter-variable={{{ [<author>!match[any]then<author-filter>else<all-filter>] }}}>
...
</$let>

In fact, I roughly know why it is so laggy. Because I filter years based on authors and journals, filter journals based on authors and years, and filter authors based on journals and years. Plus, I have a search box to match titles. All paper tiddlers are actually traversed at least four times. In addition, the transclusions of reference citation formats has caused this seemingly small number of 1,000 tiddlers to lead to lag. I think the problem stems from my code design. But I would like to ask if there is a mechanism to avoid real-time results when encountering poorly designed code like mine (which is poor in performance, but I do require such repeated filter for convenience in usage). Instead, could the lag and waiting be shifted to, for instance, when a button is pressed?

If it’s hard to achieve, I think I need to change my wikitext logic to avoid multiple transclusions which just for make it convenient to select authors, journals, and years.

#4

We will have to have a closer look. So it will need a bit of time.