The 5,000 Tiddler Club – People with the Need for Speed

[Yaisog]

Who else has a TiddlyWiki that has grown to a monstrous size?

When new versions of TW are released, I always look for the Performance Improvements section of the release notes. Looking at some of the linked-to discussions on GitHub, adding performance improvements can be a tedious process. I think contributors like @saqimtiaz and @Flibbles are not appreciated enough for their work in this area, and I’d like to show them that they’re not doing it only for themselves, but that there are many of us out there benefitting from it. So post your wiki size here if you’re in the club.

My main wiki has around 8,900 tiddlers and 10.5 MB size, no embedded images except for a few SVG icons.

There are a few places here on Talk where useful tips on performance improvements for user wikitext can also be found. I have implemented what’s sensible, and my wiki doesn’t feel sluggish (yet). mainRefresh is around 100–200 ms for most navigation (even with search results open) and up to 400 ms when opening more complex tiddlers (think tabbed toc inside a tabbed toc) or searching. So, not bad, but it comes at the price of constantly looking for possible optimizations.

I’ll try to use this thread to collect some of these optimizations together in one place. Feel free to add.

I’ll start with @saqimtiaz’ Thoughts on Performance which I regularly like to revisit for inspiration.

Have a nice day
Yaisog

[Yaisog]

For large lists that take a while to update, @saqimtiaz has posted here a widget that inhibits automatic refreshing of its content (open and close if a refresh is needed).

This might become superfluous in most cases with @Flibbles’ caching of filter results that is coming in TW 5.2.3 (unless the list content changes often) and that he had to fight hard for to get in.

[Yaisog]

Tip:

On the topic of large lists:

• Keep the list item template simple. Most of the refresh time is needed (in my understanding) for the DOM update when rendering all these elements. Each $button or other fob added with each item make the browser work so much harder.
• If you need buttons for each element, don’t use $button widget but use the $eventcatcher. See also here for a nice demo of its capabilities.
• For searches: Do you really need to show all search results? What’s the probability that someone will click on result no. 250 (and hence will have glanced at all results before)? But displaying this potentially long list takes a lot of DOM update time. Personally, I don’t show more than 30 results at a time (and a note that there are more as well as a link to open $:/AdvancedSearch with all the results). If it’s not in there, I’ll change the search string. See also this discussion.

[Yaisog]

Tip:

All open tiddlers are refreshed (though not usually completely) on every user action. If you can limit the number of tiddlers in the browser DOM, your wiki will feel much faster.

@saqimtiaz has posted an experimental single tiddler layout here (I have not tried it yet).

In this discussion I’ve shown a way to extend the zoomin story view to dynamically fold all tiddlers that are not currently shown, so their contents is not rendered into the DOM. That made a BIG difference for me when having 10+ tiddlers open in the story river.

[Yaisog]

Tip:

Here’s one that I knew to be good practice, but until today not exactly why:
Avoid string substitutions like $...$ or $(...)$ in macro definitions if possible (which may not always be the case).

Macros return unprocessed wikitext, i.e. constant strings. So they do not change even when their parameters do, and thus do not trigger refreshes. However, when using string substitutions, a change in the parameter changes the definition of the macro and results in corresponding refreshes. I hope I paraphrased that right.

See also here and here.

[Flibbles]

Oh, don’t worry. There are still huge gains to be had in reducing filter refreshing. All I’ve done is make it so the compiled filters objects get cached. They still have to be called.

There’s an enormous amount of memory misuse that comes from filters, especially $list widgets. Every time filters are refreshed, they will be recalled. Every single filter step will create a long string list of all its results, only to throw it out a millisecond later. Even $list widgets, which go to such lengths not to change parse-nodes that aren’t changed will still allocate memory for each one of its children every time.

[Mark_S]

I haven’t re-evaluated, but at the time, with 60,000 entrees, I couldn’t use tag filter operators. Apparently they are optimized for smaller groups of tags (whereas in mine, almost all the tiddlers had the same tag). In the end, I used search instead, making performance reasonable on a desktop.

[Ste_W]

60000!!!
Madness!

[Yaisog]

Tip:

The thing to avoid is action widgets in the body of a button or other invoking widget, which are rendered each time the surrounding widget is rendered and also refreshed as part of every refresh cycle.

Wisdom by Saq (also previously in his Thoughts on Performance)

[cdaven]

Based on some ideas in An over-abundance of triggered updates, I’m experimenting with a StaleWidget that prevents content from being refreshed all the time.

Widgets can contain other widgets and macros, and each widget has a refresh() method that is responsible for refreshing the widget and its children. This is called on most keypresses and clicked links in TiddlyWiki.

All StaleWidget does is return false from refresh(), so that the content is never refreshed! To refresh the contents, you would have to close and reopen – or possibly fold and unfold – the tiddler.

I have wrapped my backlinks and freelinks footer with this, and the performance penalty for having many open tiddlers has almost disappeared.

[saqimtiaz]

@cdaven

All StaleWidget does is return false from refresh(), so that the content is never refreshed! To refresh the contents, you would have to close and reopen – or possibly fold and unfold – the tiddler.

The widget referenced earlier in this thread caters for the same use case but allows the flexibility of providing a filter to determine when to refresh.

[TW_Tones]

@cdaven

prevents content from being refreshed all the time.

I developed another method that captures the html of a rendered tiddler and instead displays that, not the active tiddler. Thus a tiddler with a lot of code does not need re-rendering with every detailed change. A simple button allows you to return to the “live tiddler” and or re-snapshot it. I have not performance tested it, but it seems to work.

Happy to share if asked.

[pmario]

@Yaisog … I did move the thread back to the Discussions category, because the OP doesn’t contain any tips and ticks. … It’s a request to post and discuss them.

Once you have enough info, you can create a summary from this post and create a Tips & Tricks … wiki …

which may / should be a “closed” thread with a link back to this discussion. …

See: About the Tips & Tricks category

[jonnie45]

Hi, mine was recently around 23MB but after a good tidy and rationalisation I brought it down to just under 18MB - I have under 2000 tiddlers, my larger tiddlers are entire 4 to 8 page scanned chapters from books.

My tidy efforts were due to the perception that my increasingly sluggish Tiddlywiki was suffering a size issue, I thought my research was in danger of being size capped on performance issues.

Finally I started playing around with the performance instrumentation and achieved some large and easy wins performance wise. I had a custom tab on the right hand side of my page - it goes all the way back to when I first started with Tiddlywiki - filters used there were the performance culprit and fixing them as much as I am able at this time has delivered massive performance benefits.

My laptop is pretty fast but my android phone and tablet would not sell for much in a second hand shop - in addition I am using Tiddloid with RCX simple server ( so I can play locally stored videos ). I recently had a big slow down on android that I thought was due to me adding more and more embedded videos* - no foundation for my thinking just a vague feeling that somewhere between videos-RCX server and tiddloid I had crossed some kind of threshold - but No!!! when I made the changes mentioned above my android devices sped up dramatically as well. Just goes to show that hunches and guesses with performance are not always reliable.

( *video stored in a local directory on the same device which is the reason I use RCX server on Android )

I am now under the impression that size is not performance issue for me right now, my attention is more focused on filters. I changed the order of statements in my slowest filter with good results by following advice on the tiddler named “Performance” on https://tiddlywiki.com/

My next step is to look at replacing the wikify gadget.

Currently my worst filter performance wise is one that shows the most “important” 100 tiddlers ( 100 is an optional number controlled by a slider ) based on the number of backlinks so that filter has to count the number of backlinks for every tiddler. I have it in mind to find solutions but the big wins gained so far mean I feel I can comfortably treat performance as an ongoing issue rather than a panic.

[jonnie45]

@Yaisog

For searches: Do you really need to show all search results? What’s the probability that someone will click on result no. 250 (and hence will have glanced at all results before)? But displaying this potentially long list takes a lot of DOM update time. Personally, I don’t show more than 30 results at a time (and a note that there are more as well as a link to open $:/AdvancedSearch with all the results). If it’s not in there, I’ll change the search string

On my custom right hand side tab which I use 90% of the time I have a slider which limits the number of tiddlers shown for my various custom filters ( show all tiddlers, show tiddlers ranked by xyz, by tag and so on ).

@jonnie45

[ Then here under the line all my tags are listed as tag pills - not shown - around 200 tags in all - Blink pills used to rank tiddlers based on number of backlinks ]

Incidentally I need to revisit the post from which I “quoted” the above image because it includes code for the originally badly optimised filters which I have only recently improved - although more still improvements required.

It’s convenient, on my android devices I rarely push the slider past 30, on my laptop perhaps a 100.

My ideal would be that the slider in the picture above controlled all searches as well as my filters and tag pills eg anything that can ‘fill’ the story river with tiddlers - often the approach I use to ‘fill’ the story river is based simply on a decision which method will get me to that cluster of tiddlers - the method is usually immaterial as long as it delivers the ‘working set’ of tiddlers I need - usually one tiddler being edited and perhaps another ten that I want to cross reference, cut and paste from and so on.

So for me it all boils down to the same thing - a chosen method to get to the cluster of tiddlers I want to see on the story river and then a wish to throttle back the number of tiddlers actually displayed in most cases after ranking (1) - for me the same slider would work equally well for pretty much all methods of populating the story river.

(1) For instance buttons marked ★↑ - rank tiddlers by ascending ( or descending ) star rating - unfortunately of course it is necessary to order all tiddlers by the selected ranking method before then applying the limit of only 30 or 100.

[TW_Tones]

These optimisations follow some well understood approaches to improving performance. They key is not unnecessarily displaying content that is likely to be refreshed with changes, simple closing the side bar can help. However when you need easy access to content you need to see it, and how you design it has a big influence.

• So the trick when getting performance issues is to try and Identify what is contributing to it and target that.

All,
There a lot of ways to improve this so do ask here

[Yaisog]

In my personal experience, what slows down my wikis the most are

• Large lists of tiddlers, possibly each with a number of DOM elements (e.g. buttons).

These usually appear in my sidebar, mostly for search results. The large number of DOM elements that must be created just slows the browser down. The slowest tab of all is the Explorer with its huge list of tiddlers. Unfortunately, it’s a very handy tool.
I limit search results to 30 title matches and 30 full-text matches. If there are more (there’ll be a corresponding notice), the search query just was not specific enough. Anyways, I was never reading through more than the first 30 results anyways.

• Full-text search of all tiddlers.

Especially for large wikis that can slow things down. I guess that backlinks is essentially also a parsing of the text field of all tiddlers? If possible I put stuff in fields, as these are cached or at least can be searched faster as it’s less text to search. Instead of backlinks I use a related-field that lists tiddlers that are, well, related to the current one. With such a field, it’s easy to quickly produce lists of tiddlers which are in the related-list or in which the current tiddler is listed (or both, so one only needs an entry in one of the related tiddlers, not both). Yes, it’s extra work to maintain, but adds to the information content of a wiki.
Otherwise, I you can, reduce the number of tiddlers that must be searched by preliminary filtering (tags and fields are best for that since they’re cached).

• Large number of open tiddlers in the river.

All of these are checked if they need refreshing on every change. There are a few ways to improve things, which can be found in other discussions here on Talk TW.

Have a nice day
Yaisog

[pmario]

@Yaisog

In my personal experience, what slows down my wikis the most are

• Large lists of tiddlers, possibly each with a number of DOM elements (e.g. buttons).

These usually appear in my sidebar, mostly for search results. The large number of DOM elements that must be created just slows the browser down.

I think, it’s not the number of DOM elements, IMO it’s related to the click-event-handlers that have to be initialized and refreshed by the browser.

May be the new event-catcher widget can help here. … BUT we would need to make some experiments with long lists without buttons-widgets and without link-widgets.

[Yaisog]

@pmario

I think, it’s not the number of DOM elements, IMO it’s related to the click-event-handlers that have to be initialized and refreshed by the browser.

You’re likely right. Since I did have lists where each element comprised a number of buttons for various actions, I actually implemented the $eventcatcher method. And I’d like to think that it made a difference. Though I didn’t measure.
However, since I implemented a context menu for the wiki, the whole point became moot. There are no more buttons, in some cases not even $links. All buttons are in the context menu now. From a perspective of DOM elements and event-handlers, this is probably the minimum of what’s possible. And it has the benefit of a much cleaner and tidy look.

It would be nice if the Explorer tab in the sidebar could be accelerated by means of an $eventcatcher instead of all the links. Like I said, that thing can be sloooow when there are branches expanded…

Have a nice day
Yaisog

[linonetwo]

I think this technique can be added to the core, within a requestIdleCallback, so it works like react’s fiber arch.