The 5,000 Tiddler Club – People with the Need for Speed

#7

Oh, don’t worry. There are still huge gains to be had in reducing filter refreshing. All I’ve done is make it so the compiled filters objects get cached. They still have to be called.

There’s an enormous amount of memory misuse that comes from filters, especially $list widgets. Every time filters are refreshed, they will be recalled. Every single filter step will create a long string list of all its results, only to throw it out a millisecond later. Even $list widgets, which go to such lengths not to change parse-nodes that aren’t changed will still allocate memory for each one of its children every time.

1 Like
#8

I haven’t re-evaluated, but at the time, with 60,000 entrees, I couldn’t use tag filter operators. Apparently they are optimized for smaller groups of tags (whereas in mine, almost all the tiddlers had the same tag). In the end, I used search instead, making performance reasonable on a desktop.

2 Likes
#9

60000!!!
Madness! :grinning:

4 Likes
#10

Tip:

The thing to avoid is action widgets in the body of a button or other invoking widget, which are rendered each time the surrounding widget is rendered and also refreshed as part of every refresh cycle.

Wisdom by Saq (also previously in his Thoughts on Performance)

#11

Based on some ideas in An over-abundance of triggered updates, I’m experimenting with a StaleWidget that prevents content from being refreshed all the time.

Widgets can contain other widgets and macros, and each widget has a refresh() method that is responsible for refreshing the widget and its children. This is called on most keypresses and clicked links in TiddlyWiki.

All StaleWidget does is return false from refresh(), so that the content is never refreshed! To refresh the contents, you would have to close and reopen – or possibly fold and unfold – the tiddler.

I have wrapped my backlinks and freelinks footer with this, and the performance penalty for having many open tiddlers has almost disappeared.

#12

The widget referenced earlier in this thread caters for the same use case but allows the flexibility of providing a filter to determine when to refresh.

1 Like
#13

I developed another method that captures the html of a rendered tiddler and instead displays that, not the active tiddler. Thus a tiddler with a lot of code does not need re-rendering with every detailed change. A simple button allows you to return to the “live tiddler” and or re-snapshot it. I have not performance tested it, but it seems to work.

Happy to share if asked.

#14

@Yaisog … I did move the thread back to the Discussions category, because the OP doesn’t contain any tips and ticks. … It’s a request to post and discuss them.

Once you have enough info, you can create a summary from this post and create a Tips & Tricks … wiki …

which may / should be a “closed” thread with a link back to this discussion. …

See: About the Tips & Tricks category

#15

Hi, mine was recently around 23MB but after a good tidy and rationalisation I brought it down to just under 18MB - I have under 2000 tiddlers, my larger tiddlers are entire 4 to 8 page scanned chapters from books.

My tidy efforts were due to the perception that my increasingly sluggish Tiddlywiki was suffering a size issue, I thought my research was in danger of being size capped on performance issues.

Finally I started playing around with the performance instrumentation and achieved some large and easy wins performance wise. I had a custom tab on the right hand side of my page - it goes all the way back to when I first started with Tiddlywiki - filters used there were the performance culprit and fixing them as much as I am able at this time has delivered massive performance benefits.

My laptop is pretty fast but my android phone and tablet would not sell for much in a second hand shop - in addition I am using Tiddloid with RCX simple server ( so I can play locally stored videos ). I recently had a big slow down on android that I thought was due to me adding more and more embedded videos* - no foundation for my thinking just a vague feeling that somewhere between videos-RCX server and tiddloid I had crossed some kind of threshold - but No!!! when I made the changes mentioned above my android devices sped up dramatically as well. Just goes to show that hunches and guesses with performance are not always reliable.

( *video stored in a local directory on the same device which is the reason I use RCX server on Android )

I am now under the impression that size is not performance issue for me right now, my attention is more focused on filters. I changed the order of statements in my slowest filter with good results by following advice on the tiddler named “Performance” on https://tiddlywiki.com/

My next step is to look at replacing the wikify gadget.

Currently my worst filter performance wise is one that shows the most “important” 100 tiddlers ( 100 is an optional number controlled by a slider ) based on the number of backlinks so that filter has to count the number of backlinks for every tiddler. I have it in mind to find solutions but the big wins gained so far mean I feel I can comfortably treat performance as an ongoing issue rather than a panic.

#16

On my custom right hand side tab which I use 90% of the time I have a slider which limits the number of tiddlers shown for my various custom filters ( show all tiddlers, show tiddlers ranked by xyz, by tag and so on ).

Incidentally I need to revisit the post from which I “quoted” the above image because it includes code for the originally badly optimised filters which I have only recently improved - although more still improvements required.

It’s convenient, on my android devices I rarely push the slider past 30, on my laptop perhaps a 100.

My ideal would be that the slider in the picture above controlled all searches as well as my filters and tag pills eg anything that can ‘fill’ the story river with tiddlers - often the approach I use to ‘fill’ the story river is based simply on a decision which method will get me to that cluster of tiddlers - the method is usually immaterial as long as it delivers the ‘working set’ of tiddlers I need - usually one tiddler being edited and perhaps another ten that I want to cross reference, cut and paste from and so on.

So for me it all boils down to the same thing - a chosen method to get to the cluster of tiddlers I want to see on the story river and then a wish to throttle back the number of tiddlers actually displayed in most cases after ranking (1) - for me the same slider would work equally well for pretty much all methods of populating the story river.

(1) For instance buttons marked ★↑ - rank tiddlers by ascending ( or descending ) star rating - unfortunately of course it is necessary to order all tiddlers by the selected ranking method before then applying the limit of only 30 or 100.

#17

These optimisations follow some well understood approaches to improving performance. They key is not unnecessarily displaying content that is likely to be refreshed with changes, simple closing the side bar can help. However when you need easy access to content you need to see it, and how you design it has a big influence.

  • So the trick when getting performance issues is to try and Identify what is contributing to it and target that.

All,
There a lot of ways to improve this so do ask here

#18

In my personal experience, what slows down my wikis the most are

  • Large lists of tiddlers, possibly each with a number of DOM elements (e.g. buttons).

These usually appear in my sidebar, mostly for search results. The large number of DOM elements that must be created just slows the browser down. The slowest tab of all is the Explorer with its huge list of tiddlers. Unfortunately, it’s a very handy tool.
I limit search results to 30 title matches and 30 full-text matches. If there are more (there’ll be a corresponding notice), the search query just was not specific enough. Anyways, I was never reading through more than the first 30 results anyways.

  • Full-text search of all tiddlers.

Especially for large wikis that can slow things down. I guess that backlinks is essentially also a parsing of the text field of all tiddlers? If possible I put stuff in fields, as these are cached or at least can be searched faster as it’s less text to search. Instead of backlinks I use a related-field that lists tiddlers that are, well, related to the current one. With such a field, it’s easy to quickly produce lists of tiddlers which are in the related-list or in which the current tiddler is listed (or both, so one only needs an entry in one of the related tiddlers, not both). Yes, it’s extra work to maintain, but adds to the information content of a wiki.
Otherwise, I you can, reduce the number of tiddlers that must be searched by preliminary filtering (tags and fields are best for that since they’re cached).

  • Large number of open tiddlers in the river.

All of these are checked if they need refreshing on every change. There are a few ways to improve things, which can be found in other discussions here on Talk TW.

Have a nice day
Yaisog

#19

I think, it’s not the number of DOM elements, IMO it’s related to the click-event-handlers that have to be initialized and refreshed by the browser.

May be the new event-catcher widget can help here. … BUT we would need to make some experiments with long lists without buttons-widgets and without link-widgets.

1 Like
#20

You’re likely right. Since I did have lists where each element comprised a number of buttons for various actions, I actually implemented the $eventcatcher method. And I’d like to think that it made a difference. Though I didn’t measure.
However, since I implemented a context menu for the wiki, the whole point became moot. There are no more buttons, in some cases not even $links. All buttons are in the context menu now. From a perspective of DOM elements and event-handlers, this is probably the minimum of what’s possible. And it has the benefit of a much cleaner and tidy look.

It would be nice if the Explorer tab in the sidebar could be accelerated by means of an $eventcatcher instead of all the links. Like I said, that thing can be sloooow when there are branches expanded…

Have a nice day
Yaisog

#21

I think this technique can be added to the core, within a requestIdleCallback, so it works like react’s fiber arch.

#22

Is there any reference?

  1. Why every DOM get updated when a tiddler (even a draft) changes? I thought there is a vdom and compare algorithm like React.js
  2. Why event-catcher can prevent DOM change?

For 2. , I see

Use of the event catcher widget is beneficial when using large numbers of other trigger widgets such as the ButtonWidget is causing performance problems. The workflow it enables is akin to what is referred to as “event delegation” in JavaScript parlance.

in https://tiddlywiki.com/#EventCatcherWidget. But didn’t explain why. I guess because ButtonWidget is a widget that will refresh? But the DOM inside EventCatcherWidget will refresh too, so this doesn’t make sense.

#23

It’s not rendering the HTML and updating the DOM that is slow, it’s the number of event handlers that have to be initialized and maintained by the browser. eg: Every <$link widget does attach a javascript event-handler to the link element. So if you have 1000 links you’ll have 1000 event-handlers.

If you have a event-catcher as a wrapper that catches clicks on DOM elements that internally can be selected using a class-selector, there will be only 1 event-handler for 1000 DOM elements. That’s much faster.

Hope that explains it a bit.

2 Likes
#24

That’s not quite right.

Every change to the tiddler store (even a draft) triggers the refresh cycle. It is up to the refresh processing of each individual widget whether it needs to recreate or update itself in the light of the changes to the store.

No, TiddlyWiki doesn’t use a virtual DOM.

I’m not sure what you mean. As you note, the eventcatcher widget enables event delegation patterns to be used.

2 Likes
#25

My main project (flux.wvwlogs.com) is currently 9,592 non system tiddlers [!is[system]sort[title]] of which 8,939 are echarts data tiddlers, type:application/javascript [tag[ChartData]].

I am running on nodejs with lazyload-all.

Currently about 75mb in uncompressed initial load size. Mainly because type:application/javascript are not lazyloaded and I haven’t found the magic to get echarts to process text tiddlers that are lazyloaded. Initial load would be around 7mb if I find the work around.

Beyond the lazyload issue, I don’t think there is much complexity involved that is slowing it down.

Update: Solved my lazy load issue and modified my ChartData tiddlers to type: vnd.tiddlywiki.
I am now serving 1.5 years worth of data and charts to 250 members with a ~7mb uncompressed initial load. :slight_smile:

#26
  • The html snapshot method I used was effective, used only TiddlyWiki script and was very performant.
  • Unfortunatly I do not know what you are talking about here?
  • Perhaps we could
    • Provide some examples and guidence in the documentation
    • In 5.3.x Write a custom procedure or widget that implements this.