TW 5.2.0-prerelease significantly slower than TW 5.1.23 (instrumentation data included)

hubertgk · September 17, 2021, 9:12am

Hello,

I’ve tested the 5.2.0-prerelease over the past few days both on mobile and on desktop.

I’ve noticed significantly degraded performance on mobile and did some performance instrumentation tests on my desktop. Results below.

I’ve primarily tested the basic TW operations such as navigating links (opening, closing tiddlers), selecting checkboxes, radio buttons, clicking buttons etc.

The overall performance is significantly slower in 5.2.0 compared to the same wiki running on 5.1.23. It appears that mainRenderer and mainRefresh see the biggest slowdowns.

Here’s some data from the two identical wikis running side by side, the figures on the left correspond to the 5.1.23 version and the ones on the right are from the 5.2.0-prerelease. In each “test run” I had all tiddlers closed beforehand and then performed identical navigation:

mainRenderer values

Opening tiddlers (variable levels of complexity)
151 ms vs 276ms
149 ms vs 230ms
242 ms vs 302ms
105 ms vs 196ms
398 ms vs 503ms

Opening Control Panel
90ms vs 190ms

Opening Advanced Search
83ms vs 195ms

Clicking on a checkbox that sets a value in a data tiddler
67ms vs 160ms

Pressing a button that sets a value in a data tiddler
175mm vs 278ms

Entering a password in an encrypted wiki, upon pressing enter:
mainRender (v5.1.23): 226.00m vs mainRender (v.5.2.0-prerelease): 350.00ms
mainRefresh (v5.1.23): 74.00ms vs mainRefresh (v.5.2.0-prerelease): 192.00ms

I’ve performed these tests on my desktop with an 8-Core Ryzen CPU clocked at 3.60 GHz and 32GB RAM. The performance on my midrange Motorola is at least 5 times slower. While desktop performance slowdowns on this system are negligible, mobile performance introduces major usability issues.

The values above are obviously system dependent and wiki dependent but the point I’m trying to make is that in relative terms, 5.2.0-prerelease is significantly slower than 5.1.23. I’ve made these tests on the same system, the same identical wiki, with the only difference being the TiddlyWiki version.

I’d like to hear your thoughts about this and also if there’s anything you could suggest. Could anyone perhaps perform similar tests on their end?

One point to note is that the wiki I used was password protected, which means that the new json parsing probably does not affect it yet (unless I misunderstood). However, the core render modules are still slower to render the wiki and it would be interesting to learn why it should be the case.

Thanks in advance!

Regards,
Hubert

pmario · September 17, 2021, 9:32am

Hi Hubert,
Thanks for the detailed information!

How many tiddlers do you have in your wiki.
Which plugins do you use
Any themes?
Single file wiki or node-wiki?

You wrote “past view days” … Can you be more specific?
It seems a new version was created: 12 h ago. … So if you downloaded it a week ago there shouldn’t be too much difference. …

If you did download it a month ago, your values will be hard to reproduce. …

pmario · September 17, 2021, 9:33am

@jeremyruston … We should probably create a “reference wiki” with a lot of content, that can be used by users and by us to test such issues.

If we have a “reference” wiki, the values should be easier to reproduce. By a “lot of content” I think about 10k to 20k tiddlers

jeremyruston · September 17, 2021, 9:34am

Thank you @hubertgk that’s very helpful. We don’t have performance regression testing in place, so it’s easy to miss when it happens. I’ll create a GitHub issue for this since I think some of the people who may be interested in helping might not yet be here.

jeremyruston · September 17, 2021, 9:36am

I’ve created the ticket over at GitHub here:

github.com/Jermolene/TiddlyWiki5

Performance regression reported from v5.1.23

opened 09:35AM - 17 Sep 21 UTC

Jermolene

From https://talk.tiddlywiki.org/t/tw-5-2-0-prerelease-significantly-slower-than…-tw-5-1-23-instrumentation-data-included/776: --- I’ve tested the 5.2.0-prerelease over the past few days both on mobile and on desktop. I’ve noticed significantly degraded performance on mobile and did some performance instrumentation tests on my desktop. Results below. I’ve primarily tested the basic TW operations such as navigating links (opening, closing tiddlers), selecting checkboxes, radio buttons, clicking buttons etc. The overall performance is significantly slower in 5.2.0 compared to the same wiki running on 5.1.23. It appears that mainRenderer and mainRefresh see the biggest slowdowns. Here’s some data from the two identical wikis running side by side, the figures on the left correspond to the 5.1.23 version and the ones on the right are from the 5.2.0-prerelease. In each “test run” I had all tiddlers closed beforehand and then performed identical navigation: mainRenderer values Opening tiddlers (variable levels of complexity) 151 ms vs 276ms 149 ms vs 230ms 242 ms vs 302ms 105 ms vs 196ms 398 ms vs 503ms Opening Control Panel 90ms vs 190ms Opening Advanced Search 83ms vs 195ms Clicking on a checkbox that sets a value in a data tiddler 67ms vs 160ms Pressing a button that sets a value in a data tiddler 175mm vs 278ms Entering a password in an encrypted wiki, upon pressing enter: mainRender (v5.1.23): 226.00m vs mainRender (v.5.2.0-prerelease): 350.00ms mainRefresh (v5.1.23): 74.00ms vs mainRefresh (v.5.2.0-prerelease): 192.00ms I’ve performed these tests on my desktop with an 8-Core Ryzen CPU clocked at 3.60 GHz and 32GB RAM. The performance on my midrange Motorola is at least 5 times slower. While desktop performance slowdowns on this system are negligible, mobile performance introduces major usability issues. The values above are obviously system dependent and wiki dependent but the point I’m trying to make is that in relative terms, 5.2.0-prerelease is significantly slower than 5.1.23. I’ve made these tests on the same system, the same identical wiki, with the only difference being the TiddlyWiki version. I’d like to hear your thoughts about this and also if there’s anything you could suggest. Could anyone perhaps perform similar tests on their end? One point to note is that the wiki I used was password protected, which means that the new json parsing probably does not affect it yet (unless I misunderstood). However, the core render modules are still slower to render the wiki and it would be interesting to learn why it should be the case.

@hubertgk if you have a GitHub account do join the discussion here.

hubertgk · September 17, 2021, 9:42am

Hi Mario,

Thanks for a quick reply.

Here’s what I forgot to include:

The wiki has 2121 ordinary tiddlers
No plugins (just the core)
No additional themes (running Vanilla)
Single file wiki, just over 5.3 megs.

I’ve first used the 5.2.0-prerelease yesterday 9PM (GMT), so technically just yesterday evening and today morning (so not actually “a few days” ). So it could in fact be the second-recent version that I used.

Many thanks,
Hubert

hubertgk · September 17, 2021, 9:44am

Hi Jeremy,

Thank you for looking at this. I will try to join and contribute in any way I can.

Best regards,
Hubert

jeremyruston · September 17, 2021, 3:51pm

Here’s an update I’ve made to the GitHub issue:

I’ve now setup a very basic performance testing framework. It consists of:

A test wiki (a snapshot of editions/tw5.com)
A script to build the wiki with a specific Git commit of TiddlyWiki
A script to open a wiki under Puppeteer and extract the performance timings

The repository is here:

At the moment, after running the tests several times I’m not seeing any performance regression between v5.1.23 and master:

./build-tiddlywiki-at-version.sh v5.1.23 ./tmp/v5.1.23 && ./build-tiddlywiki-at-version.sh master ./tmp/master
node ./index.js ./tmp/v5.1.23/index.html && node ./index.js ./tmp/master/index.html

Output:

file:///xxx/git/Jermolene/tiddlywiki-performance-test-rig/tmp/v5.1.23/index.html
2021 9 17 16:43:12.200: Execute $tw.perf.log(); to see filter execution timings
2021 9 17 16:43:24.666: mainRender: 12301.70ms
2021 9 17 16:43:25.340: styleRefresh: 2.70ms
2021 9 17 16:43:26.465: mainRefresh: 1125.20ms

file:///xxx/git/Jermolene/tiddlywiki-performance-test-rig/tmp/master/index.html
2021 9 17 16:43:33.039: Execute $tw.perf.log(); to see filter execution timings
2021 9 17 16:43:45.018: mainRender: 11793.60ms
2021 9 17 16:43:45.680: styleRefresh: 2.60ms
2021 9 17 16:43:46.811: mainRefresh: 1130.20ms

The inconsistency may be because the test rig isn’t measuring the correct thing. It would be helpful to dig into your wiki – would you be prepared to share it privately?

hubertgk · September 18, 2021, 3:31pm

Thank you Jeremy. I’ve responded via GitHub.

jeremyruston · September 23, 2021, 12:09pm

Just to complete this thread, two things have now happened. First, we fixed the regression by refining the fix for the original problem, returning the performance of v5.2.0 to the v5.1.23 baseline:

Then we went on to optimise the performance of sorting tiddlers:

github.com/Jermolene/TiddlyWiki5

Optimise wiki.sortTiddlers()

Jermolene:master ← Jermolene:sort-optimisations

opened 06:16PM - 19 Sep 21 UTC

Jermolene

+92 -33

As discussed in #6042, there are some simple opportunities to make tiddler sorti…ng more efficient. These changes improve the startup time of a test wiki as follows: * v5.1.23 mainRender: **5412.20ms** * [575c233](https://github.com/Jermolene/TiddlyWiki5/commit/575c2335975cc6cc960d2061ba8fdfa726ed8618) mainRender: **11221.60ms** * with this PR mainRender: **3862.00ms** ([575c233](https://github.com/Jermolene/TiddlyWiki5/commit/575c2335975cc6cc960d2061ba8fdfa726ed8618) is the last commit before reverting the problematic commit [dbd3f83](https://github.com/Jermolene/TiddlyWiki5/commit/dbd3f835bf8399ed1a3da7cc322ec9b6ab783d53))

In tests with a wiki that contains lots of sorting, we saw an improvement in startup time of 40%.