Performance issue

#1

Has anyone on this list got a single TW with >= 10,000 tiddlers in it? If so, what is

  • the file size
  • load performance
  • filter performance for filtering all tiddlers.

Bobj

#2

My wiki stats

Wiki Information

TiddlyWiki Version: 5.3.8
Current palette: $:/palettes/Vanilla
Current theme: 
Current layout: 
Browser language setting: en-GB
Default type for missing tiddlers: 
Auto save setting: yes
Code wrapping setting: pre-wrap
Sticky titles setting: no
Sidebar layout setting: fluid-fixed
Auto focus field setting for new tiddlers: title
Current storyview setting: classic
Toolbar text setting: no
Toolbar icon setting: yes
Button class setting: tc-btn-invisible
Navigation address bar setting: no
Tiddler opening behaviour setting for navigations from outside the story river: top
Tiddler opening behaviour setting for navigations from within the story river: below
CamelCase linking setting: disable
Keyboard shortcuts that have been customised: 
Disabled plugins: $:/plugins/bj/Calendar
Plugins: 
  $:/core - 5.3.8
  $:/plugins/inmysocks/WordCount - 5.1.10-prerelease
  $:/temp/info-plugin - 
  $:/themes/tiddlywiki/snowwhite - 5.3.8
  $:/themes/tiddlywiki/vanilla - 5.3.8

Number of tiddlers 31384
Number of tags 600
Number of system tiddlers 72
Number of shadow tiddlers 2335
Number of overridden shadow tiddlers 13

File Size as of last night: 75,919KB

It contains no internal images except the $:/favicon.ico which is 17KB when the png file is downloaded.

Load performance is slugish. Edditing or saving tiddlers often cause the page to crash out with out of memory.

To get around this Iā€™ve got four main ā€˜child wikisā€™ which I update (the largest of this is 48,495KB, 16,696 tiddlers) and then export the days modified tiddlers into a json. I refresh the ā€˜parentā€™ wiki and then upload all the json files for the day and save. Unless I skip the page refresh step I usually donā€™t get the Out of Memory issue this way. I also need to refresh the page if I need to delete a tiddler (normally as itā€™s been renamed in the child wiki).

I have had filter performance issues in the past but only if a large amount of tiddlers are in the output. Iā€™ve rewritten a few in the past to get around this.

The 48,495KB, 16,696 tiddlers child Wiki is the only child wiki where I have a noticable sluggish load performance on but it is not as bad as the parent.
I should add both this and the parent are encrypted wikiā€™s.

#3

Thank you @Alan_101 , sobering statistics.

I am going to have to think very hard about my wiki and performance. I like the idea of child wikis and that would fit into my scheme perfectly.

Thanks again.

Bobj

#4

Bob,

I did and do not have time to respond but there is a lot that can be done to improve performance I can touch base tomorrow if you want.

#5

The six minimalist wikis linked to from TW5-Bibles are all translations of the Christian Bible with separate tiddlers for every book, chapter, and verse. Four are English and two are Spanish. They have identical structures:

Wiki Information

TiddlyWiki Version: 5.3.6
Current palette: $:/palettes/Vanilla
Current theme: 
Current layout: 
Browser language setting: en-US
Default type for missing tiddlers: 
Auto save setting: yes
Code wrapping setting: pre-wrap
Sticky titles setting: no
Sidebar layout setting: fixed-fluid
Auto focus field setting for new tiddlers: title
Current storyview setting: classic
Toolbar text setting: no
Toolbar icon setting: yes
Button class setting: tc-btn-invisible
Navigation address bar setting: no
Tiddler opening behaviour setting for navigations from outside the story river: top
Tiddler opening behaviour setting for navigations from within the story river: below
CamelCase linking setting: disable
Keyboard shortcuts that have been customised: 
Disabled plugins: 
Plugins: 
  $:/core - 5.3.6
  $:/temp/info-plugin - 
  $:/themes/tiddlywiki/snowwhite - 5.3.6
  $:/themes/tiddlywiki/vanilla - 5.3.6

Structure

Tiddlers:

  • 66 Book tiddlers
  • 1,189 Chapter tiddlers
  • 31,102 Verse tiddlers
  • 3 minor tiddlers (Table of Content, Book, and {BibleVersionName})
  • 5 overridden shadow tiddlers ($:/SiteTitle, $:/DefaultTiddlers, etc.)
  • 7 custom system tiddlers (CSS, procedures, templates)

Plus, for the two spanish version

  • $:/language
  • $:/language/es-ES
  • one more that I canā€™t find :slight_smile:

Total: 32,371 - 32,374 tiddlers each.

Size

The translations vary a small amount in the number of characters, and the Spanish translation tiddler adds a little heft to those two versions, but they are all between 11.1 MB and 11.5 MB

Load time

I generally have fast connection speeds. For me these wikis load from an empty cache in 2 - 3 seconds, perfectly tolerable.

Other performance

Navigation around the site has no noticeable slowdowns. Showing books and chapters involve filtering all tiddlers, so it seems that handling that many tiddlers is doable. But note that I needed help in another thread to get it to reasonable performance. It can definitely go wrong; but when it does, thereā€™s a real chance that it can be fixed.

Searches seem to go quickly. Finding the 71 matches for ā€œGalileeā€ or the 4,000+ matches for ā€œgodā€ seem instantaneous.

What can be slow is the Advanced Search > Filter tab. If I have a filter that begins by searching for all tiddlers, and Iā€™m typing additional restrictions, I think itā€™s trying to run that ā€œall tiddlersā€ search on every keystroke. That can cause a real slowdown. Iā€™m just thinking about it now, but itā€™s probably worth researching whether the delay between keystroke and search is configurable and whether itā€™s debounced. Iā€™ll look into that when I have some time. (Unless someone knows the answer!)

But thatā€™s the only slowness Iā€™ve noticed in these wikis.

#6

Whoa! Impressive!

Could you do the Tipiį¹­aka too?

TT

#7

Well, once I got my process down, new translations were only a small amount of work. Some day, Iā€™ll get back to that and add more translations and languages.

Not any time soon. But if itā€™s something youā€™re interested in doing, I can offer advice from my experience. I also donā€™t know the material that well, although I read a lot of the Sutras and a little of the commentary decades ago when I was a religion major.

The goal of those wikis were to offer something for other people to build upon; they are intentionally very minimal. Others pointed out that these techniques could be used for other texts that are often closely read. Religious and legal texts sound the most likely, but perhaps people want to do that too with Kant or Plato. I could have used it when I tried Wittgenstein! But this is mostly meant as scaffolding.

#8

Thanks for your insights everyone. I am aware that we needed to do volume planning. In our present case,

  1. every gun will be stored in 6 tiddlers, the gun landing tiddler, history, local history, gallery, technical and map tiddlers.
  2. 2,000 guns at present, thus 12,000 gun tiddlers.
  3. 8,000 images at present, each to be in its own tidder (although the image is hosted externally), thus another 8,000 tiddlers
  4. No one knows how many common data tiddlers, ie. details about gun types (approx 550), ships, forts, manufacturers, etc.

So, all up, close to 20,000 - 25,000 tiddlers all up.

Not sure how that would load onto a mobile phone when dad wants to get some info about the gun son Johnny is currently crawling all over.

Not sure how the geomap system will handle 2,000 geomarkers either.

Has anyone played around with cross wiki links, especially some form of transclusion? I could envisage a separate TW for the landing pages (or even a separate TW for landing page per state) and then having the rest in additional TW(s) with cross wiki links.

bobj

#9

One way to consider performance is at anytime what can you see on the page? Lists including those in the sidebar tend to get longer as does the number of titles a filter iterates even before subsequent filters reduce the list. These all impact how much work needs to be done to generate the results. Hiding the side bar is a quick way to see what its contribution is, as is minimising the number of open tiddlers.

Also Tiddlywiki.com documents the filters that have additional caching associated with them and using these will improve the performance of filters.

If medium length or longer lists of links are needed rather that use the default link or buttons on each a link catcher widget can be used to improve performance.

There are many more ways to improve performance from capturing output to display, rather than regenerating it every time. Especially for read only websites for visitors where since visitors donā€™t make substantial changes to detailed structure you can hide tiddlywikis dynamic and responsive to change features to when you are building content.

  • There are performance tools you can use if you are not sure what is slowing a wiki down and take steps to address the top items can quickly improve a wiki.
#10

Can refresh throttling be used to avoid this ? https://tiddlywiki.com/prerelease/#Hidden%20Setting%3A%20Typing%20Refresh%20Delay

Or can using eventcatcher widget within the advanced search code result in better performance

Maybe long term users like @saqimtiaz , @EricShulman , @CodaCoder might be knowing about this. I remember seeing posts where they were talking about the use of those mechanisms in the past.

#11

This may work but you could replace the current advanced search with another that accepts the user typing into a temp tiddler but only searches when you click a tick or go button that follows.

Depending on your data you could provide a search only against titles until a tick or go button and only then it also searches in text.

Similarly you could somehow mark a lot of tiddlers to not appear in the standard search if they a well referenced elsewhere and need not be found themself, because they will be linked to or referenced elsewhere. Thereby reducing the number of titles that can be searched/

#12

Probably yes, if the resulting list is very long. The performance killer is not the number of DOM nodes that are created. Itā€™s the number of JS event-handlers have to be attached to every DOM node. So it would be worth an experiment.

I did create a GH issue. So the idea is not forgotten.

3 Likes