Performance of Wiki and the number of tiddlers (assume same amount of information)

Hello, I have a (rather theoretical) question.

Suppose I have a wiki A with only one non-system tiddler T containing 100 lines.

For each line in T, I could excise it to its own tiddler. Thus one tiddler T = 100 tiddlers (information-wise).

I create a wiki B containing these 100 tiddles. Thus, information-wise, wiki A = wiki B. But:

  • Wiki A has one (big) tiddler
  • Wiki B has 100 (small) tiddlers

Now what if, instead of one tiddler T, I have 1000 tiddlers T. How would the wiki A (with fewer but large tiddlers) compare to the wiki B (with more but small tiddlers) in terms of the performance? For example,

  • Will searching tiddlers in B quicker or slower than in A?
  • Will navigating/reading tiddlers in B quicker or slower in A?
  • Will the answers be different if using a single file html instead of node version?

If you think the number 1,000 is still small that it doesn’t make a difference, then please change it something big like 1,000,000 (so a theoretical question).

I guess the real question is, with the same amount of information, would having a lot of tiddlers be a good/bad idea when it comes to performance in long term?

(Of course when it comes to flexibility, having many tiddlers should be more flexible as it allows more tiddler reuse).

The answer is: It depends :wink:

TLDR;
I think it’s important to find a balance between tiddler size and the number of tiddlers in a wiki.


I think, it doesn’t make sense to create 1 tiddler for every line of “prose text”. It will be a nightmare to edit it, since the default UI doesn’t have a convenient way to edit the “related” tiddlers.

On the other hand, it does make sense to have 1 tiddler per “idea” if you are in “brainstorming mode”. There is an edition / plugin, that is especially built for this purpose. Streams [1]

It’s also useful, if you want to work with an outliner. …

The advantage of such an edition is, that it is very flexible, since there are a lot of tiddlers, that only contain as much information as needed to make sense on their own. …

The disadvantage is, that you have more tiddlers.

More tiddlers doesn’t automatically mean the wiki is slower. Internally the lookup of tiddlers is highly optimized and fast even if you have 10th of thousands of them.

Drawing the UI will become slow, if you create long lists, that don’t fit on the screen. Since the DOM elements need to be created, even if they can’t be seen. … So for the user it seems the UI slows down.

So if you create visible lists with a decent size, the UI will stay responsive.

Internally the TW core uses several indexers, that can speed up tiddler lookup. Currently the core uses 3 indexers for:

  1. tags
  2. backlinks
  3. field-values

add 1)
Even if you have 10000 tiddlers and eg: 20 of them are tagged: ToDo, the core will only need to search all tiddlers once. The next time you ask for the same tag, the result will be immediate, since it comes from an internal cache.

add 2)
Tiddler backlinks [2] are also cached in a similar way as tags. Since this type of info is important, the core is optimized to have fast lookups

add 3)
Since field-values are often used to filter lists, they are also cached internally.

To conclude: Having many tiddlers doesn’t necessarily mean, that the wiki will slow down. It just means you have to find a “structure” that fits your needs.

[1] Streams — on TiddlyWiki 5.1.23
[2] https://tiddlywiki.com/#backlinks%20Operator

Mario,

May I ask you to fact check or point to some info on caching field values, since as I understand it it does not cache field names? I ask because this can change dramatically in 5.2.0

As others have mentioned, having a large number of tiddlers isn’t going to have much of an effect on performance. It is possible to run into performance issues with large TiddlyWikis, but such lag is typically more pronounced on “under powered machines”.

I would be curious what the latest guidance on managing large TiddlyWikis is. I seem to recall this topic from the Google Group discussing adjusting a value for the TypingTimeout, this single adjustment fixed a lot of the typing lag I used to experience (16 mb tiddlywiki file with 2,105,567 words in text tiddlers). I’m still a bit unclear in my mind how this works, but it does.

All,

On the performance with large numbers of tiddler, I have visited this issue quite a few times. In one case we had 70,000 tiddlers and this was made workable with a few tweaks. Throughout the google forums are a number of discussions on performance.

The key, as always is, how much work are you asking tiddlywiki or your computer to do for you?

Because tiddlywiki does a lot for you, to keep every visible tiddler, tab and list instantaneously up to date out of the box, tiddlywiki is doing a lot of work. When you start to push things there are simple ways to minimise your demands on tiddlywiki, like only a few tiddlers open or close the sidebar.

Where are you running your tiddlywiki from? Your SSD, local node or a distant server?

Then finally how many resources does your tiddlywiki have available to do its job?, this one contains a little counter intuitive knowledge to understand. Consider where tiddlywiki is;

  • TiddlyWiki Runs in a tab/window of your browser
  • Your browser runs in your operating system
  • Your Operating system runs on your device

Traditionally the Browser was your “universal window” to internet resources, otherwise we would run local applications on our local computer. In this case there was unlimited potential for the browser to consume too many resources and slow our local applications. Most if not all browsers as a result like to boast how little memory they use on your local machine and in fact they limit themselves so they are not a liability for the running of local apps.

But now these internet resources are now fully functional apps such as tiddlywiki, that is our applications also run in the “universal client”, the browser. I for one, use local apps less and less every day, instead crafting solutions and applications “on demand” in tiddlywiki, in fact I have a number of tiddlywikis that evolve as the need arises for a feature, from which I capture reusable code. When I want to “spin up an app” I mostly “spin up a tiddlywiki” and drop reusable code, including plugins on it.

Because I discovered the browsers were limiting their own memory consumption to around 2GB of RAM on my 16GB machine (an eighth), some time ago, so I do not remember how I found the details, I raised the memory that both firefox and Chrome can acquire from the local machine, I doubled it to 4Gb each, leaving 8Gb for everything else.

Hardware, browser and OS updates may have reset this but since I am not running into any performance issues I have not revisited it for a while. My point however, is there is plenty of headroom available for tiddlywiki performance improvements. Another trick is to use more than one computer, like running a node server elsewhere and “only browsing to it”, skinny tiddlers and soon the File Uploads Plugin. Then all this is without even splitting tiddlywikis into more than one along logical lines, such as building project wikis rather than multiple projects in one wiki, and there are heaps of inter wiki integrations available.

I hope this “prose” gives you a feel for the capabilities of tiddlywiki, I would also invite anyone using all the available performance tweaks mentioned herein, to find an application of tiddlywiki which has performance issues and share with the community so we can identify even more ways to improve performance. The only caveat is that you apply standard design techniques, like don’t set up a single file wiki with many gigabytes of data, in other applications you would move this into a database, tiddlywiki is not immune from the limitations of any software or app, but it is more flexible than most.

Tones

On the Original Question, my answer would be use the method that is the easiest for you to understand and design your wiki in a way you can try other method’s if needed.

Thank you all for the inputs. I don’t know how Tiddlywiki works behind the scene so I was a little worried with the performance. I feel a little assured now.

To give a situation of why you could easily have a large number of (small) tiddlers. I have a Question Bank wiki currently consisting of 500 math questions (one question per tiddler). I plan to add plenty more. I also installed the shuffle filter operator plugin. This allows me to randomly pull a selected number of questions from the bank to form a quiz/exam, like this:

[tag[qn]] +[shuffle[]first[5]]

Many of my questions are multiple-choice, typically looks like this: (forgive the silly question!)

1+1 = ?

  • A. 0
  • B. 1
  • C. 2
  • D. 4
  • E. 5

The choice names (A,B, etc) are hard coded into the tiddler which means I can’t random the choice order. I was planning to take a step further by excising each choice into its own tiddler and pull them together with transclusion and shuffle operator.

This would mean that such question (originally taking up one tiddler) would breakdown into 6 tiddlers (one for question statement and 5 for the choices). If I have 500 such questions then I would easily get 3000 tiddlers. Hence my question about the performance :slight_smile:

Btw, if there is a Tiddlywiki edition for making and generating exam that would be nice. Tiddlywiki is such a powerful and versatile software!

@TW_Tones

I am an end user of Tiddlywiki. Reusing the work of others (including more or less changed for my own use.) I am always using old reused laptops.

When we tested Twederation in 2016 many TW versions ago, My TW on tiddlyspot got really slow around 11- 12 mb. At that time I used a 32 bit netbook with 2 gb ram.

Later I went with old laptops of the type Lenovo thinkpad - the models around 10-12 ears old. Always running Linux Mint. The oldest models having a max of 4 gb ram the “newest” 8 gb ram.

My personal go to TW started 2014 and containing most of the stuff I have used and how during the years (my memory) is around 20 mb and runs totally okay on the old laptops in a browser (Firefox) with a lot of work going on in other tabs.

To diminish the growing amount of open browser tabs I have a bookmarking tiddlywiki pinned and always ready to use.

Of course I have used synchronization and cloud storage solutions - but for now I am mostly using tiddlywiki from a usb stick (the old computers has USB2 only). Easy to back that one up - and easy to move around from laptop to laptop.

Now why am I using those old laptops:

  • I like the idea of reusing stuff.
  • Love the keyboard
  • The surface is nice to caress in a thinking pause
  • Best fit for an old persons pension.
  • Surprising how much coffee and coca cola that can be accidentally spilled and still just drained by being quick turning the laptop a little and getting it out again…love that drainage.

Maybe some ideas for your “grandma” too.

2 Likes

I am confident you will be fine with that many.

However if you wanted to you could hide the answers in a data tiddler with the question part of the key. This would allow question and answer pairs to be codes so it is hard to find the answers by editing the tiddler or hacking the wiki.

To me part of tiddlywiki broadly supporting what I call “software democracy” is that it has very low hardware requirement’s. If for example you do not include imbedded media and mostly organised text, just one tiddlywiki can store many “encyclopedia Britannica’s”.

Hi, size does not affect performance 50M large / huge size of TiddlyWiki, performance remains good