Performance when TW5 has lots of data tiddlers

,
#1

I understand there are already lots of discussion about performance of TiddlyWIKI, e.g.

My personal TW has around 10K tiddlers without any performance problem at this stage. My setup is TW 5.3.3 with node.js under Windows 11.

I am starting to use echarts to visualise my data in TW and concerning about the performance in the near future. The data are much bigger than pure text tiddlers and increasing very quick.

For example, a simple visualisation of historical selected agricultural commodities takes 500 KB in text/vnd.tiddlywiki format. Shortly, it will be a problem if I visualise more data.

What should I plan now? For example

  • Store data in echarts data in application/json.
  • Lazy-load all tiddlers or just data tiddlers (i.e. application/json).
  • A separate Tiddlywiki instance or server only for data.

Thanks for any advices.

#2

application/json will not lazy load, at least I couldn’t manage to make that work. I save all my echart data tiddlers as text/vnd.tiddlywiki to utilize lazy load.

This did require me to rollback to echart v0.0.9 to get it working properly. Still an outstanding issue: LazyLoad chartdata with <$echarts $text={{LazyLoadExampleOfTextData}}/> · Issue #84 · tiddly-gittly/tw-echarts · GitHub

2 Likes
#3

Thanks for your suggestions

#4

Are you using single file or the node server?

There is a lot of ways to improve performance for large wikis

#5

I use node.js with single TW instance in my working laptop. No problem of performance at this stage. I just worried about the performance in the near future if I imported lots of data (e.g. GB+).

#7

After some testing, I believe this method should work with me.

  • Add a tag data to all tiddlers for data visualisation
  • create a tiddler $:/core/save/lazy-data (clone from $:/core/save/lazy-images, modify to exclude [tag[data]]).
\define saveTiddlerFilter()
[is[tiddler]] -[prefix[$:/state/popup/]] -[[$:/HistoryList]] -[[$:/boot/boot.css]] -[type[application/javascript]library[yes]] -[[$:/boot/boot.js]] -[[$:/boot/bootprefix.js]] -[!is[system]tag[data]] +[sort[title]] 
\end
\define skinnySaveTiddlerFilter()
[!is[system]tag[data]]
\end
{{$:/core/templates/tiddlywiki5.html}}
  • Update my vbs file to start tiddlywiki (node.js in Windows 11).
CreateObject("Wscript.Shell").Run "cmd /c tiddlywiki tiddlywiki --listen root-tiddler=$:/core/save/lazy-data port=80", 0, false