Data Plugins as alternative to Data Tiddlers

I’ve used and built plugins as well as data tiddlers for many years, but only recently started to think about plugins as an alternative mechanism for data tiddlers - for at least my particular use case.

I have multiple, similar use-cases, but here’s one:

In my project management wiki, I store information on my contacts. I have maybe a 300 contacts that are stored in there and reference regularly as part of my job. I however work at a company with 50,000 employees, and it’s useful to periodically see if any details about my contacts have changed. So, I’ve long needed a way to compare the details about my 300 contacts against the whole company directory (AD).

Doing a regular import of 50,000 new tiddlers on a temporary basis is quite slow as you can imagine - especially on a file-per-tiddler system like I use. It also slows down the whole wiki. So, I have typically used data tiddlers and helpers like @pmario 's key-values plugins to operate around this. To get my data into these tiddlers is copy/pasting HUGE blocks of text over. It’s a bit of a pain, but better than the regular import of 50,000 new tiddlers.

Instead, I recently tried packaging the 50,000 tiddlers as a plugin (via the import / rename process that @EricShulman has described elsewhere) and noticed that performance was surprisingly good. Import speed due to it being a single file was great, and much less / no noticeable wiki performance degradation from carrying the extra 50,000 shadow tiddlers - I’m assuming because most parts of TiddlyWiki ignore these by default - exactly what I want. But now I can “query” and operate with these with just a little all[shadows] prefix and I’m good to go - full filter functionality!

The last piece was figuring out how to build the plugin without first importing them - which I’m now doing using the script I outlined in the post I just made about TiddlyWikiR.

Anyways, just thought this might be interesting to someone else in a similar situation who wouldn’t have considered using plugins as an alternative to large data tiddlers.

4 Likes