I have been working on a prototype wiki application that uses MWS (the earlier beta built by Jeremy) to process “place” data (geolocation + address + other related info).
I have a bag/wiki that contains my “application” code, and another bag/wiki that just contains “data” for that application. The data wiki currently has about 70,000 tiddlers! As a result, that data wiki loads and runs VERY slowly. If I were to directly include the data bag in my application wiki, it would also load very slowly and run even slower (due to the additional application-level logic operating on those 70k tiddlers).
To work around this, I have built a utility tiddler that runs in the data wiki to let me apply a filter to the 70k tiddlers and then construct compact indexed JSON tiddlers, each containing a selected subset of the 70k tiddlers, like this:
tiddlerA: { "field1":"value1", "field2":"value2", etc. }
tiddlerB: { "field1":"value1", "field2":"value2", etc. }
... etc
Then, in the application wiki, I have an interface that uses tm-http-request to fetch just the desired JSON index tiddler using a URL that points to https://domain-name/bags/datawiki/tiddlers/sometiddler. This allows me to load the compact indexed data “on-the-fly” only when actually needed by the application logic.
The received JSON index tiddler can then be further processed by the application code using a combination of indexes[], getindex[...], jsonextract[...], and jsonget[...] to either A) “unpack” the data to construct separate tiddlers or B) access the desired field values directly from the JSON index tiddler.
Using this approach, I can avoid the slow startup time (since it is not directly loading the bag with 70k tiddlers), and the application logic is also MUCH more efficient (around 5 times faster!) since it is now processing a single JSON tiddler containing the desired data rather than a huge set of separate tiddlers.
While this method currently works, more and more place data is being added to the big data wiki on a weekly basis so it does require me to periodically load the big data wiki and use the utility to manually prepare each of the desired indexed JSON tiddlers. Eventually, there may be hundreds of thousands of individual place tiddlers and perhaps dozens of indexed JSON tiddlers. As a result, this manual process gets slower and slower and I expect it will eventually “collapse under its own weight”, or at least become a severe maintainance headache.
What might be useful would be if there were an MWS server-side handler that would allow me to use tm-http-request to submit a bag URL, some kind of tiddler filter syntax, and a list of desired tiddler fieldnames, and have it automatically construct and return a JSON index tiddler containing just the matching tiddlers and the data from the indicated fieldnames. Because it would be processed server-side, it could hopefully use much more efficient SQL-based filtering (rather than browser-based client-side TWCore wiki filter syntax).
Your thoughts?
-e
