Is there a practical limit to the number of tiddlers that can be imported via the JSON import mechanism?
By practical I mean an import that will not take too long to respond.
I am trying an import of 1761 tiddlers with little luck. I have tried import with 8000 image tiddlers which went great but this one today just won’t come back to me. The JSON is valid as per the online JSON validator.
My wiki won’t respond to clicks at all, it just sits there.
bobj
My main wiki is about 25000 tiddlers. Each time I want to try another saver etc I export them into chunks of 2000 for reimport, it’s slow but it doesn’t crash for me at 2000 so if there is a limit, it’s over that.
High on my TW wish list has been an import utility that’s more robust with a progress meter and not needing to display all the tiddlers with the checkboxes.
One thing of note is that when it’s okay for my tiddlers to be in plugin form (static lookup database like my company address book) I regularly import one json of 50000 and it’s quick. The import tiddler only shows 1 tiddler there, so it’s definitely the bottleneck.
@stobot, Is there a MB limit maybe, although I have tried to halve the file so its only 1.8Mb and it still fails.
I attach the half file for interest. Some tiddlers, ie. the gun tiddlers, are large with descriptive text whilst the others are relatively small.
SA_Converted_HTML.json (1.7 MB)
bobj
an update, I re-ran the hafl file and it eventually came back with the list of tiddlers ticked.
I now click the Import button and go away and have a half of stout 
I like your idea of the progress bar though, there is nothing like some feedback.
Environment: Mac OSX Tahoe 26, 24G RAM, Firefox V147.
bobj
Keep in mind you are actually doing two things here, including the JSON file in your wiki (using import or drop), then the second step in converting it to separate tiddlers, you are using the import mechanism to do both.
Rather than hit import, you could rename the import tiddler, and access the file as is, by changing plugin-type to plugin the tiddlers within become shadow tiddlers in a plugin.
Thus your filters may need to start [all[shadows+tiddlers].. there is no need for the Import button this way. If you did need to make them into tiddlers you could make a tool to edit all the shadows into tiddlers, but now you can time them, batch them, and provide a prompt or exit from that process.
Sorry @TW_Tones I have no idea what you are talking about. Could you be a bit more clear in your explanation? What file do I rename? How does a plugin help me and if so what plugin?
I have been thinking about alternate methods and can not see why I can not edit the htm file using a decent text editor and just paste the tiddler json’s into the file in the appropriate place . As this is just a conversion exercise, maybe I should just do that with all 2000+ guns and associated tiddlers.
Bobj
When you first drop the file on your wiki it gets named $:/Import its now a tiddler containing the files content. Rename that tiddler and follow my instructions.
I am suggesting the alternate method that gets tiddlywiki to help you.
I am just about to eat a cooked dinner so this answer is short. I think it may become clear as you step through what I said.
Ah, I see. Thank you
Bobj
I’m sure there is, but for one data point, the largest wiki I’ve worked on (by number of tiddlers) is when I was working on my King James Bible wiki. The Book, Chapter, and Verse tiddlers are available in a single JSON file, with 32,357 fairly small tiddlers, for a total of 10.0 MB.
I’m running at the moment on a developer-spec Windows machine, with 64 GB RAM and 500GB of storage, 14 cores and dual GPUs – a powerful machine. On this machine, running Chrome 143, it takes 5 - 10 seconds for the import dialog to show up when I drop that JSON onto an emtpy edition. It takes an additional 5 - 10 seconds to return after I choose Import.
From memory, on a more common spec home machine, running Brave on Windows and Brave or FF on Ubuntu, both steps take more like 20 - 30 seconds. But they do return; the machine certainly doesn’t freeze.
Pursuant to my original query, I am looking at the HTML file in the BBEdit text editor with a view to just pasting in the new tiddlers.
This raises a question, is there any document that describes the html file data structure? For example, there are loads of <li> lines, with what looks like one line per tiddler. What are these for and if I just paste in a swag of tiddlers in JSON format, do I also have to add new <li> lines?
I assume my new tiddlers need to be pasted into the section labelled “ordinary tiddlers”.
bobj
Note that the section of the HTML file that has the <li> lines starts with:
<!--~~ Static content for Google and browsers without JavaScript ~~-->
<noscript>
It provides
-e
thanks @EricShulman I did read that but the question was rhetorical, is a copy-paste solution feasible?
bobj
I have now setup a testing environment using the guns registered in South Australia.
However, I am concerned already about response times and this is 150 guns (9.2MB) out of a current total of 2,000
Could I ask someone/anyone/everyone to open the file and give me some idea as to opening times and machine/network specs.
Also, is there any info available on performance monitoring? Any tools I can use? Is the Advanced Performance plugin compatible with the latest version(s) of TW?
The text display at load time is me trying to install the splash screen example from tiddlywiki.com. No idea why this isn’t working.
bobj
I think this theme is best split into a new thread, so I am doing so, called Load Time Performance
Bobj