Practical limit on JSON import

Is there a practical limit to the number of tiddlers that can be imported via the JSON import mechanism?

By practical I mean an import that will not take too long to respond.

I am trying an import of 1761 tiddlers with little luck. I have tried import with 8000 image tiddlers which went great but this one today just won’t come back to me. The JSON is valid as per the online JSON validator.

My wiki won’t respond to clicks at all, it just sits there.

bobj

My main wiki is about 25000 tiddlers. Each time I want to try another saver etc I export them into chunks of 2000 for reimport, it’s slow but it doesn’t crash for me at 2000 so if there is a limit, it’s over that.

High on my TW wish list has been an import utility that’s more robust with a progress meter and not needing to display all the tiddlers with the checkboxes.

One thing of note is that when it’s okay for my tiddlers to be in plugin form (static lookup database like my company address book) I regularly import one json of 50000 and it’s quick. The import tiddler only shows 1 tiddler there, so it’s definitely the bottleneck.

@stobot, Is there a MB limit maybe, although I have tried to halve the file so its only 1.8Mb and it still fails.

I attach the half file for interest. Some tiddlers, ie. the gun tiddlers, are large with descriptive text whilst the others are relatively small.

SA_Converted_HTML.json (1.7 MB)

bobj

@stobot,

an update, I re-ran the hafl file and it eventually came back with the list of tiddlers ticked.

I now click the Import button and go away and have a half of stout :slight_smile:

I like your idea of the progress bar though, there is nothing like some feedback.

Environment: Mac OSX Tahoe 26, 24G RAM, Firefox V147.

bobj

Keep in mind you are actually doing two things here, including the JSON file in your wiki (using import or drop), then the second step in converting it to separate tiddlers, you are using the import mechanism to do both.

Rather than hit import, you could rename the import tiddler, and access the file as is, by changing plugin-type to plugin the tiddlers within become shadow tiddlers in a plugin.

Thus your filters may need to start [all[shadows+tiddlers].. there is no need for the Import button this way. If you did need to make them into tiddlers you could make a tool to edit all the shadows into tiddlers, but now you can time them, batch them, and provide a prompt or exit from that process.