Troubleshooting advice requested for single file wiki experiencing Out of Memory issues

#1

I’ve been having an issue with my main single file Wiki. Over the last month I’ve begun to get intermittent but frequent Out of Memory errors usually when importing tiddlers or attempting to save. I was hoping to get some troubleshooting advice.

Details

The wiki is on V5.3.8. It’s a single file encrypted wiki with a file size of 74,249KB. I don’t embed images in the wiki with the exception of the favicon which is a 17kb png file when downloaded. Images are stored in subfolders of the wiki’s location.

Plugins:
$:/core - 5.3.8
$:/plugins/bj/Calendar - 1.11.0
$:/plugins/inmysocks/WordCount - 5.1.10-prerelease
$:/temp/info-plugin -
$:/themes/tiddlywiki/snowwhite - 5.3.8
$:/themes/tiddlywiki/vanilla - 5.3.8

Number of tiddlers: 30701
Number of tags: 597
Number of system tiddlers: 75
Number of shadow tiddlers: 2348
Number of overridden shadow tiddlers: 28

Number of Missing Tiddlers listed in the More > Missing list: 9705.

I should also mention my TiddlyWiki has undergone a large expansion in the amount of tiddlers I have this year as I expand the scope of the file. My January 1st backup file is 59,080KB in size with 20,662 tiddlers.

I have a size check tiddler I’ve used for this description and these are the sizes of the top ten largest tiddlers.

$:/core 2115455
tiddler-name-removed 124742
tiddler-name-removed 109019
tiddler-name-removed 107269
tiddler-name-removed 99101
tiddler-name-removed 95912
$:/boot/boot.js 89769
$:/themes/tiddlywiki/vanilla 88543
tiddler-name-removed 84707
tiddler-name-removed 2021 80916

The problem has been seen in up to date browsers for Chrome and FireFox on two separate machines, both HP machines on Windows 10 Pro (on both machines the wiki is loaded from a network location – different location, different network).

Desktop PC, Intel Core i5-9500 3GHz, 8GB DDR4, Intel UHD
Desktop PC, Intel Core i5-8500T 2.1GHz, 8GB DDR4, Intel UHD

Chrome usually reports memory usage at 754MB. I’ve seen it as low as 600 and as high as 2.1GB.

I have a non-encrypted version of the wiki with sensitive tiddlers removed (56,296KB , 29,238 tiddlers) where I’ve yet to experience this issue.

I’m not sure what more I can provide in terms of the problem.

Things I have already tried with little or no effect.

Exporting all tiddlers into a fresh (empty) Tiddlywiki.
Changing browsers.
Changing machines (thinking there might be a hardware issue on my main machine).
Removing encryption.
Making sure no tiddlers are open when saving.

Things I’ve considered but not yet explored

Is my Wiki too big for a single file wiki? Do I need to look into other options (such as Tiddlywiki on Node.js)
Should I split my wiki up into separate single wiki’s?
Do I need a bigger boat? (a higher preforming computer, more RAM)
Is my wiki corrupted in some way? Is there something I should be looking out for to find out?

Current workarounds/ actions that have an effect.

I have more success in saving when the sidebar is on the Contents tab (I used to usually have it on More > Missing Tiddlers)
I now use a near empty TiddlyWiki file to draft tiddlers. If I need to modify existing tiddlers I move it into the draft wiki, make and save changes – import back and then attempt to save (if impot succeeds). It’s probably not worth mention that the near empty TiddlyWiki file is way faster than my main wiki.
I wait a few minutes between importing tiddlers to saving them.

So any ideas what else can I do to mitigate/ resolve the issue? Do I need to move away from a single wiki TiddlyWiki/ need more RAM? Something else I haven’t considered?

#2

I don’t have any real insights but FWIW:

Try to identify if there are any recursions (perhaps easier said than done). This is at least a known way to induce time outs so just maybe this is happening. Perhaps someone with Inspector tool expertise can explain if this is detectable therein?

Test out a “import half of the tiddlers at a time” approach; i.e first 50% into a new one and see if it works, then another 50 of the remaining (i.e 25%) and see if the new wiki still works, etc, and make sure to note what tiddlers you just imported if there’s suddenly a performance loss.

Do you have many custom viewtemplates?

Number of overridden shadow tiddlers: 28

Perhaps something critical among these?

Unlikely but: Might there be an incompatability between you tiddlers after an update? Perhaps test with an older TW?

Also unlikely but: Of course, you’ve already tested to remove the very few plugins you mentioned.

#3

I might try taking a backup from before the latest update and import my newer tiddlers. That sounds like a good avenue to persue. Thanks.

I haven’t tried removing the plug-ins yet but that’s mainly because none of these are new. I’ve had most if not all of them from when I first started the wiki back in late 2017.

Not sure how to identify recurssions. With tiddlers containing these still have an effect if not open?

I have had performance issues in the past with filters finding to many responses for one of my Defult Tiddlers and I changed my tiddler to stop that.

As for overiddent shadow tiddlers, they seem to be mostly related to the plugins. This is the list that comes up when I click on the figure (now showing 27 matches)

$:/config/bj/Calendar.json
$:/config/PageControlButtons/Visibility/$:/core/ui/Buttons/advanced-search
$:/config/PageControlButtons/Visibility/$:/core/ui/Buttons/home
$:/config/ViewToolbarButtons/Visibility/$:/core/ui/Buttons/info
$:/config/WikiParserRules/Inline/wikilink
$:/core/modules/widgets/macrorefresh.js
$:/DefaultTiddlers
$:/macros/bj/Calendar/diary.js
$:/macros/bj/Calendar/journalfmt.js
$:/macros/bj/Calendar/journalslinked.js
$:/macros/buggyj/Calendar/base.js
$:/macros/buggyj/Calendar/entry.js
$:/macros/buggyj/Calendar/entry2.js
$:/macros/buggyj/Calendar/ifnew.js
$:/macros/buggyj/Calendar/refresh.js
$:/plugins/bj/Calendar/license
$:/plugins/bj/Calendar/readme
$:/plugins/bj/calendar/sidbar
$:/plugins/inmysocks/WordCount/ReadMe
$:/plugins/inmysocks/WordCount/word-count.js
$:/plugins/inmysocks/WordCount/WordCountControls
$:/plugins/inmysocks/WordCount/WordCountViewTemplate
$:/SiteSubtitle
$:/SiteTitle
$:/state/showeditpreview
$:/temp/advancedsearch
$:/themes/tiddlywiki/vanilla/options/sidebarlayout

#4

Again, I’m grasping for straws here but:

Of the plugins you mention, I figure the calendar is something that is active all the time, and if it malfunctions then maybe it does so “all the time”.

Also, this specific core tid just might be dangerous to override, judging from its mere title:

$:/core/modules/widgets/macrorefresh.js

#5

Thanks.

According to the indormation in the macrorefresh.js tiddler

The tiddler $:/core/modules/widgets/macrorefresh.js is a shadow tiddler
It is defined in the plugin $:/plugins/bj/Calendar
It is overridden by an ordinary tiddler

I’ve disabled the Calendar plugin for now. I don’t think I really use it anyway. (not sure how to safely remove it).

#6

If you have 9000 tiddlers in the missing tab, it can be a problem for response time. So saving with Open or Recent tab should be preferred.

1 Like
#7

Hi Alan,

With AdvancedPerformance plugin you can identify filters that take more time in your wiki.
(Once the plugin is installed and activated check for the details in the performance reports)

AdvancedPerformance plugin by Evidently Cube

BR,

Eskha

#8

Performance is affected by what you can see, eg the length of the story, content in the sidebar, but it should not effect memory, although if a lot is visible it may have a slight impact of total memory.

  • Just closing the sidebar can make a big difference.

I am guessing that using encryption of the whole wiki means both encrypted and unencrypted will be in the memory.

I think out of memory comes from a run away process. The aforementioned performance tools or the browsers developer console may give you a clue.

  • Look for lists of lists or transcludes of transcludes, recursive filters but thinking about what is happening in front of your eyes ie a big list, compound tiddler or tiddler to tiddler lookups.

If you can have a wiki and an import that reliably fails, save a copy of both, so you can do it again and again whist tweaking until it does not reoccur.

#9

I went back to a backup made at the end of July (V 5.3.7) and tried adding the last few months worth of tiddlers. I kept getting Out of Memory errors in the attempt even when breaking up the imports into smaller chunks. As such I don’t think it’s related to the latest version change.

I’ve created a ‘diagnosis’ version of my wiki and installed the AdvancedPerformance plugin. It’s pretty interesting though nothing really stood out for me.

I then removed all non system tiddlers that featured the phrase ‘list filter=’. I’ve then proceeded to test the import and save features of the wiki. I’ve gone up to importing 100 test tiddlers at a time (json file is 485KB in size), saving, repeating and with the exception of one early incident I haven’t had a repeat of the Out of Memory issue.

I accidently (imported the wrong json) reintroduced the ‘list filter=’ tiddlers and continued the test with no more instances of the Out of Memory issue.

I tried adding the test tiddlers to my live wiki with no memory issues. This leads to me think it’s not a valid test.

I plan to reset my diagnosis wiki and run the two in parallel for a while and see what happens.

#10

Can you try to disable the word-count plugin as a test?

#11

Whoa. Unless my math is wrong, that’s 74Mb. I find TW is rough going at 30Mb. 10Mb is high for mobile use. The only fix is to get a bigger boat (more ram, probably) or use node.js with lazy-loading.

#12

My anecdotal experience does not confirm this. I have used an encrypted wiki slightly bigger than 10MB on an older smartphone and I’ve never got any Out Of Memory issues. The bottleneck was rather the slow CPU - saving (thus full wiki encryption) could take 20+ seconds. On a newer smartphone with faster CPU saving took 3-4 seconds only.

I’ve yet to reach 75MB in a single file wiki though, I’m afraid it will take a couple of years :stuck_out_tongue:

#13

    

#14

As I said, 30Mb is a lot even on a desktop. His machines are running 8gb ram. Upgrading to 16 or 32 Gb might help. I guess I confused things by adding that 10Mb is a lot on mobile devices, but that was just for background.

#15

Long before you get to out-of-memory issues, you have slow reading and writing, with the danger of corruption during writing. I was referring to the useful maximum size, not the possible-but-infeasible size.

#16

@Alan_101 Not sure if this is any help…

I’ve seen something similar (though not exactly the same) infrequently on my larger wikis (~50Mb) in Firefox. The symptoms are either no warning at all with a message “Out of memory” in the devtools console, or the system starts to behave really poorly and then the message appears. Worst case, it’s a total freeze ← though this is extremely rare.

Each time I have a ton of tiddlers open and I’m doing (let’s call it) “heavy macro work” – lots of open/edit/save. The point being, I think, is that it takes “all day” to get it into this state (and again, infrequently). It never happens on a freshly opened wiki.

Judging by your plugins, I’m guessing you’re authoring large tiddlers (why else would word counts matter?). Me too.

My take: The only part TiddlyWiki is playing is exposing a memory issue in Firefox.

#17

Thanks for the feedback everyone.

I’ve created a new version of my Diagnosis wiki based on the live one. Word count has been disabled, AdvancedPerformance plugin installed. I will keep it up to date as I do the live one. If I continue to experience Out of Memory errors I will remove the non System tiddlers with filters and replace them with a similarly sized amount of ‘test tiddlers’ and see if this has an effect. If it does I plan to reintroduce filter tiddlers bit a time going on the oldest modified articles first.

I definitely have performance issues with this wiki in that it’s often slow to save individual tiddlers as well as the complete wiki itself. As I think I’ve stated earlier, I redesigned my default tiddler index page due to encountering severe issues a few years ago but it’s still slow to edit. I haven’t fully grasped macros yet and don’t use them often. If I need to edit a large amount of Tiddlers at once I normally export them to JSON and edit them there. Out of memory issues has happened when the wiki has recently been opened/ page refresh but I have felt like I’m more likely to experience it the longer its been open, even if idle.

I should point out that although I’ve tested my Live Wiki on Firefox I tend to use Chrome day to day. (Chrome I use for locally hosted files, normally various TiddlyWiki’s, Firefox I use for the internet)

I get the impression 74MB is unusually big for an all text based Single file TiddlyWiki. With that in mind I’m contemplating breaking it up into smaller live wikis which and maybe have my existing wiki as a ‘Kitchen Sink edition’ where I import all the recently modified tiddlers into on a periodic basis so I still have a place where the connectivity between the different types of articles I have still functions.

I already do this for two of the types of tiddlers I’ve begun doing from the end of last year so I can access them on my mobile (iPhone with Dropbox app. My encrypted single file wiki stopped being accessible on there some years ago). The two of these files are currently about 6MB each. I just have to figure out how to split the rest of it up which will take some working out.