My TW's have memory issues

Developers
#1

I have a small VM where I host a growing number of small TW’s. Last night I found that my VM is running out of memory during routine processes, and the main hog is TW.

In terms of stored size, all my TW instances are small, human-written text; largest size so far is about 1.5M.

At runtime, I see that an instance of a ‘tiddlywiki’ process takes up about 33M. Okay.

But, what’s available on the OS side shrinks by over 100M with each instance. I haven’t been able to track down those other 60-odd megs per instance, but if I’m running ‘top’, it’s clear that the amount of available memory is closely followed by the start and stop of a TW instance.

Looking ahead: if a TW instance needs, say, 40M, and this VM isn’t doing much else, then it should be able to handle around 40-50 instances of TW. More conservatively let’s call it 20. Instead, we’re hitting trouble when the number of instances passes about six or seven.

Two questions…

  • Are there ways to make instances of TW share memory? If we all read from the same base libraries, could that 30M-ish size go down some?

  • That aside: if running one TW instance does take 30-35M, where are we using up about 100M total? Is there a number somewhere that I might turn down? Or maybe I’ve done something wrong to be causing this?

Thanks.

#2

I think most of the memory is lost through rendering. So if you closed all your tiddlers, clicked on the “open” tab, and collapsed the sidebar, your numbers would come very close to your process numbers. Not completely, because even a blank screen needs to be rendered.

Since it’s mostly about rendering, sharing base libraries wouldn’t make much of a difference.

Also, since it’s about rendering, having the “recent” tab open will take more memory, especially if there are a lot of tiddlers. That can be hacked to show fewer tiddlers, or just use the “open” tab more. Likewise, having lots of tiddlers open will take even more memory.

In terms of strategies for multiple TWs, I would think in terms of combining all those TW files and then viewing them separately using tags or structured naming.

#3

Are you talking about Node.js server instances ?

I’m not 100% sure, but I think he’s talking about server instances. …

#4

Did you have a closer look at Bob https://github.com/OokTech/TW5-Bob, which should allow to serve multiple instances with 1 executable

#5

Yes, this is a setup where a number of independent TW instances run, each under ‘node’.

Each of those is independently called by a varying set of browsers.

#6

This is funny.

I started trying to do something like that; ran into problems; the fix was, “give each set of users their own name on DNS.”

#7

Which OS. How do you start your node instances? Have you had a look with htop ? It’s able to show a tree like structure.

#8

Yea, If different users are involved, it may be better to keep them separated.

#9

Which OS: Linux.

Which set is ntop usually part of? I don’t find it as its own package.

FWIW, on most *nix flavors, a call like ‘ps auxwf’ (or the like) puts out an elaborate tree. If the question is, “is tiddlywiki being called by node here?” the answer is “yes”.

#10

Which Linux exactly?

Htop … On ubuntu apt install htop should find it.

Where does this 33M value come from?

#11

I did start some of them with ubuntu 20.04 and every one of them needs about 100M

#12

$ echo “How many tiddlers?” ; ps -ef | grep giddl | wc -l ; echo “Memory stats?” ; free -m
How many tiddlers?
1
Memory stats?
total used free shared buff/cache available
Mem: 975 550 309 1 115 297
Swap: 972 599 372

…then, with five TW instances running:

$ ps -ef | grep tiddl | grep node | wc -l
5

…and, with no user-side connections yet, memory is being bobbled up:

$ free -m
total used free shared buff/cache available
Mem: 975 845 61 1 68 25
Swap: 972 778 194

Maybe not obvious given the formatting here: the VM is already using up almost 4/5 of its swap space. I can also say that interacting at the shell gets slow.

Now, what’s been confusing me: with those instances running, a call to the system has five ‘node’ instances taking between 1.9 and 11.2 megs each, for a total of less than about 40M.

But, something somewhere is eating up all that memory; and, when I send TW away, all that hogged memory comes back.

#13

Oops, my bad:

Nope, my mistake, I was misreading ‘ps’; it’s been showing, not base amount of memory, but percentage. Oops! So no, not “10 megs”, but “10% of available”.

#14

Anyhow. So I guess the answer is, “TW takes about 100MB, period.” At least in its Node.JS variant.

“Variant”? Is there a way to have it running in multiple net-reachable states (i.e., within a web browser, like we do), without Node.JS?

#15

Oops: part of my trouble is that I’ve been running tiddlywiki under screen(1), instead of some standard shell script. Active instance? Memory hog!

Adjusted that. Things are much better.

I’d still like to find ways to tweak memory usage, for instances we want to stick around but that aren’t very large or active. (Also I would like to be able to fly, at will.)

Flying at will -- you should try it
Flying at will -- you should try it