Typing frozen for 45 seconds when changing title, huge number of Relink orphans

If you disable relink and or other plugins, save and reload does the performance problem go?

This may also eliminate a relinking configuration setting that is introducing the delay. If relink off not problem, relink on a problem you can be more confident. But you can do the same with all plugins to see if the fault is caused by one of them.

1 Like

I too would like to know if there’s a difference, because there’s clearly a problem, and if it’s Relink, I must fix it. These delays are egregious.

@Flibbles I disabled Relink, reloaded the wiki, and the problem went away: On typing in the title field, the wiki did not freeze, the “update tags and lists of other tiddlers” notification instantly appeared, I could immediately click the save and close button.

I then re-enabled Relink, re-loaded the wiki, and the problem is back: On typing in the title field the wiki freezes for 45 seconds before the relink notification appears.

I don’t know if this means anything but I looked in the cotents tab of the Relink plugin. Every item in the list of shadow tiddlers is bold except for this one: $:/plugins/flibbles/relink/Filters/Orphans

Perhaps copy that tiddler rename and remove tags or module type. The delete the overwritten tiddler and see if that fixes it

@TW_Tones, Resetting the Orphan’s filter shouldn’t really make a difference to the load.

@Sapphireslinger, when you say the problem was fixed, do you mean everything was snappy fast then, or did you still have 5-10 second delays with basic operations like opening tiddlers? Was the 45-second one-time delay the only thing that disappeared?

@Flibbles The 45 second one-time delay on typing to change a title was the only thing that disappeared when disabling Relink.

I ran some more tests. It may not have any bearing but I discovered that :

Opening the first tiddler of the session

  • If it is a new blank tiddler it opens fast 2:20
  • But If the first tiddler to be opened is an already existing tiddler it opens slow 45:65. After that, saving open tiddlers is fast, and opening new tiddlers of either type after that is fast also…
  • However, if the first tiddler of the session is a new blank tiddler and I do not open any more tiddlers but save that new one, it will save slow (49 seconds). After that hurdle however, opening an already existing tiddler will be fast and opening and saving tiddlers of either type will be fast.
  • But if the first tiddler of the session is a new blank tiddler (opens fast) and then I leave it open and go open an already existing tiddler, the already existing tiddler will open slow. Then I return and save the first tiddler and it saves fast instead of slow, and opening and saving tiddlers is fast after that.

However none of that solved the main problem (the change-an-existing-title-delay), that remains constant.

Some simple “Relink Enabled” versus “Relink Disabled” stats.

(A slight difference of a few seconds should not mean anything because I’m working with my smartphone’s stopwatch and it is harder to hit the button just right, than with a real stopwatch.)

opening wiki:
Relink Enabled: 6:38
Relink Disabled: 6:43

opening first tiddler of session - if a new blank tiddler :
Relink Enabled: 1:74
Relink Disabled: 2:20

saving first tiddler of the session - if that same new blank tiddler
Relink Enabled: 48:51
Relink Disabled: 49:65

opening second tiddler of session (an already existing one):
Relink Enabled: 1:72
Relink Disabled: 1:66

saving second tiddler of session (an already existing one):
Relink Enabled: 5:70
Relink Disabled: 5:81

A bit of another scenario:

opening first tiddler of the session - if already existing tiddler
Relink Enabled: 45:37
Relink Disabled: 45:65

saving first tiddler of the session - if that same already existing tiddler
Relink Enabled: 5:87
Relink Disabled: 5:94

More Stats:

I am using Linux Mint, and Firefox.

I see. Wow. Yeah, it looks like Relink is only responsible for that first 45 second delay, specifically concerning changing a character in a tiddler’s title (which I could only mitigate by pre-caching in the background or something). Everything else is probably within the margin of error. Which means, in general, Tiddlywiki is just super slow for you. I honestly don’t know how you’re tolerating it.

It’s been well known that Tiddlywiki becomes super sluggish once you start cresting 10,000 tiddlers. We’re a club of people all in the same helpless boat. I would love for the Tiddlywiki core team to really take performance seriously. I would be the first to jump and start profiling code and tackling big offenders.

A lot of us (core team and super users) have in the past, and one reason this is somewhat vexed is tiddlywiki gives you enough rope to hang yourself. If you shortened the rope to stop people hanging themself, you would loose a lot of functionality.

I encourage people with performance problems to simp[ly ask for help, ideally with a link to an example, because there is a number of “low hanging fruit” and a lot of performance improvement methods.

  • Experience shows if someone makes an example to share they often solve their problem while doing so and if not someone else solves it quickly.

This is quality in many areas of software, for example SQL allows you to ask stupid questions that cause serious performance issues because it is crontrary to the structure of the data, it will still do it for you, but you should not have asked :nerd_face:

You’re right. I may be being overly harsh here. I’m really only speaking from my experience where getting performance PRs merged takes months of convincing.

When I say taking performance seriously, I mean taking on some serious tasks, such as underlying changes to the way widget hierarchies are handled, or adopting lazy filter operators to reduce memory thrashing. Big fruit, Not low hanging. But they’re things I and a few others have been pushing for for a while.

2 Likes

@Flibbles and @TW_Tones I think the Relink delay problem has been partially solved.

I didn’t think it was the large number of tiddlers that was causing the Relink problem because before I updated to 5.3.3 I had the same number of tiddlers and no Relink problem.

And I didn’t know how to begin investigating which of my modifications to my wiki was clashing with the update to cause the problem because I hadn’t kept track over the years and didn’t know where to find them.

With my own wiki so messed up, I thought it was as good a time as any to try switching over to a completely different edition, the beautiful Timimi site.

But asking the question how to import all my non-system tiddlers into another tiddlywiki CleverNote (the 1.0.49 version at the Timimi site) got me thinking again - Why not try that with the standard empty.html to solve this problem here?

Previously I only knew how to update my tiddlywiki the standard way - dragging it onto the update page. It never occurred to me that It was possible to download an empty.html first and then export all my non-system tiddlers over to it, thus getting rid of all my modifications at once.

Once I realized there was this alternative way to update, I searched in this forum for exporting non-system tiddlers and this came up: Simplest way to port tiddlers to a new TiddlyWiki where Telumire says:

You can be more selective by exporting your tiddlers with the advanced search, you could write

[all[tiddlers]!is[system]]

for example to only get non system tiddlers, then export in json. To import the tiddlers in the new wiki, drag and drop the json.

I had had no idea all these years that was what one did with jsons - drag and drop them like plugins. I remember one time laboriously opening up a json and trying to recreate it in my wiki ^o^ >_< (Or make a tiddler and put the json contents inside or something.) If I ever did know at one time, I must have promptly forgotten it.

Now I have this broad idea that a json is just code for a bunch of tiddlers, like a plugin. And @TW_Tones, now I can understand more of your tips here.

So did this alternative way of updating and sloughing off all mods solve the Relink problem?

Well so far, now I only have the Relink problem on the first tiddler of the session. After that, relinking is snappy! I can live with that!

Now I will import all the rest of the plugins and mods again as I feel or discover that I need them and update you on any changes.

Thank you both for all the help.

1 Like

@Flibbles and @TW_Tones I’m so sorry I was mistaken, and gave incorrect information in my last post. Starting with an empty tiddlywiki and porting over only non-system tiddlers, did not correct the problem at all, not even temporarily.

I thought I was running those tests with the Relink plugin installed, when in fact Relink was NOT installed. (After drag and dropping the Relink plugin, I had clicked the “save” and “reload” buttons up in the notification bar instead of clicking the “save” button down in the Tiddlywiki and closing out the tab. So the relink plugin hadn’t gotten installed.)

After truly installing Relink

Today I installed Relink properly, and this time made sure it was showing up in the plugins before running the tests. The long delay remains constant.

I then deleted Relink and ran the tests again. This time there was only a long delay on the first rename, subsequent renames are fast.

Additional Information

My tiddlywiki used to be 37 mb. After transferring all my non-system tiddlers over to an empty, the size is only 32.4 mb.

Sorry I can’t give you a copy to delve into and fix, it’s a personal wiki.

I don’t know how much of a difference it will make, but I recently pushed a new version of Relink which does better about caching. It probably won’t fix your problem, but it does resolve a caching issue that was slowing things down.

1 Like

You are right, there is still the delay.

@Flibbles

  1. I would like to return to using my snappy older version of Relink (1.9.1) Are there any really bad bugs I need to look out for if using it with Tiddlywiki 5-3-3?

I saw the mention of a bug here. I could be careful about not having other stuff open when relinking.

  1. Where can I find the Relink versions between 1.9.1 and 2.4.2? I would like to test to see if there is one later than 1.9.1 that doesn’t cause the long delay.

Why I Ask

I was combing through all the past backups of my Tiddlywiki to see exactly when the problem emerged. (I opened a copy of each backup and tried to relink something to see how long it took, and took note of the Relink version.)

My Tiddlywiki was upgraded to 5.3.3 by December 25th. And I continued to use Relink 1.9.1 until sometime around February 29 when I upgraded to Relink 2.4.2. The backup labeled 2024-02-29 is where I first see Relink 2.4.2 and a long delay. The backup before that labeled 2024-02-27 has Relink 1.9.1 and no delay.

  • With Tiddlywiki 5.1.22 Relink 1.9.1 was lightning fast.
  • With Tiddlywiki 5.3.3 Relink 1.9.1 had maybe a half-a-second delay. Still very fast.
  • With Tiddlywiki 5.3.3 and Relink 2.4.2 the 45-second delay emerges.

So is there any reason I should not go back to Relink 1.9.1?

hi,
I know, I’m late to the story, but I did try to recreate a wiki similar to yours in case of numbers.

  • 14500 tiddlers and the size of 27MByte.
  • Every tiddler has 4 links to other tiddlers
    • 90% of linked tiddlers do exist
    • 10% do not, so there should be a good variety

Renaming a title needs about 3 seconds the first time.
So my testcase is definitely not sufficient to reproduce your problem.

So I would like to better understand the structure of your tiddlers.

  • How many links to other tiddlers do you have in 1 tiddler
  • Do you use macros, that contain links to tiddler titles, so relink needs to check them
    • If yes - How many custom macros do you use?
  • Is it possible to name the plugins you use
    • This may give us some info about which macros are in use.

-mario

@Flibbles … What is the most time consuming “relink” action? So I may be able to include it in the code that creates the test tiddlers.

Click to see the code to create test tiddlers from the dev-console
var ntext = "" +
/* The next line can be copy / pasted, to make the wiki bigger */
  "123456789 123456789 123456789 123456789 " + "123456789 123456789 123456789 123456789 " +
  "123456789 123456789 123456789 123456789 " + "123456789 123456789 123456789 123456789 ";

var xtext = "" +
  "123456789 123456789 123456789 123456789 " + "123456789 123456789 123456789 123456789 " +
  "123456789 123456789 123456789 123456789 " + "123456789 123456789 123456789 123456789 ";

var tiddler = $tw.wiki.getTiddler("GettingStarted");
var aliases = "";
var x = 0;
for (var x = 0; x < 13; x++) {
  for (var i = 0; i < 35; i++) {
    tag = "t-" + i;
    for (var k = 0; k < 32; k++) {
      aliases = "a-" + x + "-" + i + "-" + k;
      $tw.wiki.addTiddler(new $tw.Tiddler(tiddler, {
        title: "x-" + x + "-a-" + i + "-" + k,
        text: ntext + " [[" + "x-" + x + "-a-" + (i-1) + "-" + k + "]] " + " [[" + "x-" + x + "-a-" + (i+1) + "-" + k + "]] " + 
              xtext + " [[" + "x-" + x + "-a-" + i + "-" + (k-1) + "]] " + " [[" + "x-" + x + "-a-" + i + "-" + (k+1) + "]] ",
        tags: [tag],
        aliases: aliases
      }/*, $tw.wiki.getModificationFields()*/));
    }
  }
}

@pmario Thank you so much.

  • How many links to other tiddlers do you have in 1 tiddler

Is there a code I can use to find which of my tiddlers have the most links to other tiddlers?

  • Do you use macros, that contain links to tiddler titles, so relink needs to check them
    • If yes - How many custom macros do you use?
  • I think custom macros may turn out to be the problem. I thought I had gotten rid of all modifications by porting all non-system tiddlers over to an empty Tiddlywiki, but no, I just now found some book quote macros that are evidently not considered system tiddlers. And I know I have journal and reading history macros. I found some by searching in my Tiddlywiki using keywords like “quotes” and “macro”. I will also look up all my old forum posts to find code I was given in the past. It will take me some time to discover them all and try turning them off by enclosing them in formatting boxes. Will keep you updated.

  • I heavily use Subsume list filters to organize my wiki instead of TOCs, but I’ve been doing that for some time:

<$list filter="[tag[REPLACE]sort[title]]">
<$macrocall $name=subsume tid=<<currentTiddler>>/>
</$list>

Is it possible to name the plugins you use

I already tried putting all my non-system tiddlers in an empty Tiddlywiki with no plugins (except Relink) and the problem persisted. I will now focus on turning off the macros as I am now going forward with the “no plugin” fork. (It has relink and subsume back in there now though!)

@pmario, the most expensive operation Relink performs is indexing. This occurs the first time the “Relink References” panel is opened, or a character in a tiddler you’re editing is modified (but before the save). It sounds like @Sapphireslinger isn’t using the “Relink References” panel, and it’s all about typing that first character into the title bar of that first tiddler.

@Sapphireslinger, so to make sure I understand, that 45 second delay is occurring the first time only? Or it’s occurring every time you try to edit a tiddler’s title?

…Actually, looking back up, I’m seeing you’re saying the delays happen when saving or opening tiddlers. Which makes no sense to me. Relink isn’t supposed to be doing anything at those time. Is that really when it’s happening?

You can find a pre 2.0 version of Relink here. Just save that file to your desktop and then drag it over. This is the version from before I switched over to using a TW indexer internally… which was a change I made to improve performance :confused:

It won’t have support for the latest Tiddlywiki 5.3.x stuff, but there shouldn’t be any major bugs.

It’s not needed to be exact. I only wanted to have an estimate. Is it 5, 10 or 100 links. As I wrote my test tiddlers use 4 links at the moment and the indexing seems to need maximum 3 seconds on my machine.

Which version of the subsume macro do you use? I think subsume is a plugin. I found this version: The Subsume Plugin — Turn links into sliders! linked from this thread: [tw5] Reboot of Subsume plugin

I do not use it, so I’ll need some time to investigate the mechanism and see if I can come up with auto-created test tiddlers.

How many subsume macros do you have in 1 tiddler - in average?