How to disable all lists update on every click to make Tiddlywiki work fast with 10000+ tiddlers?

@Siniy-Kit another approach is to take a “html” snapshot of a tiddler in html and display that rather than rendering the whole tiddlers and lists therein, again, each time the tiddler is opened.

Attached is my snapshot tool that illustrates this, but expect you may need to test and customise for your use. Please try on tiddlywiki.com before installing in your wiki.

snapshot.json (6.2 KB)

1 Like

Thank you Tones. I’ll test it. I think my problem is not many lists refresh. The main problem is LONG lists. For example here is my experiment
many short lists Интернет-магазин Heeg.html загружается..... — Heeg.html

and the same TW but long list open Интернет-магазин Heeg.html загружается..... — Heeg.html

@Siniy-Kit when I follow your second link I only see 4 columns with short lists Information/Catalogue/Our address and Call us. So I can see a long list with a view to using the shapshot tool.

This snapshot approach will not stop all list updates, only those in a tiddler (or part there of) with a snapshot.

  • I assume this is a single file wiki, not node?

I think there would be value getting some metrics to determine if what you ask for is an actual performance bottleneck before looking for a solution.

I am not sure if this approach is going to help you, however I have a hunch it can.

A few thoughts;

  • The snapshot (by the Wiki Author) takes a copy the the HTML rendered for a particular tiddler and stores it in another tiddler. When viewing the original tiddler the “pre-rendered” HTML is displayed rather than its body, this means the list widget need not be rendered at “run time”.
    • Theoretically this allows you to pre-render and save the result for visitors, however the tiddler view will/may not respond to changes unless another snapshot is generated.
  • In the case of an online shop I am not sure how a html snapshot will work with interactive products eg add to cart component. However it works a bit like a static html tiddler while still being inside the tiddlywiki.
  • In TiddlyWiki 5.2.x the new cascade mechanism for the view templates may help implement this is a smart way. The html snapshot can be rendered without any macros and other code be included and render time.

Ideas;

  • The concept of pre-rendering tiddlers makes use of the standard rendering through the body template. However it is possible to have an alternate body template designed to support pre-rendering of tiddlers in particular ways, this may include pulling content such as pre-rendered lists but include other interactive content.
    • The question is will this help meet your objectives to enhance performance?
    • Perhaps a mechanism to transclude html snapshot’s as parts of a tiddler makes sense?
  • Pre-rendering could actually pull html from the file system, which would allow it to be externalised and cached separately whilst existing within the interactive tiddlywiki.
  • I can imagine the idea of navigating to snapshot tiddlers by default with only the designer being in a position to rerender the tiddlers snapshot using the wikitext and $list widgets etc… If a tiddler need not be re-rendered on the live site the source tiddler could be removed from the published wiki.

I hope I am helping, not confusing you, but it seems this is an area that has a number of innovative possibilities.

second link has a mistake. here is write link Интернет-магазин Heeg.html загружается..... — Heeg.html

is it possible to make automatically “html” snapshot of all !is[system] tiddlers “on open” my TW?

@Siniy-Kit I can drop my package on your site and it seems ready to work, but as a rule it takes a snapshot of the result of rendering the body, on the link you showed me most of the content is rendered via some other method such as the view template.

I think it could be modified to capture the body and result after applying the view template, but I admit I struggle with the other language on the site.

Here is another package of two buttons I use to open a new window with the full rendering, which demonstrates what could be done with the snapshot tool, by opening the tiddler in a window including with the view template applied.

new-windows.json (13.3 KB)

Again this can be dropped on your site.

I imagine the answer is yes, but I need to know more about your solution. And I suspect there may be better ways.

  • Is this a single-file wiki or node?
  • does the user only have read-only access?
  • do you only update it when you open it?
  • How does a users order get saved/dispatched?
  • What evidence do you have that it is the list widget causing slowness?, I can’t see any slowing!

Unfortunately snapshot is not working on my site
here demo with snapshot Обувь и всё для обуви — Уход за обувью
and here is snapshot tiddler Обувь и всё для обуви — Уход за обувью

it is empty, because my tiddlers text field is emty. And all data to it is put by template Обувь и всё для обуви — Уход за обувью
with special tag $:/tags/ViewTemplate
so all content is not in tiddler-body and there is nothing to snapshot…

and if I put my data to text field of ONE tiddler my tiddlywiki file become 2mb large, and works slow too Обувь и всё для обуви — Уход за обувью

and snapshot is not empty Обувь и всё для обуви — Уход за обувью
but very big. So , our problem is big DOM tree?

As I said;

Thats why I suggested this;

However I asked

See if you can make use of my code, and ideas and feel free to direct any questions to me however I can’t help much more right now due to;

  • The use of other language’s on your site (Its hard for me to follow/explore)
  • Not enough info from you (See questions asked you have not yet answered)
  • Personally busy.

I’m not in a position to try this right now, but I was wondering if using the select widget for selecting a tiddler from a large list would have any better performance than selecting from a displayed list. Just curious.

I managed to do a little testing today with a displayed list versus the select widget and found no significant performance difference. I was expecting this to some degree since the same list is being generated. I was curious to see if the fact that the list wasn’t always displayed in the select widget played a performance role. Answered.

I just wanted to point out people are using the new event catcher widget to reduce the complexity of the HTML generated using the list widget. Apparently it has performance advantages.

Perhaps this could help?

Also consider reworking the code so when you click on a list item it updates something that is not visible and does not cause a need to refresh the tiddler containing the list. For example add the item to the cart, but don’t indicate it is in the cart in the original list.

@Siniy-Kit what part of the site are you experiencing performance problems with:

  1. How long it takes to show a long list the first time?
  2. How long it takes when you click on something in the list to show the item?
  3. How long it takes to go forwards and back in the long list?

Attached is a very quick experiment at implementing a new layout for performance testing purposes that avoids the need for the zoomin view, it also removes the sidebar. Please try this on your site and report back if it helps with performance.

  • Import all the tiddlers
  • Ctrl+Shift+L and choose the single tiddler layout.

You could also try combining this with the refresh disabling widget, setting a filter such that the refresh is only disabled when single tiddler layout is active.

single-tiddler-layout.json (2.6 KB)

The next thing to try would be to simplify the templates used to render the product details.

3 Likes

Thank you, Saqimtiaz. I’ll Try it.

Here I make experiment with 2500 tiddler list

and it takes 4-5 seconds to open Recent tiddlers on my 10 year old computer and low price android smartphone and 1 second on (not my) new computer.

Absolutely the same result I see on long list in on-line stores people make on my tiddlywiki template Интернет-магазин Heeg.html загружается..... — Heeg.html my computer opens list on this link 6 seconds (to many)

So I make my experiment. I render static cut of tiddler with long list
Косметика для обуви it opens 1 cecond on my old computer…
and tiddlywiki variant Интернет-магазин Heeg.html загружается..... — Heeg.html

9-10 seconds full page load

Hey @saqimtiaz, thanks for the $refresh widget. I took a crack at improving it in a way that it would (alternatively) allow for an arbitrary filter to control refreshing, not just a subfilter with the list of changed tiddlers as input. A bit like the way that $action-listops works. This way it’s maybe a bit more straightforward to couple refreshing to the existence of some state tiddler. Towards that goal I also added a parameter that inverts the logic and refreshes when the filter evaluates to an empty result.

$ _yaisog_modules_widgets_conditional-refresh.js.json (2.4 KB)

Attribute Description
subfilter An optional subfilter expression, which takes the list of changed tiddlers as input. The content of the widget refreshes only if the subfilter produces any output.
filter An optional filter expression. The content of the widget only refreshes if the filter produces any output.
invertFilter Optionally set to “yes” to make the widget refresh if the filter does not produce any output. Does not apply to the subfilter parameter.

For dependence on a single state tiddler $:/state/toggle-refresh to not exist, use:

<$conditional-refresh filter="[[$:/state/toggle-refresh]!is[tiddler]]">
    <$log message="Freshly refreshed." />
</$conditional-refresh>

To display the log message only when no state tiddler with the prefix $:/state/refresh/ exists, you could use

<$conditional-refresh filter="[prefix[$:/state/refresh/]]" invertFilter="yes">
    <$log message="Freshly refreshed." />
</$conditional-refresh>

The !prefix[] syntax wouldn’t work here, as it would return a large number of tiddlers.

A beautiful application of this widget will be (I hope) to pause refreshes for the invisible tiddlers in the zoomin storyview, instead of folding / unfolding them, to make the UI more responsive. All I need to do (I think) is include $conditional-refresh in the ViewTemplate and tie it via the filter parameter to the current-tiddler field of $:/HistoryList.
Folding has the disadvantage that the tiddlers need to be rendered anew when they are shown again, which would be slower than refreshing, so switching between tiddlers should become somewhat faster.

Have a nice day
Yaisog

PS: Your code won’t allow to use variables like <currentTiddler> in the subfilter expression. I think you need to change this line:

filteredChanges = this.wiki.filterTiddlers(this.filter,null,this.wiki.makeTiddlerIterator(changedTiddlerTitles));

and replace null with this. That made it work for me.

2 Likes

Glad to hear you got something working to your liking.

You can also simply ignore the input list of changed tiddlers and start your filter with all[tiddlers] or all[shadows+tiddlers]

I would probably use something along these lines: [prefix[$:/state/refresh/]count[]match[0]]

The refresh mechanism passes each widget the list of tiddlers that have changed and allows the widget to determine if it needs to refresh or re-render in response to changes in those particular tiddlers. Unless you cache the titles of changed tiddlers for which the refresh was not carried out and force a custom refresh using that data when needed, you would need to re-render anyway.

For developers, I have an approach to reduce refresh My approach to allow widget to debounce refresh · Discussion #6654 · Jermolene/TiddlyWiki5 · GitHub

My wiki has 15000+ tiddlers too, so performance is on my mind too.

Hi Saq, good to have you back. And in tip-top shape I see… :wink:

I did think about this, but wanted to differentiate the same way that $action-listops does. More semantic, I guess?

Oh man, how did I not come up with that?
This is good enough to get rid of the ugly invertFilter parameter. BUT, I’ll use [prefix[$:/state/refresh/]count[]compare:number:eq[0]]. :grinning_face_with_smiling_eyes:

You mean that tiddlers that were refresh-suppressed for a while can be outdated, and just calling a refresh on the tiddler itself will not retroactively apply previous updates? Ah, dang. If I need to re-render anyways, I might as well just stick with folding/unfolding… :cry:

Not unless you cache and provide a list of tiddlers that have changed since the last refresh. Each widget refreshes or re-renders itself depending on which tiddlers have changed in the wiki.

That would be too much work and too error-prone for the gains to be had, I think. Too bad.

It sounds as if we had a queueing mechanism into which you could push titles then trigger actions on those titles later would be a useful code pattern.

  • In this case it would be a refresh tiddler queue.
  • However there are many other applications of such a queue we could use if we extended this to push and pop LIFO/FIFO

The key here is we need tools that bridge the gap between variables (memory) and saving to tiddler fields (requires triggers). Imagine pushing and popping items on a memory based queue throughout a session only saving or discarding before closing the wiki.

  • The Key difference to any mechanism’s I am familiar with, is this queue can accumulate across time and multiple processes could access it. Without the need to commit with a trigger.