Is there a way to improve performance of many nested transclusions?

[Scott_Sauyet]

I’m still working on the Bible wiki I’ve been discussing. This latest version is having performance problems. I used the performance tool to isolate the two problematic filters, which were totally unsurprising. And I was wondering if anyone had a good way to improve them.

For this version, I wanted the books to have the contents of all their chapters transcluded. I was able to do so like this:

<% if [<currentTiddler>tag[Book]] %>
  <$list filter="[tag[Chapter]] :filter[get[book]match<..currentTiddler>] +[nsort[seq]]">  
  <h2><$link>Chapter {{!!chapter}}</$link></h2>
<$transclude $tiddler="$:/_/bible/templates/chapter" />
</$list>
<% endif %>

which transcludes this tiddler:

<div class="chapter">
<$let book={{{ [<currentTiddler>get[book]] }}} chapter={{{ [<currentTiddler>get[chapter]] }}} >
<$list filter="[tag[Verse]] :filter[book<book>] :filter[chapter<chapter>] :map[get[para]] +[unique[]] +[nsort[]]" variable="para">
<p>
  <$list filter="[tag[Verse]] :filter[book<book>] :filter[chapter<chapter>] :filter[para<para>] +[nsort[seq]]">
    <span class="verse">
      <sup class="verse-number"><$link>{{!!verse}}</$link></sup>
      <span class="text">{{!!text}}</span>
    </span>
  </$list>
</p>
</$list>
</$let>
</div>

And it’s too slow.

Far too slow.

To load the book Genesis, on my moderately configured laptop it takes 10 - 12 seconds, and on my much more powerful work machine, about 10. Genesis has 50 chapters containing 1,533 verses. It’s one of the larger ones, but far from the largest (Psalms has 150 / 2461 and takes long enough that browsers warn about long-running scripts.

The performance tool made it clear that two filters were the main culprits:

• [tag[Verse]] :filter[book<book>] :filter[chapter<chapter>] :filter[para<para>] +[nsort[seq]] runs 239 times for a total of 8.1 seconds
• [tag[Verse]] :filter[book<book>] :filter[chapter<chapter>] :map[get[para]] +[unique[]] +[nsort[]] runs 50 times for a total of 1.7 seconds.

(on the powerful work machine.)

I’m not sure how to interpret the performance result. Are those numbers the runtimes of the filters themselves or of the entire list widgets based on them?

In either case, does anyone have suggestions for improving these filters? (I did try combining filter steps, (e.g. [tag[Verse]] :filter[book<book>chapter<chapter>para<para>] +[nsort[seq]]), but there was no noticeable difference.)

Or is this simply fundamental? If I try to transclude that many tiddlers, will it simply take that long to render? (I can live with that. The previous versions, without trying to render entire chapters are fast enough.)

If you want to look at the raw tiddler data it’s at

http://scott.sauyet.com/Tiddlywiki/Demo/KingJamesBible/perf1/sources/TW5-KJV.json

and it’s structured like this:

[
  {"title": "Genesis", "tags": "Book", "book-seq": "1", "seq": "1"},
  {"title": "Genesis 1", "tags": "Chapter [[Genesis]]", "book": "Genesis", "chapter": "1", "seq": "2"},
  {"title": "Genesis 1:1", "tags": "Verse [[Genesis 1]]", "book": "Genesis", "chapter": "1", 
            "verse": "1", "text": "In the beginning...", "seq": "3", "para": "1"
  },
  {"title": "Genesis 1:2", "tags": "Verse [[Genesis 1]]", "book": "Genesis", "chapter": "1",
            "verse": "2", "text": "And the earth was without form...", "seq": "4", "para": "1"},
  {"title": "Genesis 1:3", "tags": "Verse [[Genesis 1]]", "book": "Genesis", "chapter": "1",
            "verse": "3", "text": "And God said...", "seq": "5", "para": "2"},
  {"title": "Genesis 1:4", "tags": "Verse [[Genesis 1]]", "book": "Genesis", "chapter": "1",
            "verse": "4", "text": "And God saw the light...", "seq": "6", "para": "2"},
  {"title": "Genesis 1:5", "tags": "Verse [[Genesis 1]]", "book": "Genesis", "chapter": "1", 
            "verse": "5", "text": "And God called the light Day...", "seq": "7", "para": "2"},
  ...
]

I am in charge of this structure, so if there is a reconfiguration that would help here, it’s certainly doable, but I would like to keep the one-tiddler-per-verse design if at all possible.

[EricShulman]

@Scott_Sauyet

<$list filter="[tag[Verse]] :filter[book<book>] :filter[chapter<chapter>] :map[get[para]] +[unique[]] +[nsort[]]" variable="para">
<p>
  <$list filter="[tag[Verse]] :filter[book<book>] :filter[chapter<chapter>] :filter[para<para>] +[nsort[seq]]">

One possible improvement would be to “pre-compute” the common portion of the two filters shown above, so that work is only performed once. You might also gain some more speed by combining the filter operators into single runs and eliminating the somewhat costly unique[] operator from the second filter.

Something like this:

<$set name=verses filter="[tag[Verse]book<book>chapter<chapter>]">
<$list filter="[enlist<verses>get[para]unique[]nsort[]]" variable="para">
<p>
  <$list filter="[enlist<verses>para<para>nsort[seq]]">

Let me know how it goes…
-e

Note: I just tried my suggestions on your latest version, and just by counting seconds (“1 mississippi, 2 mississippi, 3 mississippi”, etc.) it seems to be about 10 TIMES faster. For example, to display Psalms on my rediculously underpowered 9 year old HP pavilion desktop system took just 3 or 4 seconds, compared with nearly 40 seconds (including several browser “wait” messages) in the unmodified version.

[saqimtiaz]

Filter timings only include the calculation time of the filters themselves and now how long it takes to render any corresponding widget output. I suspect the real slowdown is in the rendering of the output rather than the filters. Two quick thoughts that came to mind:

• Replace all the link widgets with static HTML and wrap $eventcatcher around the whole thing to handle navigation.
• Consider using the Dynaview plugin from the official library, its designed to only render the portion of the tiddler that is in view.

If you created a single tiddler that held the combined text of Genesis and then checked if it had the same performance issues, it would help identify if the slowdown comes from the rendering of that much content, or from the filters and transclusions that piece it together.

[pmario]

@saqimtiaz

• Replace all the link widgets with static HTML and wrap $eventcatcher around the whole thing to handle navigation.

I think Eric’s filter improvements and Saq’s event-catcher idea will go a long way.

[Scott_Sauyet]

@EricShulman

One possible improvement would be to “pre-compute” the common portion of the two filters shown above, so that work is only performed once.

That makes sense, and should have been obvious to me!

@EricShulman

You might also gain some more speed by combining the filter operators into single runs

I did try that and noticed no improvement, but I will probably put it back for cleanliness. (They got separated during some debugging issue or another.)

@EricShulman

and eliminating the somewhat costly unique[] operator from the second filter.

It will be necessary for the output to be correct, I believe. I use it to wrap groups of verses into single paragraphs. Without the unique, I would probably duplicate every paragraph, once for each verse it contains. Or maybe I’m missing your thrust. But these are not very large lists. The very largest chapter has 126 elements, and I’m assuming unique is O(n log n)

@EricShulman

I just tried my suggestions on your latest version, and just by counting seconds (“1 mississippi, 2 mississippi, 3 mississippi”, etc.) it seems to be about 10 TIMES faster.

Fantastic!

@saqimtiaz

Filter timings only include the calculation time of the filters themselves and now how long it takes to render any corresponding widget output.

Thank you. Not only wasn’t I sure, I couldn’t even make a reasonable guess.

@saqimtiaz

I suspect the real slowdown is in the rendering of the output rather than the filters.

Well, the time listed for those two filters covered most of the (“mississippi”) time it took to render them.

@saqimtiaz

• Replace all the link widgets with static HTML and wrap $eventcatcher around the whole thing to handle navigation.

That’s sensible, just as a general principle, given the size of the data.

@saqimtiaz

Consider using the Dynaview plugin from the official library, its designed to only render the portion of the tiddler that is in view.

I haven’t looked into that. I will do so, but I’m very hesitant to use any external plugins. My thought is that this wiki is mostly throw-away, just a reasonable place to home the Book, Chapter, and Verse tiddlers. Others could use either the data or the basic wiki to build their own tools. But I still want it useful as a Bible, which is why I would like the full Books to be available if I can get reasonable performance. That makes me want to do the simplest code I can come up with. Plugins don’t fit well with that.

@saqimtiaz

If you created a single tiddler that held the combined text of Genesis and then checked if it had the same performance issues, it would help identify if the slowdown comes from the rendering of that much content, or from the filters and transclusions that piece it together.

Oh, that’s a very good idea. It’s one I can try right away… Done, and it renders in about one mississippi, so the bulk of the time is in TW filters and transclusions. And if I understand your explanation of the perf monitor output, then it’s almost all in the filters.

@pmario

I think Eric’s filter improvements and Saq’s event-catcher idea will go a long way.

I expect you’re right.

I will try these this evening.

Thank you all very much! This is my very first time running into any perf issues in TW; it’s great to have the help of the brain trust.

[EricShulman]

@Scott_Sauyet

@EricShulman

and eliminating the somewhat costly unique[] operator from the second filter.

It will be necessary for the output to be correct, I believe. I use it to wrap groups of verses into single paragraphs. Without the unique, I would probably duplicate every paragraph, once for each verse it contains. Or maybe I’m missing your thrust. But these are not very large lists. The very largest chapter has 126 elements, and I’m assuming unique is O(n log n)

The FIRST filter uses unique[] to avoid duplicating paragraphs, so the SECOND filter doesn’t need to apply unique[] again.

[Springer]

If you could refactor so that you had a (hard-coded) field like “end-of-para” then everything would be smooth and fast, as far as I can tell. Then you just have the chapter find the verses tagging it, ordered by seq, and add a conditional paragraph break if end-of-para is true.

You wouldn’t even need to have the paragraphs numbered, given that you already have the seq field, it seems to me.

[Scott_Sauyet]

@Springer

If you could refactor so that you had a (hard-coded) field like “end-of-para” then everything would be smooth and fast,

I’m sorry, but I just had to at that. My previous issues on this subject was exactly about how to avoid doing that, which was what I’d done on my first approach. The trouble is that I cannot start a <p> in one list item and put the </p> in a subsequent one, for reasons best described in Jeremy’s response.

That fix worked fine, and, in retrospect, it seems more logical now to have

[
  {tag: 'Verse', para: "1", ...}, {tag: 'Verse', para: "1", ...}, 
  {tag: 'Verse', para: "2", ...}, {tag: 'Verse', para: "2", ...}, 
  {tag: 'Verse', para: "2", ...}, ...
]

than my previous

[
  {tag: 'Verse', new-para: "yes", ...}, {tag: 'Verse', new-para: "no", ...}, 
  {tag: 'Verse', new-para: "yes", ...}, {tag: 'Verse', new-para: "no", ...}, 
  {tag: 'Verse', new-para: "no", ...},  ...
]

Which is much like your end-of-para, but works from the opposite direction.

I am here in the enviable position of being able to massage my data to support my presentation. I do have a self-imposed restriction that I want to keep the Book, Chapter, and Verse tiddlers as logically simple as possible, but within that, I’m free to do as I choose. At some point, I do want to come back and analyze another possible technique to do that out of new-para fields or end-of-para ones, just as a learning experience. It should be possible, but it’s not urgent at the moment.

Thank you for looking into this!

[Springer]

In haste, multitasking, but in case it’s helpful — my version of $:/_/bible/templates/chapter without requiring you to refactor anything.

<% if [<currentTiddler>tag[Chapter]] %>
<div class="chapter">

<$list filter="[<currentTiddler>tagging[]tag[Verse]nsort[seq]]">
<span class="verse">
      <sup class="verse-number"><$link>{{!!verse}}</$link></sup></span>
<span class="verse">{{!!text}}</span>
<$let 
seqplus={{{ [<currentTiddler>get[seq]add[1]] }}} 
seqpluspara={{{ [seq<seqplus>get[para]] }}}
>
<% if [<currentTiddler>get[para]!match<seqpluspara>] %>
<p/>
<% endif %> 
</$let>
</$list>
</div>
<% endif %> 

This works in a responsive way, for me. I also would trim the book-level stuff a bit, but having the above as content of the chapter template works fast for me.

Update: here’s my $:/_/bible/templates/book:

<$list filter="[<currentTiddler>tag[Book]tagging[]nsort[seq]]">
<h2><$link>Chapter {{!!chapter}}</$link></h2>
<$transclude $tiddler="$:/_/bible/templates/chapter" />
</$list>

OOPS, first version did not put the breaks in the correct place. Fixed now, I hope!

[Scott_Sauyet]

@Springer

OOPS, first version did not put the breaks in the correct place. Fixed now, I hope!

Went to reply noting that, only to find you’d already fixed it!

Thank you for the contribution.

Yes, you can do something like that to get the same default look as I’m getting. The trouble is that it is not logically the same. Some of the semantics are missing. And that makes it difficult to apply styles to the document appropriately.

My markup ends up with something like this:

Scott's HTML

<h2><a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201">Chapter 1</a></h2>

<div class="chapter">

<p>
  <span class="verse">
    <sup class="verse-number">
      <a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201%3A1">1</a>
    </sup>
     <span class="text">In the beginning God created the heaven and the earth.</span>
  </span>
  
  <span class="verse">
    <sup class="verse-number">
      <a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201%3A2">2</a>
    </sup>
    <span class="text">And the earth was without form, and void; and darkness <em>was</em> upon the face of the deep. And the Spirit of God moved upon the face of the waters.</span>
  </span>

</p>

<p>
  <span class="verse">
    <sup class="verse-number">
      <a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201%3A3">3</a>
    </sup>
    <span class="text">And God said, Let there be light: and there was light.</span>
  </span>
  
  <span class="verse">
    <sup class="verse-number">
      <a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201%3A4">4</a>
    </sup>
    <span class="text">And God saw the light, that <em>it was</em> good: and God divided the light from the darkness.</span>
  </span>
  
  <span class="verse">
    <sup class="verse-number">
      <a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201%3A5">5</a>
    </sup>
    <span class="text">And God called the light Day, and the darkness he called Night. And the evening and the morning were the first day.</span>
  </span>

</p>

While yours looks like this:

Springer's HTML

<h2><a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201">Chapter 1</a></h2>

<div class="chapter"><p>

<span class="verse">
  <sup class="verse-number">
    <a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201%3A1">1</a>
  </sup>
</span>
<span class="verse">In the beginning God created the heaven and the earth.</span>


<span class="verse">
  <sup class="verse-number">
    <a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201%3A2">2</a>
  </sup>
</span>
<span class="verse">And the earth was without form, and void; and darkness <em>was</em> upon the face of the deep. And the Spirit of God moved upon the face of the waters.</span>


<p></p>
 
<span class="verse">
  <sup class="verse-number">
    <a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201%3A3">3</a>
  </sup>
</span>
<span class="verse">And God said, Let there be light: and there was light.</span>


<span class="verse">
  <sup class="verse-number">
    <a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201%3A4">4</a>
  </sup>
</span>
<span class="verse">And God saw the light, that <em>it was</em> good: and God divided the light from the darkness.</span>


<span class="verse">
  <sup class="verse-number">
    <a class="tc-tiddlylink tc-tiddlylink-resolves" href="#Genesis%201%3A5">5</a>
  </sup>
</span>
<span class="verse">And God called the light Day, and the darkness he called Night. And the evening and the morning were the first day.</span>


<p></p>

Note that my version has an HTML container for each paragraph, so I could add CSS like this:

.chapter p:nth-child(odd)  {
   background: #ffe;
}

to highlight alternating paragraphs. There’s no equivalent way to do so with this forced-line-break approach.

My original approach did something similar to yours, but generated wikitext markup… which I then fed through wikify. That would allow similar output structure, but felt like an awful way to accomplish something.

I will try the other suggestions tonight, and if they don’t speed things up enough, I may well look at this, as yours is certainly much faster.

[Springer]

@Scott_Sauyet

to highlight alternating paragraphs

OK, I understand better now. (I probably missed some earlier note about this desired effect for the paragraphs. I just saw a puzzle and hacked away at the superficial question of how to make the filters less cumbersome.) The following variation just takes paragraph hierarchy more seriously. It remains fairly nimble performance-wise — even with a small-scale unique[] operation:

<div class="chapter">
<$list filter="[<currentTiddler>tag[Chapter]tagging[]tag[Verse]get[para]unique[]nsort[]]" variable="thisPara">
<p>
<$list filter="[<currentTiddler>tag[Chapter]tagging[]tag[Verse]para<thisPara>sort[seq]]">
<span class="verse"><sup class="verse-number"><$link>{{!!verse}}</$link> </sup> {{!!text}}</span></$list>
</p>
</$list>
</div>

[Scott_Sauyet]

@Springer

@Scott_Sauyet

to highlight alternating paragraphs

OK. (I probably missed some earlier note about this desired effect for the paragraphs.

No, not at all. I simply want semantic markup with enough hooks that it’s straightforward to style. I hadn’t discussed this, and I don’t actually want such a zebra pattern here. That was just the simplest, quickest thing I could think of to demo why proper structure might be important.

@Springer

At any rate, the following variation just takes paragraph hierarchy more seriously.

Thank you very much. This is exactly the sort of thing I was hoping for. I always forget about tagging. This covered most of the slowdown. I’ll check next if the EventCatcher widget will cover much of the rest. But it’s a busy evening, so I probably won’t publish a full version right away.

But this is what seems to be working well for me right now:

\whitespace trim
<% if [<currentTiddler>tag[Chapter]] %>
<div class="chapter">
<$list filter="[<currentTiddler>tagging[]tag[Verse]get[para]unique[]nsort[]]" variable="thisPara">
  <p><$list filter="[<currentTiddler>tagging[]tag[Verse]para<thisPara>sort[seq]]">
    <span class="verse">
      <sup class="verse-number">
        <$link>{{!!verse}}</$link>
      </sup>&nbsp;<span class="text">{{!!text}}</span>
    </span>
  </$list></p>
</$list>
</div>
<% endif %>

I didn’t need the tag[Chapter] in the two filters and it was confusing me. I’m guessing it was just detritus, since removing it did no harm.

@Springer

It remains fairly nimble performance-wise — even with a small-scale unique[] operation.)

Yes, and that unique is necessary. I didn’t follow Eric’s point above. As far as I recall, there was never more than a single unique call in any version.

As always, thank you very much for your help!

[CodaCoder]

@Scott_Sauyet

I’ll check next if the EventCatcher widget will cover much of the rest

If a <$list> is producing more than a couple dozen links (or $buttons - anything that uses event wireup), I use $eventcatcher.

Every.

Time.

[Springer]

@Scott_Sauyet

I didn’t need the tag[Chapter] in the two filters and it was confusing me

Ah, that’s because I was just jumping straight in with a view template for testing, bypassing the <% if %> framing. Actually, in the big picture I’d still bypass the conditional shortcut framing around the edges and bypass the tag[Chapter] start to the list filter, using just a cascade condition (rather than $:/tags/ViewTemplate) to target the chapter tiddlers. And the book template is already applying this transcluded bit only to chapter-tiddlers.

[saqimtiaz]

@Scott_Sauyet

Thank you very much. This is exactly the sort of thing I was hoping for. I always forget about tagging. This covered most of the slowdown.

Note that replacing <currentTiddler>tagging[] with tag<currentTiddler> at the beginning of a filter run should be faster as the latter is indexed:

Whichever of <currentTiddler>tagging[] (or tag<currentTiddler>) and tag[Verse] is expected to return fewer results should be at the start of the filter run.

@Springer

Actually, in the big picture I’d still bypass the conditional shortcut framing around the edges and bypass the tag[Chapter] start to the list filter, using just a cascade condition (rather than $:/tags/ViewTemplate) to target the chapter tiddlers. And the book template is already applying this transcluded bit only to chapter-tiddlers.

Agreed.

[Springer]

@saqimtiaz

replacing <currentTiddler>tagging[] with tag<currentTiddler> at the beginning of a filter run should be faster as the latter is indexed

Oh my, I had assumed that <currentTiddler>tagging[] would somehow draw on the same tag-indexing efficiency as tag<currentTiddler>. Learning stuff here!

[saqimtiaz]

@Springer

Oh my, I had assumed that <currentTiddler>tagging[] would somehow draw on the same tag-indexing efficiency as tag<currentTiddler>. Learning stuff here!

You know what? I just looked at the code and it does. Even if our documentation on performance does not state so, the underlying code path is indeed the same! Thank you for prompting me to look at that.

[Scott_Sauyet]

@saqimtiaz

@Springer

Actually, in the big picture I’d still bypass the conditional shortcut framing around the edges and bypass the tag[Chapter] start to the list filter, using just a cascade condition (rather than $:/tags/ViewTemplate) to target the chapter tiddlers. And the book template is already applying this transcluded bit only to chapter-tiddlers.

Agreed.

Not that I necessarily object, but could one of you explain why? I don’t yet have any real intuitive understanding of the dis/advantages of the two techniques.

But if anything, I would have thought of moving in the opposite direction, adding additional ViewTemplates; these templates would appear underneath the default body template – and moving the verse content from text to a custom field. This would allow users to claim the text field for their own annotations at all levels: Verse, Chapter, and Book.

@saqimtiaz

Whichever of <currentTiddler>tagging[] (or tag<currentTiddler>) and tag[Verse] is expected to return fewer results should be at the start of the filter run.

That makes sense, and it’s definitely the former. Chapters are structured collection of subsets of the Verses.

[Scott_Sauyet]

@CodaCoder

If a <$list> is producing more than a couple dozen links (or $buttons - anything that uses event wireup), I use $eventcatcher.

I’ve done it with buttons. I don’t know why it seems harder with links. But I’m sure I’ll figure it out.

[saqimtiaz]

@Scott_Sauyet

I’ve done it with buttons. I don’t know why it seems harder with links. But I’m sure I’ll figure it out.

Something along these lines:

\procedure clickactions()
<$action-navigate $to=<<dom-data-target-tiddler>>/>
\end

<$eventcatcher selector=".mylinks" $click=<<clickactions>> >
<a class="mylinks" data-target-tiddler="this is what I want to link to">my text</a>
</$eventcatcher>