Check if at least one tiddler exists - faster way?

#1

I have a filter that checks if at least one message exists for a given thread:

<$list filter="[field:thread_id{!!thread_id}tag[message]first[]]" variable=none>

This works but it is slow. I think it is caused by the fact that firstly a list of all messages is created which finally will be mostly thrown away via the first[] operator. (Since the number of messages it sometimes high and it is iterated over all threads it takes some time.)

So my question is: Can I check for some tiddler without creating the list first?

#2

Keep in mind this is all done in the memory of your browser, sometimes it is the order in which you filter.

Try this instead [tag[message]field:thread_id{!!thread_id}first[]] Athopugh they may be similar.

See this https://tiddlywiki.com/#Performance for a little more information.

I am sure we can optimise it further, perhaps you can share how many tiddlers have the field or tag.

#3

Currently I have 26553 tiddlers of type message distributed unevenly over 105 threads. (And there are many more tiddlers of other type.)

My use case is: I export the whole database of my Signal Messenger and convert it to TiddlyWiki. It might be that TiddlyWiki is not the best base to do something like that. It’s an experiment…

#4

There is still a lot you can do to optimise, I assisted with a 65,000 word tiddler Esperanto wiki once and performance was reasonable but only after we arranged things correctly.

  • If content does not change too much, such as historical lists you may be able to snapshot the list or tiddler so it need not iterate the list every time.
  • You may also find you own data tiddlers and other structure that suit.
    There will be other opportunities and it can depend if its single file, node or the newly proposed sqllite approach.
  • But you could offload data to another solution and get tiddlywiki to query it (like my comments on chat)
#5

Hi Dottore,
My question is, why do you need this filter at all. The list-widget has an emptyMessage, which will be shown, if the list is empty.

So you probably have a second list-widget somewhere, which then shows the list, if there is any message. So the whole work has to be done twice.

Internally the [tag[message]] filter is cached in memory. So as Tony suggested, it may be worth a try to move that filter into the front of the filter string. . eg:

In my example the I do set t-id=test, which in your example will need to be a real thread_id

\procedure t-id() test

<$list filter="[tag[message]] :filter[{!!thread_id}match<t-id>]" emptyMessage="n/a">

Hope that makes sense.

There is also an open pull request at GitHub, where I try to optimize the tag-caching … I’ll update the draft soon and post new info here. It may be an additional improvement.

-mario

#6

Thanks for all your ideas!

I want to list all threads with messages in it (omitting empty threads). By clicking on a thread I will display all the messages.

I tried it. I can’t feel a difference. As soon as the tiddler is shown TiddlyWiki becomes slow.

#7

What does the code for the entire tiddler look like? how are you getting the thread_id values?

#8

Here it goes:

<$list filter="[tag[recipient]has:field[thread_id]] +[sort[]]">
    <$list filter="[tag[message]field:thread_id{!!thread_id}first[]]" variable=none>

        <$button class="tc-btn-invisible tc-tiddlylink"><$transclude/>
            <$action-sendmessage
                $message="tm-open-window"
                $param="$:/temp/openme"
                template="ThreadWindow"
                windowTitle=`${[{!!title}]}$ Thread ${[{!!thread_id}]}$`
                width="500"
                height="580"
                windowID=`thread-window${[{!!recipient_id}]}$`
                recipient_id=`${[{!!recipient_id}]}$`/>
        </$button>
    </$list>
</$list>
#9

I would try to rewrite that code to iterate over all the message tiddlers just once. I am a bit rushed for time but here is something that hopefully conveys the idea:


<!-- save all the threads with at least one message -->
<$let threads={{{ [tag[message]get[thread_id]!is[blank]unique[]format:titlelist[]join[ ]] }}}>
	<!-- iterate over recipients filtering out those not in the threads we found -->
	<$list filter="[tag[recipient]] :filter[enlist<threads>match{!!thread_id}]">
		<!--button goes here-->
	</$list>
</$let>
#10

In addition to Saq’s suggestion for the outer $list filters, you can reduce the overhead for the $action-sendmessage by moving it into a separate macro definition, invoked via <$button ... actions=<<...>>>. This will defer evaluation of the $action-sendmessage parameters until you press the button to trigger the action. This avoids having to retrieve all those field values for $buttons that aren’t actually pressed.

Also, you can simplify the field references in the parameters by using {{!!fieldname}} syntax instead of ${[{!!fieldname}]}$. Note that this doesn’t really save all that much processing overhead, but does make the code a bit easier to read.

Thus:

\define openThreadWindow()
<$action-sendmessage $message="tm-open-window" $param="$:/temp/openme"
	template="ThreadWindow"
	windowTitle=`{{!!title}} Thread {{!!thread_id}}`
	width="500"
	height="580"
	windowID=`thread-window{{!!recipient_id}}`
	recipient_id=`{{!!recipient_id}}`/>
\end

<$let threads={{{ [tag[message]get[thread_id]format:titlelist[]join[ ]] }}}>
<$list filter="[tag[recipient]sort[]] :filter[enlist<threads>match{!!thread_id}]">
	<$button class="tc-btn-invisible tc-tiddlylink" actions=<<openThreadWindow>>>
		<$transclude/>
	</$button>
</$list>
</$let>

enjoy,
-e

1 Like
#11

I learned quite a few things in this thread. Thanks!

I tried your suggestions but it gave no speed up :neutral_face: So I activated the Performance Instrumentation in TiddlyWiki and received these numbers. My notebook has an i7-10750H.

Switching from contents to open in my Wiki’s menu with all tiddlers closed:
performance: styleRefresh: 11.00ms
performance: mainRefresh: 42.00ms

My existing code:
performance: styleRefresh: 8.00ms
performance: mainRefresh: 531.00ms

Eric’s code:
performance: styleRefresh: 11.00ms
performance: mainRefresh: 629.00ms

#12

Try adding quotes around the actions="..." parameter value. This will ensure that the macro isn’t invoked until the $button is pressed:

<$button class="tc-btn-invisible tc-tiddlylink" actions="<<openThreadWindow>>">

-e

#13

I tried. Unfortunately no change.

#14

How many recipients are there in total? That is how many buttons are created? If the number is high then using eventcatcher and button elements would be beneficial instead of the button widget.

#15

74 buttons were created.
Even if I remove the buttons completely there is no speed up.

#16

Given the number of tiddlers at play, would it be worth replacing [tag[message]...] constructs with [all[tiddlers]tag[message]...] as suggested here: https://tiddlywiki.com/#Performance?

#17

I’m happy to test all your suggestions :grinning:
But also using all[tiddlers] makes no difference.

#18

I can’t help but think there is something else going on here that is impacting performance. Unfortunately it is impossible to investigate without being hands on with the wiki, and I imagine as an export of a messenger app this has private data and isn’t something you can share.

Another avenue you could explore is making amendments to the processing during import, or running actions in the wiki the first time after import. For example, why are there threads with no messages? Could there be tiddlers created for each thread with messages?

What fields do the recipient and message tiddlers have?

#19

I can produce an export with anonymized data. The file is 9.3M (775k compressed). Shall I upload the data somewhere? What’s the best location?

Because Signal creates one thread per recipient even if there was no conversation with this recipient. The recipient can be a member of a group and all conversation has been done via this group (which is a separate recipient/thread).

Sure I could process the data much more. But then I would lose the dynamic features of TiddlyWiki. For example I would like to be able to import new messages into the Wiki without changing the state of tiddlers (i.e. threads). In the extreme the tiddlers would be completely static.

#20

Yes, you can upload it anywhere you are comfortable with. Google Drive, Dropbox or https://wetransfer.com/ are some options.