Performance: Data Tiddlers - aka Dictionaries - 4000 Tiddlers with 100 Fields each

, ,
#1

Hi Folks,

I’m still new to TiddlyWiki - relative to working within the nuts & bolts - so please forgive my ignorance.

I recently read that Dictionaries are faster than regular Data Tiddlers - and this got me thinking,…

  • I have roughly 4000+ Tiddlers that i will never change - they are “Dictionary”-type (not actual languages) but qualify
  • Each Dictionary Tiddler has about 100 indexes - and the content of the indexes are strings, with : . ’ , ( ) — but no double quotes — "

I am currently getting them into my TiddlyWiki as Dictionaries - - and have a few questions:

  1. Anyone know if these are faster (search wise) than regular data tiddlers? (as per the post - can’t find it :frowning: )
  2. I am looking to see how I can easily access the “value” - so that I can just show what I am searching for – can’t seem to find anything - easily - I know it’s here somewhere - as I’ve read it,…
  3. I want to know if searching for the value in an index in a Dictionary Tiddler is “faster” in terms of TiddlyWiki performance - verses looking up via tags

Note: I avoid tags - since I have noticed that they do slow down my tiddly when I have a large amount of data displayed.

Thanks!

TwN00b

Mixed thoughts on how to improve stuff for new users
Dictionaries are FAST - How Fast? ,
Dictionaries are FAST - How Fast? ,
#2

Data stored in dictionary tiddlers (type application/x-tiddler-dictionary) or JSON data tiddlers (type application/json) uses an index. The two main filter operators of interest are indexes and getindex.

indexes is used to retrieve a list of all the indexes in a given data tiddler. So if you had a dictionary tiddler TestData with the following content:

A: 1
B: 2
C: 3
Cat: 4

the filter [[TestData]indexes[]] (used in a $list widget or a filtered transclusion) will give you A B C Cat.

getindex is used to look up the value of a specific index. So [[TestData]getindex[B]] will give you 2.

  • Like most filter operators, getindex can also be used with a variable or transcluded parameter, e.g getindex<variable> or getindex{!!field}, where <<variable>> or {{!!field}} contains the name of the index you want to look up.

You can also combine indexes and getindex, e.g.

[[TestData]indexes[]] :map[[TestData]getindex<currentTiddler>] --> 1 2 3 4
[[TestData]indexes[]prefix[C]] :map[[TestData]getindex<currentTiddler>] --> 3 4

All these techniques will work regardless of whether you’re using a dictionary tiddler or JSON data tiddler. The difference is the way in which the data is stored in the tiddler.

2 Likes
#3

Thanks @etardiff ,

This helps and I followed the examples and can see the data. And so far - much faster than using tags.

For the next part - the searching - how can I search my dictionaries for the specific text values and display the “Dictionary # - index: Text” of the matches.

Say I have:

Dictionary 1:
* index 1: This is some text
* Index 2: Another piece of information
* index 3: Yet another reference to some information
* index 4: This is not some text - Yup!

Dictionary 2:
* index 1: This is not some text
* Index 2: Another piece of information
* index 3: Yet another reference to some information
* index 4: This is not some text - No Way!

Dictionary 3:
* index 1: This is my personal some text
* Index 2: Another piece of information
* index 3: Yet another reference to some information
* index 4: This is not - Really?

Dictionary 4:
* index 1: This is your personal text
* Index 2: Another piece of information
* index 3: Yet another reference to some information
* index 4: This is not some text - again?

Search: And I want to search - for the following:

  1. <some regex case insensitive filter search?: “This is some text”
  2. <some regex case insensitive filter search?: “This !”
  3. <some regex case insensitive filter search?: “Not ?”
  4. <some regex case insensitive filter search?: " aNoThEr "

Numbers:

  1. Should show me:
  • Dictionary 1: index 1, 4
  • Dictionary 2: index 1, 4
  • Dictionary 3: index 1
  • Dictionary 4: index 4
  1. Should show me:
  • Dictionary 1: index 4
  • Dictionary 2: index 4
  • Dictionary 3: index (nothing)
  • Dictionary 4: index (nothing)
  1. Should show me:
  • Dictionary 1: index (nothing)
  • Dictionary 2: index (nothing)
  • Dictionary 3: index 4
  • Dictionary 4: index (nothing)
  1. Should show me:
  • Dictionary 1: index 2
  • Dictionary 2: index 2
  • Dictionary 3: index 2, 3
  • Dictionary 4: index 2, 4

I’ve looked on tiddlywiki’s site - not much luck on how to search.

Best guess is I need a list filter (and I will have to tag my dictionaries) and then - I get stuck.

Thanks Again,

TwN00b

#4

Yes, I would start by tagging all your dictionaries, and then you can start your filter with [tag[Dictionary]]. In fact, since you want to display which dictionary a value comes from (not just the name of the relevant index), you’ll probably want a nested list, with the outer layer returning the name of the dictionary and the inner list finding all the relevant indexes. And we’ll add one more outmost list that simply checks whether the search term meets a minimum length threshold, so your results don’t get too out of hand.

You can do this (less efficiently) with vanilla TW, but I recommend @pmario’s KeyValues plugin if you’ll be doing a lot of work with data tiddlers.

\define regex() search term goes here

<$list filter="[<regex>minlength{$:/config/Search/MinLength}]" variable="NULL">
<$list filter="[tag[Dictionary]] :filter[keyvalues:::usevalue<regex>]" variable="dictionary">
<div>
	<$link to=<<dictionary>> />: <!-- replace with <$view field="title" /> if you don't want a link -->
	<$list filter="[<dictionary>keyvalues:key::usevalue<regex>]" join=", ">{{!!title}}</$list>
</div>
</$list>
</$list>
</$list>

Here, I used the variable <<regex>> (defined via the pragma at the top) to store your search term, but you could just as easily store it in a field and replace <regex> with a {transclusion} - for example, {$:/temp/volatile/dictionary-search}. The advantage to this approach is that you can use a paired $edit-text widget to input your search without changing the underlying code of the tiddler that displays the results:

<$edit-text tiddler="$:/temp/volatile/dictionary-search" field="text" tag="input" placeholder="search with regex" />

The $:/temp/volatile adds automatic refresh throttling—very important when inputting something that will affect a long list in real time.

1 Like
#5

Thanks @etardiff - this is a lot to digest - and let me try it and get back to you - - as it looks like it will do what I want. I’m still unsure on some parts - but let me try.

Thanks Again - this is super helpful - as I spent many hours last doing google searches - to try and get the values from indexes in dictionaries - for above - and what you show here is Gold.

:exploding_head:

TwN00b

1 Like
#6

Hi @etardiff and folks,

Try as I may - I can’t get the example above to work - no idea why.

So,… I went to the plugin mentioned: KeyValues — advanced data-tiddler functions ← to try it.

No Luck :slightly_frowning_face:

I installed the plugin, saved, restarted (aka re-loaded) saw the plugin enabled.

image
image1920×842 69.6 KB

Here is a screen shot - showing it’s not working:

image
image940×1602 119 KB

!!! Here is the data tiddler - JSON Dictionary
<<list-links "[[data]]">>
<<list-links “[[data]]”>>

Content: application/json

{
"04":1,
"01":1,
"02":3,
"03":2,
"05":2
}

!!! Here is the data 1 tiddler - Data Dictionary

<<list-links "[[data 1]]">>
<<list-links “[[data 1]]”>>

Content: application/x-tiddler-dictionary

01:6
04:5
02:7
03:8
09:9
test:12
test-a:This is a tesTest
a-test:A bit more text

[tag[test-data]keyvalues[test]]

image

Any idea what I am doing wrong?

(I am starting to think I should stay a consumer,… - as I was hoping to contribute back – maybe in 3 to 5 months time – to TiddlyWiki users - a ‘neat’ idea I have,… in the spirit of TiddlyWiki - aka Free!)

TwN00b

— Update:

I just tried it with a new empty TiddlyWiki file - - - does not work.

image
image1382×1818 113 KB

image
image1180×1770 136 KB

#7

I am soooo sorry - I got it to work - apparently the examples are in-complete - I should somehow know that by now,…

Even though this is an example:

image
image1206×616 78 KB

It actually requires more - if you edit the page then follow the rabbit-hole - you discover you need to give the filter more information,…

,…

,…

,…

The bottom examples actually work - you can copy/paste them:

  • This does not have the “Try it” button - so I thought they are not valid to use as examples - - but lo-and-behold - if you copy/paste them into your tiddler - boom! They work! (Go Figure - woa, whaaaaat?)

This works (below) - even though the instructions don’t use them (Try it) button:

A bit more code to test with:

<$set name="digit-pattern" value="[0][0-9]">
<$list filter="[tag[test-data]keyvalues:key,value:--><digit-pattern>sort[]]" variable=item>
<<item>><br>
</$list>
</$set>

<$set name="digit-pattern" value="[0][0-9]">
<$list filter="[tag[test-data]keyvalues:key,value:--><digit-pattern>sum[]]" variable=item>
<<item>><br>
</$list>
</$set>

Solved!

Now - more baby steps - to see if I can get this idea to come to fruition,…

Thanks Again All!

TwN00b