[tw5] appending to a 'list variable' through cycles

Hallo everyone, how do I append to a list of items through different cycle-runs of the code? The most intuitive way to do it would be haveing a variable that stores my list, but as far as I can see, in Tiddlywiki you cannot assign to a ‘variable’ other than at declaration time.
Once you have written <$set name=“myVar” value=“whatever”>, it’s not possible to assign to myVar, as

<$set name=“myVar” value=“whatever”>

myVar=“some other value”
</$set>

and because of that, it becomes very hard to incrementally append items to a list of items from within different cycle-runs of a <$list> widget.

The only ‘variables’ you can assign to in Tiddlywiki seem to be tiddlers’ fields and/or indexes, but this cannot be done other than interactively, by means of action-widgets like action-setfield and interactive elements like buttons, checkboxes etc.
Moreover, you have to desperately squeeze all the processing logic of filters inside widgets (<$list> or <$set> for the most part), so this chunk of code, which would be a snap in every other programming environment, is not correct and it is horribly hard to write an equivalent that is syntactically correct:

<$set name=“myList” value="">
<$list filter="[tag[dict_entry]match[keyword]]"
variable=“db_hit”>
[append<db_hit>]
</$list>

[[myList]sort[]]+[unique[]]
Entries matching keyword: <>
</$set>

As one can see, the reason why I need to store the output of every cycle-run of the <$list> inside a list variable instead of just outputting it to screen is that I might want to process my list when I’m done appending to it (removing duplicates for example) before outputting it, as in the (syntactically wrong) statement

[[myList]sort[]]+[unique[]]

Tiddlywiki is a great platform to process text, I can guess its huge potentiality, but lacking true variables (and if-then-else flow branching, and functions) is an horribly frustrating restriction, it takes me hours, if not days, to try and write code that would take me minutes in other environments to write, and most of the time I end up giving in…

Hi,

Widgets are no programming language. … The <$list widget is no loop. … It’s more like a template to create text output …

Filters produce lists so [tag[dict_entry]] will return a list of tiddler titles, that are tagged dict_entry … Tiddler titles are unique.

So [tag[dict_entry]match[keyword]] will only return 1 tiddler name: keyword, since there can’t be a second tiddler with the same name. … So this part of your “code” doesn’t make too much sense.

[tag[dict_entry]get[db_hit]match[keyword] would make sense, if db_hit is a field of a tiddler. … BUT that’s probably not what you want.

That’s why I use the tiddler text as the field that contains the keyword.
Let’s say we have 3 tiddlers


title: a
text: keyword
db_hit: aa

title: b

text: keyword
db_hit: bb

title: c
text: no keyword
db_hit: cc

So [tag[dict-entry]] :filter[get[text]match[keyword]] will return a list: a b . So we have a list of tiddlers that match the keyword.

Now we need to read the db_hit value

The following wikitext is probably all you need, as far as I can interpret your “code”.


<$set name="myList" filter="[tag[dict-entry]] :filter[get[text]match[keyword]] ">
<$set name=hitList filter="[enlist<myList>get[db_hit]addsuffix[, ]]">
Entries matching keyword: <<hitList>>
</$set></$set>

In TW filters are the powerful features. … Widgets are there to create text output with templates. Filters have match, then, else and other fancy stuff.

I’m not sure, if that’s what you really want. … You should describe your data structure first similar to what I did with my 3 example tiddlers.

From your code we don’t see where the **db_hit** value comes from.
We don’t see, where the keyword is defined

Whith that info missing it’s hard to know what you really need.

See the docs: https://tiddlywiki.com/#tag%20Operator:[[tag%20Operator]]%20[[enlist%20Operator]]%20[[get%20Operator]]%20[[Filter%20Expression]]%20[[filter%20Operator]]%20[[Filter%20Operators]]

I hope that helps.
Mario

Hi PMario, thanks a lot, indeed my code is inconsistent because in an attempt to give a generic example, not tied tho the specific issue, I tried to simplify it (and I ended up to over-simplify it).

Lots of stuff to reply to, let’s go through it point by point:

  1. indeed widgets are no programming language, that’s why I used the word ‘environment’, not ‘language’;

  2. regardless of the (poor) consistency of my pseudo-code (I should have better stressed out that it was such) the point at the core of my post is this: it is frustratingly difficult to output a number of text entries (filtered out from a db made of data tiddlers) excluding duplicates, and this because in TW you cannot assign to a ‘variable’ other than at declaration time, so you cannot incrementally append your entries to a ‘list’ variable for post-processing. Acknowledging that TW is not a full-featured programming language, I’m pretty much sure that it was conceived to enable users to do such kind of stuff: filter out and manipulate chunks of text;

  3. filters have if-then-else logic but only to a limited scope. Filters’ if-then-else statements can only output text strings;

  4. filters indeed create and deal with lists, so the only place where hopefully I can do my processing is within a filter, but it is quite difficult to write a one-liner that does what I want, let me paste my true code below (I’m sure that a better developer could write a more elegant, more synthetic one merging some of the <$list> widgets together into a single one)…

<$list filter=[enlist{!!keywords}] variable='keyword'> <!-- for EVERY keyword in a keywords list... -->
<$list filter="[tag[dict-AR]]" variable='entry'> <!-- entries are nothing but data tiddlers ... -->
<$list filter="[<entry>indexes[]]" variable='indx'>
<$set name="hit" filter="[<entry>getindex<indx>split[,]trim[ ]match<keyword>]" <!-- entries' indexes store a collection (a string) of words like word1, word2, ... -->
value="<$macrocall $name='astr' entry=<<entry>> />" <!-- prepare the data tiddler as a table row on a match test hit -->
emptyValue=""
>
<<hit>> <!-- what the code actually do is output the data tiddler immediately upon a match test hit, this DOESN'T prevent outputting duplicates -->
</$set>
</$list>
</$list>
</$list>

As you can see, there is no simple way to prevent outputting duplicates with the code above. It should be completely re-arranged to append ‘uniquely’ (greedily? I miss the exact word to say ‘without duplicates’) to the list of hits before outputting it. My dictionary is made up of data tiddler entries structured as below:

Entry1
indx11: word1a, word1b, …

indx12: word1c, word1d, …

Entry2
indx21: word2a, word2b, …

indx22: word2c, word2d, …

Thanks and regards,
CG

Hi PMario, thanks a lot, indeed my code is inconsistent because in an attempt to give a generic example, not tied tho the specific issue, I tried to simplify it (and I ended up to over-simplify it).

I thought about this, that’s why I did try to create something that could have been a possibility and shows how TW filters work. … Filters are the powerful thing in TW.

If you come from a programmers background, TW is different and needs a bit of a different thinking, because it is targeted to non-programmers …

The whole TW UI is built with filters and wikitext templates and makes heavy use of recursions especially in the core code.

Lots of stuff to reply to, let’s go through it point by point:

  1. indeed widgets are no programming language, that’s why I used the word ‘environment’, not ‘language’;

Widgets don’t have a return value … They manipulate the DOM … eg:

<ul>
<$list filter="[tag[todo]]">
<li><$text text=<<currentTiddler>> /></li>
</$list>
</ul>

From a programmers point of view, you’d probably want to “add” some variables inside the “list -loop”… But

Let’s say I have 2 tiddlers “a” and “b” tagged “todo”

[tag[todo]] returns a list and the list widget internally assigned the element to an internal variable named currentTiddler, then it iterates over the list and outputs the dom elements in the “list-body”.

The “list-body” is between <$list filter ...> list body is here </$list> which looks similar to a function, but it isn’t…

The resulting HTML code will look like this:

<ul>

<li>a</li>
<li>b</li>
</ul>

So widgets don’t calculate variables, they are used to manipulate the HTML output …

<ul>
<$list filter="[tag[todo]]">
<li><$link /></li>
</$list>
</ul>

This example uses a lot of shortcuts with TW default values. <$list filter="[tag[todo]]"> is a shortcut for: <$list filter="[tag[todo]]" variable="currentTiddler" >. <$link /> is a shortcut for <$link to=<<currentTiddler>> />

The HTML output is

<ul>
<li><a class="tc-tiddlylink tc-tiddlylink-resolves" href="#a">a</a></li>
<li><a class="tc-tiddlylink tc-tiddlylink-resolves" href="#b">b</a></li>
</ul>
  1. … it is frustratingly difficult to output a number of text entries (filtered out from a db made of data tiddlers) excluding duplicates, and this because in TW you cannot assign to a ‘variable’ other than at declaration time, so you cannot incrementally append your entries to a ‘list’ variable for post-processing.

That’s right from a programmers point of view. … But it will be very elegant, once you understand the mechanism. …

As I wrote TW uses sensible defaults for widget parameters, that’s why it is important to have a closer look at the docs. …

Acknowledging that TW is not a full-featured programming language, I’m pretty much sure that it was conceived to enable users to do such kind of stuff: filter out and manipulate chunks of text;

As I wrote it’s built to manipulate text. … As above it’s designed to manipulate the DOM … May be this view will make it clearer for you.

  1. filters have if-then-else logic but only to a limited scope. Filters’ if-then-else statements can only output text strings;

Not really, they can also be used with variables eg: [tag[todo]get<variable>] or [tag[todo]get{reference}]

With the first example if you $set variable to eg: hugo, it will read the content of the hugo field.

The second example will use the ‘text’ field of the tiddler ‘reference’, reads the content of that field and uses this value to read the value of the “todo” tiddler. … So it’s similar to a “pointer”.

  1. filters indeed create and deal with lists, so the only place where hopefully I can do my processing is within a filter, but it is quite difficult to write a one-liner that does what I want,

Yes, filters can become complex. So sometimes it is needed to define several variables upfront, that contain “filtered” elements already. … It’s also a bit of the speed advantage, because wikitext is “interpreted”, but heavily cached. …

let me paste my true code below (I’m sure that a better developer could write a more elegant, more synthetic one merging some of the <$list> widgets together into a single one)…

I’ll have a closer look at the code and write a new post …

subfilters are key! I just came across ‘filter’ operator and I’m pretty much confident it can do what I want.
So far I was able to implement this test code

<$set name=“subf” value="[getindex[IT-01]split[,]trim[]match]">

<$list filter="[tag[dict-AR]filter]" variable=“entry”>
<>

</$list>

</$set>

which behaves the way I want because the filter in the inner <$list> scans ALL the dictionary entries (tag[dict-AR]) and outputs those whose IT-01 index’s content matches the keyword dominantly appending them -that’s the word I was missing- to the output, so ridding the output of any duplicates.
Now the problem is to parameterize the code so to have it run on ALL the indexes (IT-01, IT-02, IT-03, …) of every data tiddler, and this last step, once again, is CHALLENGING.

actually, the task of parameterizing the code over a whole data tiddler’s indexes set is not challenging in itself, I already did and pasted above. What is challenging, better said: impossible! , is to complete this latter chunk of code

<$set name=“subf” value="[getindex[IT-01]split[,]trim[]match]">

<$list filter="[tag[dict-AR]filter]" variable=“entry”>
<>

</$list>

</$set>

with a (sub)filter step -possibly stored in a variable- that parameterizes it so it can work over the whole set of a dictionary entry’s indexes… Anyway the subfilter’s way is a winner, I’ll stick to it.

1 Like

Sorry for the late reply, but I’ve to do “family business”. I’ve read the last post, but I didn’t have a closer look to the data-tiddler structure yet. …

I did create a plugin which should allow you to access the value part of data-tiddlers more easily. KeyValues — advanced data-tiddler functions

You can play a bit with the demo page, while I’ll have a closer look to the rest of your second post.

-mario

1 Like

Now the problem is to parameterize the code so to have it run on ALL the indexes (IT-01, IT-02, IT-03, …) of every data tiddler, and this last step, once again, is CHALLENGING.

There is an indexes[] operator https://tiddlywiki.com/#indexes%20Operator that lists all the operators.

How should the resulting output look like?

-m

I’m not 100% sure but I think the following code may do the trick.


<$list filter=[tag[dict-AR]indexes] variable=index>

<$set name="subf" value="[getindex**<index>**split[,]trim[]match<keyword>]">
<$list filter="[tag[dict-AR]filter<subf>]" variable="entry">
<<entry>> <br/>
</$list>
</$set>
</$list>

I didn’t test the code, since I don’t have a wiki with test data.

Could you provide a minimal test case wiki, that contains some data tiddler with your structure and show the structure, how the output should look like.

I can put something together myself with the data from the 2nd post, but I can’t be sure, if it really looks like your test data.

There is 1 more question. Why did you use data-tiddlers? Is your data collected with TW, or does it come from a 3rd party app?

-mario

Dear PMario, your code

<$list filter=[tag[dict-AR]indexes] variable=index>
<$set name=“subf” value="[getindexsplit[,]trim[]match]">

<$list filter="[tag[dict-AR]filter]" variable=“entry”>
<>

</$list>
</$set>
</$list>

COULD do the trick, but:

  1. it crashes TW with an ‘internal Javascript error’ red alert. I also tried myself to write something similar, and every time I tried to use a variable inside a subfilter, like here, it made TW crash, which made me think it was syntactically wrong, but now that you also use this syntax I will try to troubleshoot the problem;

  2. since the ‘match’ test must not be done against a single keyword but against a set of keywords, in order to get rid of duplicate lines of output the ‘match’ test in the subfilter must be re-written accordingly. Lacking the flexibility of true variables, the only workaround I could come up with to avoid duplicate lines of output is to perform the test on ALL the keywords of the set for every property of every data tiddler, I mean to say that each property must be tested for matching against EVERY keyword in a single filter’s pass. If you bother reading why, you can find a thorough clarification why futher below.

Anyway, the two crucial points still to be worked out are:

a) writing a subfilter’s ‘match’ test that testes against not just one single keyword but against a SET of keywords in a single filter’s pass and
b) making TW accept variables inside subfilters without crashing

I will work on point a), while for point b) there is little I can do but maybe trying to update TW to the latest version (the one I currently use is 5.1.23) or maybe update the javascript engine of my browser…

Now for a very verbose description of how my dictionary is structured and the trick to avoid duplicate lines of output:

my dictionary is a cross-language dictionary (say for example French to English -these are not the true languages though) where each entry is a data tiddler whose properties are groups of words translating the entry’s meaning like for example:

permettre
IT-01: allow
IT-02: enable, permit
IT-03: let, license

sentir
IT-01: feel
IT-02: smell, sense, sniff, stink

toucher
IT-01: touch
IT-02: affect
IT-03: feel
IT-04: hit, receive, contact

croire
IT-01: believe, think
IT-02: suppose, imagine
IT-03: feel
IT-04: consider

As you might have noticed, for every french entry there are different groups of english words translating it, because (as it is common in many dictionaries) some translations’ meanings are more ‘nearly related’ to each other (almost interchangeably) than others and so they are grouped together in a single row. I could have gone for a different structure, having -say- just one dictionary tiddler whose indexes were french words and whose properties were groups of english words translating the indexes, but I decided to exclude this solution because it lacks the rich data structure that individual tiddlers’ fields offer and which I use to store the meta-data of each entry (grammatical category, infinitive, participles, preposition it goes with, etc.). Moreover, having the entries stored as individual tiddlers allows for the best exploitation possible of TW rich search-and-filter features, which I use to study and learn the target language with a lot of custom code. So the wiki is not only a language dictionary, but actually a platform built on top of it to enable the study of the language.

One of the customized pieces of code I care of most is this search (call it apropos if you like) for french equivalent of some english keywords (to continue with the example of the French-English dictionary), because one of the exercises I do is write a text in English and try to translate it on the fly to French, so having some code that quickly spits out one or more french equivalents for some of the english words whose translation I’m in doubt of is invaluable. It just takes for me to ‘tag’ each word with an hyperlink (hyperlink that doesn’t point to an actual tiddler, is just a convenient way of tagging words in a text for the code to easily collect them through links[] operator) and my code, fed with the exercise’s text, does all the burden of searching through the dictionary for me: super!

Say for example that I want to search for the french word(s) for ‘feel’, my code would hopefully output this:

sentir: feel; smell, sense, sniff, stink
croire: believe, think; suppose, imagine; feel; consider

In my output, since I want to render each entry in a single text row, I use a semicolon ‘;’ to join all the different groups of words (rows) together, while keeping the comma ‘,’ as a separator for words belonging to the same group.

So far so good, but when I say ‘nice format’ I also mean avoiding duplicates. Duplicates can occur when you search for more than an english word at the same time. Say that you search for words ‘feel’ and ‘smell’ together. Without a means to avoid duplicates, the code returns

sentir: feel; smell, sense, sniff, stink
sentir: feel; smell, sense, sniff, stink
croire: believe, think; suppose, imagine; feel; consider

the entry ‘sentir’ is duplicated because the search had two hits: one for english word ‘feel’ and one for ‘smell’. How do I get rid of such duplicates that dirty up my output with spurious rows? This question introduces the more general subject of how to incrementally append elements to a list and do some post-processing when the list is done, and this could be a good subject for a -badly needed IMHO- ‘coding patterns in Tiddlywiki’ book, specifically aimed at programmers.

Coming down to my more specifical issue, I was wondering that by design there are no two entries (data tiddlers) with the same name (title); moreover, most likely there aren’t two same ‘rows’ (two identical groups of related words) for the same dictionary entry or in TW jargon there aren’t two properties with the same value for the same data tiddler, so duplicates can only occur when you have two or more hits for the same ‘row’ of the same entry, because the words in that row match more than one keyword as for ‘sentir’ above matched by both keywords ‘feel’ and ‘smell’. To avoid duplicates, then, it just takes for every ‘row’ of every entry to do the ‘match’ test against ALL the search keywords in just one single filter’s pass, provided that a proper (sub)filter to do the job can be found, because in this case duplicates are automatically discarded by the very filter processing logic.
Take for example the -serialized in one row- entry ‘sentir’

sentir: feel; smell, sense, sniff, stink

if we test it against the WHOLE keywords set {‘feel’, ‘smell’} in one filter’s pass, the filter’s output will be just one single instance of the entry’s title, ‘sentir’, regardless of having had two hits while running the filter.

Hope I have made myself understandable enough, and not too much boring.

Thanks a ton for your very deeply committed help.

CG

1 Like
  1. it crashes TW with an ‘internal Javascript error’ red alert. I also tried myself to write something similar, and every time I tried to use a variable inside a subfilter, like here, it made TW crash, which made me think it was syntactically wrong, but now that you also use this syntax I will try to troubleshoot the problem;

This may be a bug in the core code.

Could you open an issue at GitHub? I’ll be at home tomorrow afternoon.
-mario

You are welcome. … It’s all much clearer now.

@ALL Everyone else who is following the thread :wink:

We can create some test tiddlers now and test our code against some test data. So it should be much easier to come up with a working solution. …

-mario

it is indeed! I updated to TW 5.2.0 and it is gone now.

There is an open PR at GitHub, … I’m not sure, if the corresponding problem could hit us in a different way. We’ll see Support macro parameters in filter run prefixes by saqimtiaz · Pull Request #6164 · Jermolene/TiddlyWiki5 · GitHub

-m

I do have a solution, based on what’s possible atm, using my keyvalues plugin, since it was developed to create output similar as you want it. … Based on what’s possible atm, there will be no duplicates …

So far so good, but when I say ‘nice format’ I also mean avoiding duplicates. Duplicates can occur when you search for more than an english word at the same time. Say that you search for words ‘feel’ and ‘smell’ together. Without a means to avoid duplicates, the code returns

sentir: feel; smell, sense, sniff, stink
sentir: feel; smell, sense, sniff, stink
croire: believe, think; suppose, imagine; feel; consider

This is a completely new search algorithm that TW doesn’t support at the moment.

If you add your example tiddlers to tiddlywiki.com and search for “feel smell” or “smell feel” in the default search input you will only get 1 result: “sentir”, since it’s the only tiddler that contains both “tokens”.

See the docs for the search operator: https://tiddlywiki.com/#search%20Operator that’s how the search works at the moment.

I also think, that the IT-xx information is needed in the output, since it contains information about the “rank” of the result. … As far as I understand your example data tiddlers.

BACKUP first – BACKUP first – BACKUP first – BACKUP first –

Extract the attachment and drag & drop import it into an empty wiki or your wiki, so you can play with it.

As I wrote it doesn’t produce the exact output as you describe it, since there are no duplicates with the default search and if IT-xx is shown.

I’d need to think about it, if it is possible to search for several tokens … or sorting by IT-xx … I think I would like to sort that way. eg: If you search for “feel” with the test data, you’ll get 3 results with my code:

Keyword: feel

croire: IT-03: feel
sentir: IT-01: feel ← I personally would like to have this at position 1 but I would need to think a bit more how I would implement it.
toucher: IT-03: feel

Hope that works for you.

mario

(Attachment test-data-plugin-carlo.zip is missing)

@Carlo,

Did you test the attached file I posted?

-mario

1 Like

Hi Mario, her I am, sorry for the late reply but I’ve been very much busy lately…
Your code works well, and I will thoroughly study it to improve my knowledge of Tiddlywiki, but it doesn’t do what I’m looking for. When I input more than one keyword, the code should return ALL the tiddlers that contain AT LEAST one of them, listing the returned tiddlers so that they are not displayed more than once.
With your code when I input, say, ‘feel’ and ‘smell’ I only get ‘sentir’ as output, i.e. I only get those entries that translate to BOTH the words I input, while instead the output should be

sentir, croire, toucher

because EACH of the above tiddlers translates to AT LEAST one of the keywords.

My code behaves according to the requirements, except for the fact that it outputs the same entry as many times as the number of keywords it translates to, so searching for ‘feel’ and ‘smell’ returns

sentir <----- one hit for ‘feel’

sentir <----- one hit for ‘smell’
croire
toucher

Thanks a lot for your kind and committed help.

Regards,
CG

PS: numbered indexes IT-xx do indeed establish a slight ranking among the translations of an entry, not so important though…

Hi,

As I wrote, TW search doesn’t provide an easy way to search for “any” search term. … That’s new … I think the desire is valid and we should support it out of the box. …

I try to implement a new “search mode: any” for the search operator, so the outer list-widget in my example should be able to return the right list of tiddlers, which will make your logic much easier.

It will need some tests and docs … I’ll create a PR once I have something that works. … We’ll see.

-mario

1 Like

Thanks a ton!

CG

Hi,

Just to be sure for future reference: https://tiddlywiki.com/#search%20Operator The new “some” parameter is available since TW v5.2.2
-mario

1 Like