Does search:*:words have an issue or do I misunderstand?

,
#1

In my TiddlyFlex Layout here I’m using a filter expression which you can inspect if you write <<tdff.tiddlyflex-story-river-filter>> in a Tiddler.

It is called by filter="[subfilter<tdff.tiddlyflex-story-river-filter>]"

It looks like this:

[list<tv-story-list>] :filter[{$:/state/tiddlyflex/story-river/filter}match[yes]then<currentTiddler>search:*:words{$:/temp/search/input}else{$:/state/tiddlyflex/story-river/filter}!match[yes]then<currentTiddler>]

And I thought it would search in all present fields of all tiddlers in the <tv-story-list>
But now I’ve noticed that in one of my wikis it doesn’t show me some tiddlers that are shown if I omit the * (it’s configurable in the Control Panel under Settings > TiddlyFlex > Story-River Filter Fields-Suffix)

I had a brief look at the search filter operator and the wiki’s search method but don’t yet grasp it
The documentations states that it “causes the search to be performed across all fields available on each tiddler” - but is that so?

Now first of all I’m doubting the correctness of my constructs and then I think about bugs in the core

That’s why I need to dig deeper into my filters…

Writing this I realize that it’s this filter (<<tdff.tiddlyflex-enlist-columns>>) that is responsible for filtering the columns out of the way (which is the case in my issue):

[list[$:/columns]] :map[{$:/state/tiddlyflex/story-river/filter}match[yes]then<currentTiddler>addprefix[$:/StoryList-]get[list]enlist-input[]search:*:words{$:/temp/search/input}limit[1]then<currentTiddler>else{$:/state/tiddlyflex/story-river/filter}!match[yes]then<currentTiddler>] +[!match[]] ~1

I need to explain it a little bit… It should list all columns (they are numbers in the order 1 2 3 …)
then if the filtering is enabled then map the column as suffix to $:/StoryList-, get that tiddlers list field and enlist it
then search in the enlisted tiddlers for the search term and if there is one (limit[1]) then return the column number (which is <currentTiddler> in this :map[] filter-run)

The issue in my case is, that it filters out column number 1 of 2 columns, where the search term is in a tiddler title in column 1 and in two tiddler’s text in column 2

I don’t know if anyone here has got the muse to troubleshoot my filters, but if then it’s this second filter I’ve posted that is the culprit (I think)

Then maybe the issue is how I construct the filters which is a bit crazy but always worked:

\function tdff.tiddlyflex-enlist-columns() [[list]addsuffix<tiddlyFlexPrefix>addsuffix[$:/columns]addsuffix<tiddlyFlexSuffix>addsuffix<tiddlyFlexSuffix>addsuffix[ :map]addsuffix<tiddlyFlexPrefix>addsuffix[{$:/state/tiddlyflex/story-river/filter}match]addsuffix<tiddlyFlexPrefix>addsuffix[yes]addsuffix<tiddlyFlexSuffix>addsuffix[then<currentTiddler>addprefix]addsuffix<tiddlyFlexPrefix>addsuffix[$:/StoryList-]addsuffix<tiddlyFlexSuffix>addsuffix[get]addsuffix<tiddlyFlexPrefix>addsuffix[list]addsuffix<tiddlyFlexSuffix>addsuffix[enlist-input]addsuffix<tiddlyFlexPrefix>addsuffix<tiddlyFlexSuffix>addsuffix[search:]addsuffix{$:/config/tiddlyflex/story-river/filter/fields}addsuffix[:]addsuffix{$:/config/tiddlyflex/story-river/filter/flags}addsuffix[{$:/temp/search/input}]addsuffix[limit]addsuffix<tiddlyFlexPrefix>addsuffix[1]addsuffix<tiddlyFlexSuffix>addsuffix[then<currentTiddler>else{$:/state/tiddlyflex/story-river/filter}!match]addsuffix<tiddlyFlexPrefix>addsuffix[yes]addsuffix<tiddlyFlexSuffix>addsuffix[then<currentTiddler>]addsuffix<tiddlyFlexSuffix>addsuffix[ +]addsuffix<tiddlyFlexPrefix>addsuffix[!match]addsuffix<tiddlyFlexPrefix>addsuffix<tiddlyFlexSuffix>addsuffix<tiddlyFlexSuffix>] +[addprefix<tiddlyFlexPrefix>] +[addsuffix[ ~1]]

But I better post the link to the GitHub page with the filters:

I’m constructing the filters like this because it’s difficult to explain but I needed them like that. I’ll try to explain later, maybe somebody has a better idea how to solve the issue I’m trying to workaround using these constructions.

Anyhow, if you want to give it a look, please let me know your findings!

Thank you in advance,
Simon

#2

Now I’ll explain why I use these difficult to read constructs

These are something like filters that create filters based on conditions
If this condition meets, the filter looks like this, otherwise it looks like that

But why I really need them is because it’s impossible to write search:{$:/config/fields}:{$:/config/flags}{$:/temp/search/input} or search:{{$:/config/fields}}:{{$:/config/flags}}{{$:/temp/search/input}} or search:{{$:/config/fields}}:{{$:/config/flags}}{$:/temp/search/input}

And in TiddlyFlex I reuse these filters all over the place and it’s more convenient and in some filters otherwise not possible than to write subfilter<tdff.tiddlyflex-enlist-columns>

#3

Would you consider rewriting filters like tdff.tiddlyflex-enlist-columns sequentially with +[join[]] rather than that long string of addsuffix/addprefix? I’m not sure it would solve your issue, but it would make it much easier to read and debug!

#4

You can use Substituted Attribute Values (the “backtick” attribute syntax).

Give this a try

fields:<br>
<$select tiddler="$:/config/fields" multiple>
   <option value="*">* = all fields</option>
   <$list filter="[fields[]sort[]]"><option><<currentTiddler>></option></$list>
</$select><br>
flags:<br>
<$select tiddler="$:/config/flags" multiple>
   <$list filter="literal whitespace regexp words some casesensitive anchored">
      <option><<currentTiddler>></option>
   </$list>
</$select><br>
search:<br>
<$edit-text tag=input tiddler="$:/temp/search/input"/>

<$let fields={{{ [enlist{$:/config/fields}join[,]] }}}>
<$let flags={{{ [enlist{$:/config/flags}join[,]] }}}>
Filter = <$text text=`[search:$(fields)$:$(flags)${$:/temp/search/input}]`/>
<p/>
<$list filter=`[search:$(fields)$:$(flags)${$:/temp/search/input}]`><$link/><br></$list>
</$let>
</$let>

edit: field and flags can have multiple values that need to be converted from space-separated lists to comma-separated lists in the filter syntax.

enjoy,
-e

4 Likes
#5

Awesome @EricShulman , thanks!
I haven’t yet understood this syntax and will need time to adopt it correctly.

How would you solve the following filter:

[subfilter<tdff.tiddlyflex-enlist-columns>] +[count[]!match[1]] :then[subfilter<tdff.tiddlyflex-current-column-filtered-before>] ~[subfilter<tdff.tiddlyflex-enlist-columns>] ~1

where tdff.enlist-columns looks like shown above and tdff.tiddlyflex-current-column-filtered-before looks like this:

[subfilter<tdff.tiddlyflex-enlist-columns>] [{$:/columns!!current-column}] +[unique[]] +[nsort[]] +[before{$:/columns!!current-column}]

I couldn’t put the filters into one massive filter string because

  • it gets too complex
  • I need to reuse subfilter for different purposes in different filters
  • the nesting level I’m using seemed impossible to accomplish to me without the use of “constructed” filters

Thank you very much!

#6

Hi @etardiff

Yes I know it’s difficult to debug…
I had some +[join[]] there in the beginning but then I couldn’t debug it easily and for me finally it was easier to do it this way … one just has to get used to it

  • addsuffix does concatenate or adds the opening [ of a filter operator
  • addprefix mostly just adds the closing ] of a filter operator

But if there were easier ways to accomplish these tasks I’d be happy to rewrite my filters

Maybe addsuffix<tiddlyFlexPrefix> can be shortened to <tdff.prefix> and addsuffix<tiddlyFlexSuffix> to <tdff.suffix> (both functions) - I didn’t think about that before

#7

Ah! Now I found an answer to the original Issue:

Why do tiddlers disappear when filtered by search:*:words[searchterm] even if they contain the search term in the title?

Those tiddlers have the field _canonical_uri and type application/pdf
But it’s sufficient that they have the application/pdf type to disappear from the search results

Try in @EricShulman’s example on https://tiddlywiki.com and create a tiddler that you can find in the search results list
Set fields to *
Set flags to words
Search for that tiddler
Now set its type to application/pdf
Notice how it’s not in the search results anymore

#8

This appears to be a bug
I’ve created a pull request: exports.search: don't break the loop if encoding is base64 by BurningTreeC · Pull Request #9247 · TiddlyWiki/TiddlyWiki5 · GitHub

1 Like