Boolean logic and TiddlyWiki filters, documentation draft

[Springer]

We gets lots of questions from beginners about “how do I get a filter to retrieve this OR that?” “How do I get a filter to get me this set of tiddlers but not this other?” (One recent example )

Try dropping this on tiddlywiki.com and please let me know what you think:

Boolean logic (“and” “or” etc.) in TiddlyWiki (oper-ex).json (4.0 KB)

[UPDATE: edited to employ .example macrocall with “Try it” buttons, a few other fixes]

It’s frustrating that beginners //cannot// search for “or” at tw-com (search string is too short), and short words like “and” and “not” yield a forest of results. I’d like to add a tiddler whose title — for new users confused with a version of this FAQ — will look really salient among the search results (better search result prioritizing is a separate but important theme). For that reason, I’ve put “and” and “or” (with smart quotes) in the title, though I realize that there could be reasons not to do so.

Anyway, I’ve drafted this overview. I post it here for comments only because it’s much more complex than other documentation PRs, and the key question is not about accuracy, but translation — that is, trying to put ourselves in the heads of beginners and offering a clear mini-tutorial to get them thinking in filter syntax without sending them to the railroad diagram tiddlers (which are beautiful, but not what they need).

The tricky thing for many of us long-term folks is that we easily forget what it’s like to search for such documentation without the right words already in hand, and once filter syntax becomes more obvious to us, we no longer find the documentation gap to be frustrating. What say you all?

[Scott_Sauyet]

I find this overview excellent! It’s just the level of documentation that I’ve been looking for.

The only minor caveat is that house style usually has “Try It” buttons to see the results in real-time. I actually prefer the style presented here, but I don’t know if we really want to mix and match.

---

Edit: And immediately looking again, I see a good reason for the “Try It” buttons. Our documentation then doesn’t have to keep pace with every update to the content. If we add another “About” tiddler, this style would fall behind, but “Try It” would not.

[etardiff]

Great idea, @Springer—and I really appreciate the thought given to searchability!

My very first impressions (as someone who doesn’t really need this explanation) are…

1. I do think it’s a good idea to include “Boolean” in the text and title, since people who know the proper terminology will probably search for it. It may be helpful to include an external link to a simple definition/overview (perhaps on Wikipedia?) for those who aren’t familiar with the term.
2. I took an undergrad semantics course, so I’m familiar with set theory, but I suspect this may be less common outside the fields of linguistics/philosophy. I’m a little concerned that your first paragraph—while concise and helpful to those who do know the terms—may be an intimidating introduction for those who are less familiar, based purely on the density of new terms.

Of course, it’s very possible that such users will simply skip down to the clear and helpful headings, so I may be overthinking this.

Otherwise, I concur with Scott; it’s a great overview!

[Springer]

@Scott_Sauyet

house style usually has “Try It” buttons to see the results in real-time

Actually, I’m interested in converting everything to that example syntax, largely because it will conserve space and maximize scannability. It’s not really necessary to see the filter results, except on a “Try it” basis.

[Springer]

@Scott_Sauyet

a good reason for the “Try It” buttons.

The json has been updated to use the .example macrocall mechanism. I’ve made a few other fixes, including a bit more focus on how negation works in filters.

[Springer]

@etardiff

I’m a little concerned that your first paragraph—while concise and helpful to those who do know the terms—may be an intimidating introduction for those who are less familiar, based purely on the density of new terms.

I’ve made an update (replaced the json above) to demote the technical-sounding stuff a bit. I still want to include those words, though, to maximize the odds that someone who searches for “disjunction” or “union” (etc.) will come across this explanation.

[Scott_Sauyet]

@Springer

I’ve made an update (replaced the json above) to demote the technical-sounding stuff a bit

I liked it before. But this is better.

@Springer

I still want to include those words, though, to maximize the odds that someone who searches for “disjunction” or “union” (etc.) will come across this explanation.

Agreed. The mathematician in me will look for union/intersection. The programmer in me might try boolean, and I’m sure there are others who would reach for con/disjunction. Having them all there for findability is the best of all worlds.

[Springer]

I now realize that — as of v5.1.23 — I really should not say that TiddlyWiki does not use these boolean operators. They can indeed be spelled out as filter run prefixes.

I’ve made an alternate (somewhat longer) version of the tiddler that explicitly notes that you can spell out filter run prefixes such as :and and :or. I think if I understood that, as a beginner, I might have typed them out, as training wheels, before turning to the more efficient-looking (but less explicit) variants that are standard practice in the community.

So… see below for again-updated version.

The question is: what’s most useful for beginners? There may not be any one simple answer, since people have different background habits and associations. I still hope to contribute at least one methodical (and not-too-overwhelming) documentation tiddler we can send folks to, when they come here with “How do I get [insert boolean expression here]?”

[Scott_Sauyet]

@Springer

The question is: what’s most useful for beginners?

I’d say that being thorough and clear easily trumps being concise.

Also, those extra examples with their “Try It” buttons don’t add too much conceptual weight.

[pmario]

Operators do have its own example macro

<<.operator-example 1 "[all[shadows]]">>

[Springer]

@pmario

<<.operator-example 1 “[all[shadows]]”>>

Oh! Indeed. And it’s more visually clear that way (filter contents in red, less noise with the triple-braces).

Replacing again.

Boolean logic (“and” “or” etc.) in TiddlyWiki (oper-ex).json (4.0 KB)

[TwN00b]

Hi @Springer - wow - I am a n00b and this helps - i still don’t understand a part of it - but I can now see why there is no “And” or “Or” or “Not” in TiddlyWiki - Mystery Solved!

What would help me and people like me - is to have the “Examples” in the doc - - - I am for many, many, many, many, many,… (did I mention many?) examples.

Otherwise - it’s back to Google, Reddit - etc - looking for examples on the examples in TiddlyWiki.

---

Here is what I typically do (recently) for looking for something: (like the dictionary question I posted today)

I come to TiddlyWiki - click on the side search and type in:

• Filter example dictionary

I get this: (image)

• I skip Release 5.1.23 - unsure why it shows up - but thats ok
• I don’t want to learn the json data one - so I skip it - looks minimal anyways and not much details
• indexes Operator - - ok well - I guess that’s ok - there is no mention of a dictionary - unsure why it shows up - fuzzy logic - - - hey, look - a cat -

I look at the indexes operator link - - yup it’s not a dictionary - so - I know to “skip” this in my web search:

So - I copy my search and go to Google - and add in my search - getting rid of the stuff that did not work:

Then - I start working through the list,…

If we had more examples in the doc - in TiddlyWiki itself - it would be awesome.

• I am still puzzled why there are not many examples in TiddlyWiki on the main site - considering what it’s designed for - maybe it’s the “tag” limitations?

Anyways, Thanks Again for your examples - now I understand in filter searches:

• A “space” means “OR”
• And “No Space” means “AND”

Nice!

TwN00b

[Springer]

@TwN00b

What would help me and people like me - is to have the “Examples” in the doc

Well, exactly the thing that my suggested tiddler does is to provide examples for these basic concepts (and, or, negation, and some nested combinations of these). I agree that examples are helpful. Part of our challenge is to prioritize which terms and concepts need examples.

@TwN00b

• A “space” means “OR”
• And “No Space” means “AND”

This is not quite accurate. Two filter runs separated by a space does summarize the “or” pattern (tiddlers that meet this condition OR that one): [tag[About]] [tag[HelloThere]]

But the “and” pattern differs by more than a lack of space. Notice the initial filter run (which would be [tag[About]] by itself), is not fully closed — the last bracket has not been added, so it’s like [tag[About] — and the second operator goes to work on that existing “open” filter run in progress: [tag[About]tag[HelloThere]]

[TwN00b]

Thanks @Springer ,

Ooooh - - I had it wrong - light-bulb turning on - - more like Glow-Plugs,…

ok - so I just got lucky in my tiddlers - - as I placed spaces there - to make them easier to read - and they just so happened to be “OR” searches.

This also explains why my “AND” ones never worked the way I wanted - so I had to think about how to make them work as "OR"s,… - - most sort-of work - but now I need to go back and re-do them - as I get this.

So - if I understand this:

1. I can [tag[MyLargeList]tag[MySmallList]tag[MySuperSmallList]}
   ** BUT - it’s better if I do this instead:
2. [tag[MySuperSmallList]tag[MySmallList]tag[MyLargeList]]

I tested and both return the same thing - only item 1 - takes about 6 seconds and item 2 is almost instant - wow! They are like Linux pipes |'s – cat IstiddlyWikiCool.txt | head -1 (Output: “Yes!”) - I know I can just head -1 the file,… (sh is easy compared to TiddlyWiki - still)

Next - to do the “OR” - I can also search smaller lists to larger lists - for speed efficiency,… - nice!

Is there a way we can add the information to the examples section - so that others like me will find it?

Also, how’s about this - can you add a “link” in the TiddlyWiki doc’s - - that point folks over here - to the various sections - - as this will save them from going to Google,…

Thanks Again!

TwN00b

[pmario]

@TwN00b

I tested and both return the same thing - only item 1 - takes about 6 seconds and item 2 is almost instant - wow! They are like Linux pipes |'s – cat IstiddlyWikiCool.txt | head -1 (Output: “Yes!”) - I know I can just head -1 the file,… (sh is easy compared to TiddlyWiki - still)

That’s true. For filters you should always try to start with a “low number of resulting” tiddlers if that is possible.

That’s why it also makes sense to “store” filter results in a variable, if you need to reuse them further doewn in your code. eg:

<$let tempSomething={{{ [tag[HelloThere]format:titlelist[]join[ ]] }}}>
   <$list filter="[<tempSomething>enlist-input[]]" variable="item">
       <<item>><br>
   </$list>
</$let>

Instead of [tag[HelloThere] you can use your own filter string, that reduces a large list to a much smaller one.

Since variable={{{ [tag[HelloThere]] }}} only sets the first item to the variable we need the format:titlelist operator and then join[ ].

format:titlelist is needed because tiddler title can have spaces

have fun!

[TwN00b]

Thanks Guys - This is awesome!

Next on my list to tackle - how to search through the many dictionaries I am creating - for words in the text,… fun, fun, fun!

[Scott_Sauyet]

@pmario

That’s why it also makes sense to “store” filter results in a variable, if you need to reuse them further doewn in your code. eg:

<$let tempSomething={{{ [tag[HelloThere]format:titlelist[]join[ ]] }}}>

Thank you! That points me to a simple way to handle lists in variables, something that has bothered me for a while:

\function as.list() [format:titlelist[]join[ ]]