Filter, meta actions, Discussion

#1

Folks it is quite common to want to alter the results of a filter, the most common may be altering the sort order, or field on which to sort, or adding the negation !.

I know there are a number of ways to do such a parametised sort, and I am not asking for those here, but I want us to discuss the idea of constructing filters before we use them. A common way would be using backticks as an attribute, when defining a variable to be used as a filter, or on a filter= parameter as an attribute value. Or even the deprecated define, or new substitute operator.

I have long thought we could as a community develop a shared “best practice” for programmatically defining and constructing filters, before their use.

Filters are so important to TiddlyWiki and there are numerous tips and tricks available out there, but when one goes beyond correcting filters, or sharing self contained filters, it is difficult to research and publish filter tricks without having to build a full and precise filters and examples.

  • For example consider the use of this in a function to turn it into a list +[format:titlelist[]join[ ]] it can just be appended to a filter but what if it was only a sub filter.

Some authors and many users would love to have written guides on how to construct filters, beyond the per operator documentation, including dial on demand filter generation to use in our “TiddlyWiki Script”. Until we have a collection of methods to handle constructing filters we cant progress this much further. This would help the community;

  • write interactive filter generations tools
  • provide more sophisticated documentation and interactive lessons.
  • Communicate more readily when supporting complex filter problems.

So to be clear this is a meta view of constructing filters, to subsequently use as filters.

What are your thoughts?

Extending Substituted attribute values to include direct transclusions
#2

I think it always depends on the usecase and the “flow” of the code.

  • I often start simple and then refine my filters with several iterations.
  • Now I mainly use functions, because they are easy to test and debug afterwards
  • I avoid it to make “monster” filters, that try to do everything in one run, because they are hard to debug

That’s basically it.

3 Likes
#3

So-called best practice achieves one thing remarkably well: prevention of innovation.

Had @jeremyruston stuck to best practice, TiddlyWiki would have never been a twinkle in his eye let alone become the software we all know and love today.

As for the sentiment you expressed, especially improved user guides re filters, sure, and of course. But I also agree with @pmario 100%; I have to understand the filter 6 months from now. If it gets too convoluted or complex, all I’ve done is confuse future-me. :confused:

#4

interesting attitude to “best practice”, I did suggest we develop one. It is more a target or a process than some fixed thing. once you get close to such a thing, changes and innovation may very well move the goal posts and what may be best practice changes, but all such things demand continuous improvement and bending and breaking such practices something I encourage with all systems and rules.

However I do think a collaborative process can help us avoid the use of bad practices, ones that make our code hard to understand, learn, maintain and change.

one thing is I suggest focusing on a neglected part of tiddlywiki and putting a collective focus on it will inform us and help guide development of filters which is becoming more complex with every release. what I ask about here would make a good foundation even just to improve how we discuss and document filters.

#5

Mario,

my approach is similar to yours, but I think we can share more of our tacit knowledge to new and experienced users a like. Key to this is how we write about, display, document and provide examples.

one observation if you look at the documentation is most filter examples use literals, and are usually very brief and still leave a lot out.

#6

I’d like to chime in on having more examples in the documentation. Especially for using variables and “transclusions” within filters. As an example, the following filter can be useful as part of a search interface for Books that use variables to pass in search parameters to the filter: [tag[Book]get[publishdate]compare:gteq:date<startDate>compare:lteq:date<endDate>]

Another similar approach is to directly inject tiddler fields into a search filter (I am not sure what is the technical name but I refer to them as transclusions) : [tag[Book]get[publishdate]compare:gteq:date{BookSearch!!startDate}compare:lteq:date{BookSearch!!endDate}]

Without examples like these in the documentation, most users may not even be aware that variables and transclusions can be used within filters.

2 Likes
#7

I think this is essential to improving our documentation. I am mostly concerned about clean and fairly
complete documentation on our filters. We discussed some interesting possibilities a few months ago, regarding the formatting of filter examples in the documentation.

But I would love to see something larger than that. I think that the railroad diagrams for the filter syntax are a fantastic tool for coders, but they are merely one part of what coders would want, and are probably gobbledy-gook to many users.

What I think is especially missing is a tutorial-style documentation about how to build filters.

I know there are people who’ve worked on these. @EricShulman? @linonetwo? @Mohammad? Others? I’m less inclined to worry about these. They may be extremely worthwhile when complete, but I’m much more interested in teaching people how such filters work than in helping them build specific ones.

I agree completely. I’m not sure about the degree of interactivity, but at the very least we should show variants of our examples, so that it’s clear that many parts do not have to be literal values, but also could be variable-references or transcluded tiddler fields. These would be well shown with radio selectors or some hover mechanism.

Could you elaborate on what you mean here?

#8

Perhaps Grok TiddlyWiki 2.0 is live! is the place for tutorial style documentation? Grok was produced more than a decade after I learned TiddlyWiki so I have tended not to use it, as I need to work though a lot of content to learn something new. perhaps @sobjornstad could accept some of these examples?

Could you elaborate on what you mean here?

Perhaps I was not clear, whilst I would like to see more documentation and better examples with filters, my current focus on filters is on handling filters in tiddlywiki before use. Such as a meta language for filters. How can one manipulate filters before they are applied. That is why I use the term meta.

I am struggling a little here with the right words and the general concept. An example may be an interactive method to provide a skeleton for a filter, with options to alter the values of operators, change the suffixes and switches in use, to be used in specific cases. I am interested in exploring and curating filters in general. Perhaps the aim is;

Smart filter templates one can modify interactively with specifics to alter the tags and other customisations then one can copy the resulting filter and add it to a filter parameter.
Guiding learners on the application within filters of; transclusions {!!fieldname} {tiddlername} {!tiddlername!!fieldname} variables , [[varname]getvariable[]], now list vars (listvar), of course literals etc… and any of these as a suffix of an Operator.
If we had a few good examples of using these there will be no need to update all the diocumentation to make these visible, perhaps just add a link to operator documentation for example “using other values in a filter”.
The idea would be a COP - developing Community of practice around “Filter ‘meta’ uses” which would ultimately Identify new and better ways to show, generate and document filters.

Once we have a more developed understanding we can “better” support the examples below;

Examples of what we may advance;

Documenting the use of safe recursion solutions via functions
The programmatic use of filter operator suffixes etc…
Using and building custom operators
Better document the use and examples of filter runs
others arising

#9

I could see a good case for creating a transparent filter-generation tool within an exhibition / demo wiki (what do we call these… community editions?).

I think it’s helpful for relative novices to see such tools within a robustly-populated and intuitive data set.

Here’s a first go at a filter-builder on the biblio site, which incorporates variables and field values selected with drop-down menus. It’s not perfect, but something like this, with explanations, might help someone tinker with the mechanics of a complex filter.

1 Like
#10

I feel we do need a number of rich data represented in tiddlers that allows a shared data set we can share and write features against, sometime ago I registered this for the purpose https://data.tiddlyhost.com/ however I did add the periodic table here https://test-data.tiddlyhost.com/ so I think the data sources would be separate from you proposed “exhibition / demo wiki” but perhaps using one or more of the publicly available tiddler sets.

[Edited] Perhaps one way to illustrate why I started this discussion is perhaps to develop a “filter algebra” and the tools to document and discuss it. Almost a language in itself the documentation of filters is largely divided into separate operators.

I am starting to think a tiddlywiki “all about filters” would be a good idea. It may even start with an image that gives a physical analogue to what a filter is. Chat GPT gave me food for thought I documented here I am particularly keen on the “The conveyor belt sorting line (warehouse / post office)”

#11

Here is a useful example about Filter meta activity. Consider $:/DefaultTiddlers
 containing Home [list[$:/StoryList]] [[Lists in text fields]] you can see here there are titles and a filter. It is simple to use a listField version of the checkbox widget, however “out of the box” there is little guidance on how to toggle the presence of the filter [list[$:/StoryList]] and other filters perhaps in tiddlers with $:/tags/Filter.

A little research on my part demonstrates manipulating filters at what is arguably a meta level. The toggling of filters in a list. Install on tiddlywiki.com to see;

Note: the below does not always work well with filters, be careful not to save tiddlywiki.com while experimenting

\procedure action-save-unchecked()
   <$action-listops $tiddler="$:/DefaultTiddlers" $field="history-list" $subfilter="+[{filter}>]"/>
\end

ALso provided $:/core/Filters/StoryList

DefaultTiddlers $:/DefaultTiddlers is an example of a title/filter list in the text field

"<$text text={{$:/DefaultTiddlers}}/>"
<hr>
"<$text text={{$:/DefaultTiddlers!!history-list}}/>"

;Titles to be used in $:/DefaultTiddlers

<$list filter="[subfilter{$:/DefaultTiddlers}]">

</$list>

<hr>

;Entries including filters in $:/DefaultTiddlers
<$list filter="[{$:/DefaultTiddlers}enlist-input[]] [{$:/DefaultTiddlers!!history-list}enlist-input[]] +[sort[]]">
   <$checkbox tiddler="$:/DefaultTiddlers" listField="text" checked=<<currentTiddler>> uncheckactions=<<action-save-unchecked>> > <$text text=`$(currentTiddler)$`/> in $:/DefaultTiddlers</$checkbox><br>
</$list>

;Add filters found in tiddlers tagged <<tag $:/tags/Filter>>
<$list filter="[all[shadows+tiddlers]tag[$:/tags/Filter]]">
   <$checkbox tiddler="$:/DefaultTiddlers" listField="text" checked={{!!filter}} uncheckactions=<<action-save-unchecked>> > <$link to=<<currentTiddler>> >{{!!description}}</$link> <span title=`Add filter found in $(currentTiddler)$ to $:/DefaultTiddlers`>{{$:/core/images/info-button}}</span> '{{!!filter}}'</$checkbox><br>
</$list>

This partial solution introduces a method to select from titles or a list of filters to add to the “filter” defined in $:/DefaultTiddlers, it demonstrates how adding/removing filters from a text or list field is fragile.

If you have a solution please advise;

Questions may be

  • what if you want to “enlist” a filter, so you can work with it.
  • What about filters with more than one filter run?