Thinking about visual filter builder

[linonetwo]

I still remember the feeling back to 2018 when I trying to understand list widget + filter expression, I don’t know how many bracket [ should it be in a filter expression. (I use analogy of [img[xxx]] later to remember it, and get it finally)

So to help more new user understand and quick get to use filter expression, I’d like to make a visual filter builder.

The design of filter expression DSL is very good, make it much easier to build low code builder tools.

I’m now considering using Blockly  |  Google for Developers (aka. scratch ), mainly because my new game project will use it for scripting.
And in the future version of WYSIWYG editor, after feat: serialize AST node back to wikitext string by linonetwo · Pull Request #8258 · TiddlyWiki/TiddlyWiki5 · GitHub and [IDEA] Add JSDoc to AST · Issue #8532 · TiddlyWiki/TiddlyWiki5 · GitHub , I will make widget to be a form, where you can see “filter” field as a friendly slot, that can put in a blockly builder.

[Mohammad]

That would be great, especially in the form of Blocky!
By the way have you tried the filter maker by @ericshulman?
Open TiddlyTools for TW v5.3.5 — Small Tools for Big Ideas!™
Click on Packages
Then Select Search tab
And then click on TiddlyTools/Search/Filters

Then note the predefined filter, or open the filter maker.

[twMat]

@linonetwo

I’m now considering using Blockly | Google for Developers (aka. scratch ), mainly because my new game project will use it for scripting.

I think visual building like in Scratch is a wonderful idea, if possible. Although, it would make more sense if it was (eventually) created in wikitext.

You may also be interested to check out @saqimtiaz idea for a TiddlyWiki filter visualizer/helper tool.

[linonetwo]

Try out my quick demo on 未命名插件 — 未填描述

I will refactor the blockly part, to allow also build widget fields.

[pmario]

@linonetwo

I will refactor the blockly part, to allow also build widget fields.

A Filter-Expression does not have any braces. It is the Filter-Run where the outer braces start.

The Filter-Parameter is responsible for the “inner” braces.

If you import my Trails-plugin into tiddlywiki.com you will get a nice “hierarchical” navigation and a trail, that let’s you immediately see, where you are in the filter hierarchy.

BC & Trails

[linonetwo]

It now looks like this

I’m not sure if this is the best & most beautiful UI, but it add more restrain on what you can do, so is easier to learn.

I need help on [IDEA] Inline document for filter operator and action widget · Issue #8154 · TiddlyWiki/TiddlyWiki5 · GitHub so I can auto generate all blocks from the doc. You can give some advice or encourge on my RFC there.

[pmario]

Your blue elements are "Run"s – IMO you should keep that description in between the eg:

Run [ ... ] … default run

:sort Run [ ... ] … :sort prefix Run

You should avoid to use ALL UPPERCASE for your elements except TW supports it. Changing case between the visual editor and the real filter “code” will cause a lot of confusion.

Also you need to use :sort instead of sort since the difference here will also create confusion.

In your example SORT and CURRENT are written in all-uppercase letters, which means they belong to “the same category”, which is absolutely not the case.

The other problem is, that ALLUPPERCASE is difficult to read for humans.

I’m OK if the filter operators are below each other, but the “Run Block” should look more like this. Where the closing braces are on the right.

“All” does not need it’s own block. all is a standard operator

just my thoughts.

[Scott_Sauyet]

@twMat

You may also be interested to check out @saqimtiaz idea for a TiddlyWiki filter visualizer/helper tool

My new job has been relatively free of boring meetings, which is great, but I had two hours of them today. In those, I can generally do something slightly creative, as long as it doesn’t take too much brainpower, and I thought I’d take a stab at following up on those ideas with a somewhat improved UI.

You can test it by downloading the following and dragging it to tiddlywiki.com:

FilterBreakdown.json (11.2 KB)

(You can also drag it elsewhere, but fewer links will work.)

I did two versions. One has the various parts of the filter run in a list:

The other has them nested in a tree:

This is all hard-coded, of course. I think Saq was experimenting with generating this from a Syntax Tree. If that is working, and if the column numbers are available for the nodes (which seemed to be in question), then we could probably automate some variant of this. The difficult part would be to get short descriptions of every single operator, and to figure out a reasonable way to handle custom functions used as operators.

But the other possibility would be to write a documentation mechanism that allowed you to somewhat easily create such “diagrams”. It wouldn’t be useful for describing your own filters, nor is it related to a filter builder. But it could be a good teaching tool.

[twMat]

Neat! That is of course not about building a filter, but instead about explaining a given filter or about filter structure in general. I can see how that would be useful for the docs.

@Scott_Sauyet

The difficult part would be to get short descriptions of every single operator, and to figure out a reasonable way to handle custom functions used as operators.

No problem; again referring to how your thing may be useful in the docs; there area already short descriptions for every single operator that are easily accessible.

[twMat]

As for visually building filters, I think a key aspect would be to be able to see the “status” for each filter step, i.e what titles are remaining thus far. This should be effective for finding errors.

Maybe the blocks could, by default, show the count thus far. And clicking the block shows a popup with the corresponding titles at that stage.

I think this would be very useful.

[linonetwo]

Updated.

] can’t move to the end, limintation on Blockly.

I also add a “Widget” tab for demo, this will be move to another plugin later. And will be used in WYSIWYG editor to handle editing of widget & macro call. @jeremyruston talked about his vision on editing widget in WYSIWYG editor in the live, I’m not sure I get them all. But this is what I’m doing.

The benefit of blockly here is it have type check, each block have limintation like nextStatement: 'filter_run',, so user won’t need to trying many combination of a-z 26 character, only need to try few combinations of different colors of blocks. The “limitation” is a source of affordance.

[linonetwo]

That might be possible, but hard to implement, because blocks in the editor are not real filter operators, they are just some string, but of course I know how to do it, I just think I will delay it until I put visual editor into Wikitext WYSIWYG editor.

[linonetwo]

Yes, that can be a good resource for learning filter operator, while it is not a builder.

I think example like this is better than abstract representation of rail road diagram. I hope you could try contribute it to the core tw5-com.

[CodaCoder]

Looking good apart from…

[Ste_W]

I have too little TWfoo to provide any insight on scratch/ blocky (looks really good!) but, just a thought, would a flow chart give a good visual handle on filters. I only ask as at my work the students use a program called flowcode to set out PLC programs and don’t have to touch and C code, which is generated automatically.

[wattahay]

I don’t know if that visual tool could do some form of the following:

• understanding that there are effectively results at every single “step” of the filter from left to right that get narrowed down or changed. (I guess it would show results as you built it.)

It’s easy to take it for granted: At any point, if you cut the filter off, then there is something that comes out, albeit in an un-finalized form. (obviously, assuming it is cut off syntactically.)

Especially from a never-programmer’s perspective, (kind-of who this would be for) that is the initial visualization that does not happen naturally. Data is being passed like a ball, invisibly until the end of the filter.

Additionally, understanding where in the filter that data becomes transient/deliverable between operators and stuff is pretty informative of how they function.

[linonetwo]

It has already suggested above, but no one have time for this, especially when there are more important tasks like [IDEA] Inline document for filter operator and action widget · Issue #8154 · TiddlyWiki/TiddlyWiki5 · GitHub

[linonetwo]

A qq group menber ask why <$action-navigate $to={{{ [subfilter[{$:/config/RandomSelection/Filter}]random[1]] }}} /> is wrong. Where it is [{ that wrong.

@linonetwo

back to 2018 when I trying to understand list widget + filter expression, I don’t know how many bracket [ should it be in a filter expression

So how many bracket should add is really a hard problem for new users. Even we old users think it is obvious.

[pmario]

Is there a question related to your post?

[linonetwo]

In OP I said few years ago I have this same question too, and I believe visual builder will eliminate this kind of question.

By posting this example I just want to show it is not only me, every new user may have encounter it once. (This kind of question can consume hours of time to solve by oneself, because error message is not pointing out the key part.)