Quotes as filter parameter delimiters?

btheado · December 26, 2022, 4:34am

Filter parameters currently can be delimited by [], {}, or <>. Recently I had the idea that it would be useful to allow parameters to also be delimited by either single quotes or double quotes. The behavior of these quotes would be identical to the square braces with the only difference being that square braces are allowed inside.

This would allow for filter expressions to be written “inline” for operators like subfilter, filter, etc. It would also allow the regular expression operator to have square braces directly within.

Examples:

[subfilter"[all[]] :map[split[]first[1]]"filter'[prefix["]]']
[regexp"[0-9]{2}"]

I admit the readability isn’t very good compared to the brackets, but it would be useful functionality to have. And it is consistent with the way that filter runs can already be delimited by single quotes, double quotes, or square braces. There the readability is better because filter runs are separated by spaces.

In another thread, @saqimtiaz mentioned he experimented with backticks as delimiter:

Here the backticks are not synonyms of the square braces, so the implementation would have been more complicated than what I’m suggesting. I wonder @saqimtiaz if you already considered my idea in the past?

Also in that thread @saqimtiaz mentions that the new feature of <> delimited filter parameters allows arguments to be passed. Creative use of that feature allows passing subfilters inline just by using a specially crafted macro.

The two drawbacks of the angle bracket approach are:

Angle brackets can’t be used inside (this breaks the use case I was playing with which sparked my ideas)
It requires using textual substitution for the macro.

I’m looking for feedback on my idea, but already I feel the readability issue is a pretty big argument against it. And there may be other disadvantages I haven’t thought of.

pmario · December 26, 2022, 10:41am

You cannot use single or double quotes in filters other than they are already used. eg copy the string below to the advanced search → Filter tab

"title with spaces" [[a title with spaces]] 'title x' "[[aaa bbb]]"

Is a valid filter syntax already. So it’s already defined to be a substitution for double braces.

The regexp for the filter parser is at:

github.com

Jermolene/TiddlyWiki5/blob/9f69161709259dda51c01ce081850f6ec1dc3cb4/core/modules/filters.js#L145

    
      
          
          
/*
          Parse a filter string
          */
          exports.parseFilter = function(filterString) {
          	filterString = filterString || "";
          	var results = [], // Array of arrays of operator nodes {operator:,operand:}
          		p = 0, // Current position in the filter string
          		match;
          	var whitespaceRegExp = /(\s+)/mg,
          		operandRegExp = /((?:\+|\-|~|=|\:(\w+)(?:\:([\w\:, ]*))?)?)(?:(\[)|(?:"([^"]*)")|(?:'([^']*)')|([^\s\[\]]+))/mg;
          	while(p < filterString.length) {
          		// Skip any whitespace
          		whitespaceRegExp.lastIndex = p;
          		match = whitespaceRegExp.exec(filterString);
          		if(match && match.index === p) {
          			p = p + match[0].length;
          		}
          		// Match the start of the operation
          		if(p < filterString.length) {
          			operandRegExp.lastIndex = p;

btheado · December 26, 2022, 3:53pm

pmario:

You cannot use single or double quotes in filters other than they are already used. eg copy the string below to the advanced search → Filter tab
"title with spaces" [[a title with spaces]] 'title x' "[[aaa bbb]]"
Is a valid filter syntax already. So it’s already defined to be a substitution for double braces.

Thanks for your response. Now I’m confused because what you are pointing me to is exactly what led me to my idea in the first place .

The quotes you mention are for delimiting filter runs. IOW, the bottom two branches here:

But I’m talking about filter parameter delimiters which can only occur in the “step” box of the first branch above. I’m proposing to add two branches to the diagram below. One for single quotes and one for double quotes:
.

Since filter parameters only occur inside an outer pair of square braces, I don’t understand why quotes cannot be used there. Could you explain more?

If I’m reading the regexp correctly the open square brace is mutually exclusive with the use of quotes (exactly as implied by the first railroad diagram above). I’ve expanded the pertinent portion of the regexp here to make it easier for me to read:

(?:
(\[)
|
(?:"([^"]*)")
|
(?:'([^']*)')
|
([^\s\[\]]+)
)

As far as I can tell, this part of the code is free to make use of single and double quotes:

github.com

Jermolene/TiddlyWiki5/blob/9f69161709259dda51c01ce081850f6ec1dc3cb4/core/modules/filters.js#L71

    
      
          				});
          			});
          		}
          		// Empty operator means: title
          		else if(operator.operator === "") {
          			operator.operator = "title";
          		}
          		operator.operands = [];
          		var parseOperand = function(bracketType) {
          			var operand = {};
          			switch (bracketType) {
          				case "{": // Curly brackets
          					operand.indirect = true;
          					nextBracketPos = filterString.indexOf("}",p);
          					break;
          				case "[": // Square brackets
          					nextBracketPos = filterString.indexOf("]",p);
          					break;
          				case "<": // Angle brackets
          					operand.variable = true;
          					nextBracketPos = filterString.indexOf(">",p);

jeremyruston · January 3, 2023, 2:02pm

Hi @btheado that’s a really intriguing idea, and I think it is practical. The argument against is that the resulting filters do not read well, perhaps because we’re not used to seeing double quotes used in a way that supports nesting.

I wonder if we might achieve the same thing by allowing other Unicode bracket characters. For example, we could allow «» to be a synonym for the inner [] of filter operands. It’s not easy to type on most keyboards, but it will not be needed often, and could be copied and pasted when required.

Your example would then be:

[subfilter«[all[]] :map[split[]first[1]]»filter«[prefix["]]»]

Mohammad · January 4, 2023, 5:10am

difficult to read and difficult to write!

telumire · January 4, 2023, 5:30am

Why note use parenthesis as delimiter instead ? They are easy to types and are used in macro def so they are somewhat related to wiki-text already. And if they are needed in the string, they could be escaped with a ~~backtick~~ backslash if needs be : \(

A filter with parenthesis would look like this :

[subfilter( [all[]] :map[split[]first[1]] )filter( [prefix["]] )]

IMO it make sense, we group filters … and it’s both easy to type and to read.

Mohammad · January 4, 2023, 5:46am

I think this is the solution, when a character is not a delimiter, then escape it to be a literal string! I also prefer parenthesis.

I like

to be able to add comments to multiline filters, @saqimtiaz has used \\ for commenting inside filters
to be able to escape a char
to have tiddler of type of filter (now I use text/plain tiddler for this purpose)

jeremyruston · January 4, 2023, 10:54am

Fair point, but I think it applies to all the proposals here. I don’t think it’s a problem because we’re trying to cater for relatively rare use cases, and the context is filter strings that are already feature complex nesting rules.

Yes that makes much more sense. However, the fact that parenthesis are so easy to type and yet do not currently have a special meaning within filters is actually one of our latent opportunities (the inverse of “technical debt”). There are a lot of things we might want do with parentheses in filters: like grouping, or providing a context for local variable definitions.

While I recognise that escaping is common for programmers I believe it gives ordinary users a lot of difficulties, and so I have tried to avoid using it in TiddlyWiki. I think we already have reasonable workarounds that are fairly elegantly because they reuse existing mechanisms (ie the variety of quotes for literal attribute values, and macro definitions).

I think there’s a PR for that.

Yaisog · January 4, 2023, 2:41pm

If you need inspiration for bracket alternatives…

telumire · January 4, 2023, 3:14pm

Another idea : using double signs delimiters, for example

“>> <<” or even “<< >>” to distinguish vs the single bracket notation used in filters ?

telumire · January 4, 2023, 3:37pm

Maybe the javascript template literal syntax could be re-used here :

[subfilter${expression1}filter${expression2}]

But I guess it could cause issues with filtered transclusion ?

Springer · January 4, 2023, 4:40pm

This also makes parentheses inviting to use inside tiddler titles. I’d be wary of changing things in any way that may make us regret such titles.

We should also be aware that some unicode bracket shapes have conventional uses in other languages / use contexts. (Japanese, for example, uses 「」『』— and Japanese users of TiddlyWiki may have assumed these brackets are safe to use within tiddler titles.)

-Springer

TW_Tones · January 4, 2023, 8:48pm

I admit I am having trouble following the details in this thread but would like to point out two things especialy in the use of parentheses

apart from simple delimiters in maths and programing these are used to represent precedence and the fact they have left and right forms assists with nesting.
personaly I have not being making full use of additional quotes ´ " """ within filter “strings” and wonder if these or a slight variation can address the OT
- there are both advantages and disadvantages between delimiters that need one on each side and those with left and right forms. It may look subtly different but it can be of some consequence.

jeremyruston · January 4, 2023, 9:12pm

That might be a possibility, but it wouldn’t be entirely backwards compatible; I think [<<somemacro>>] is currently interpreted as an invocation of a macro called <somemacro>.

A minor inconvenience is that it would make it slightly harder for JavaScript programmers to use templated strings to generate filter expressions. The other consideration is that it isn’t entirely consistent with other syntaxes within TW.

Good point, that’s an important constraint. It does raise the question of how often literal tiddler titles have to be written into filter expressions – fairly frequently, filter expressions are written in terms of the current tiddler rather than a specific named tiddler.

Those additional quotes are not currently supported within filters, they can only be used to mark the start and end of widget/element attributes.

Springer · January 4, 2023, 9:47pm

Indeed! Still, while troubleshooting or tinkering with something under development, being able to enumerate tiddlers – or more often, to exclude one or two from an otherwise adequate run – is helpful!

TW_Tones · January 5, 2023, 6:32am

Another observation - the match operator allows variables but also strings as do many other operators eg;

match<varname>
match[(new)]
or even just as a title (title)

Whilst any string can be used by setting varname it is easier in many cases, to use literal strings. Any new delimiters may compromise the the number of literals we can use, making more filters demand variables be used.

I just hope we reman aware of this when making such choices.