Storing filters without evaluation? Bug or bugbear?

Developers
#1

With out expanding on the reasons I want to “Store filters without evaluation?” I would consider it reasonable to be able to store filter strings, as they are for later use. Perhaps to use them in documentation, then later in an example piece of code.

Consider the following;

\define canned-filter() [<transclusion>]
\procedure canned-filter2() [<transclusion>]
\procedure display-string(string) <<string>>

# <<canned-filter>>
# <<canned-filter2>>
# {{somefilter}}
# <$text text=<<canned-filter>>/>
# <<display-string "[<transclusion>]">>
# <$macrocall $name=display-string string=`$(canned-filter)$`>>

But this is the result I get;

Snag_1b4a2d5d

I want to see [<transclusion>] that I could pass as a “literal” parameter to a widget or function etc…

The html preview shows

<ol>
<li>[<transclusion>]</transclusion></li>
<li>[<transclusion>]</transclusion></li>
<li>[<transclusion>]</transclusion></li>
<li>[&lt;transclusion&gt;]</li> <!-- note the HTML encoding -->
<li>[<transclusion>]</transclusion></li>
<li>[<transclusion>]</transclusion></li>
</ol>
  • Line breaks added for =ease of reading
  • You can see here the parser is autoclosing this as a html tag.

I have tried a number of other ways but as you can see we can only “view” the real value of a filter stored in a variable by using the text widget.

Basically it seems that the parser refuses to leave the <varname> alone and treats it as html, even although it is inside a filter?

It is understandable because we may want html to be stored inside a variable or tiddler to be parsed correctly.

What could we do about this?

  • Are there any work arounds I could use?
  • Is this a bug or unintended consequence?
  • Can we get a fix or a way to annotate this?
  • perhaps a new widget/pragma or variation of the set widget would allow us to set a variable to a string without first attempting to apply HTML rules to it?
#2

Fields? I store filters in fields. Like, always.

But I think what you’ve shown is pretty broken. Even wrapping with """ doesn’t seem to help.

#3

Your could stop tagging your canned filter definitions with $:/tags/Global and use the import pragma instead.

IMO transcluding filter expressions like you suggested in the OP does not make too much sense, since \import exists

#4

If you dont see the need I can spell it out, but I think it compelling never the less.

#5

There has been the possibility to use a $:/global/ namespace in the “Parameterised Transclusions” info thread at GH.

It was removed because of performance problems and it never came back.

#6

We can somewhat manufacture one ourselves if desired, how did you think it would relate to this?

I would have imagined [<transclusion>] in wiki text would appear as is, it is neither a valid tiddler title, it’s not how you use filters, yet it resembles one.

  • I can see the parser may see it as an opening html tag, but would have thought the parser would consider it was delimited by [< >] and not a html attribute tag.

Following your link I saw my own prior contribution

Personaly I have long felt it would help if we made available in wikitext to be able to convert between keyword=“value” pairs and variables with values and the reverse, generate a set of keyword value pairs and handling the delimiters ’ " “”".

An example may be programatically building a set of keyword value pairs and passing them to actioncreatetiddler or taking a set of variables and save them as settings in a tiddler field or text.

  • This has in part being satisfied with the parameter widget $params attribute, it makes a small JSON array
  • I was in the process of populating such an array with some keyword value pairs, but where the value is a canned filter string when I came across this unexpected behaviour [<transclusion>] only returns [] so I was unable to set keyword to a filter as the value. [<currentTiddler>] also has the same result as it does here []
#7

I did have a closer look at your OP again. – I have to say, that I did misunderstand the first 3 lines.

They are the macro definitions to be shown by the numbered list.

I thought the \define canned-filter() [<transclusion>] should “import” the definition from a variable named “transclusion”. Which causes a performance problem (and my confusion)

So if the usecase is exclusive for documentation and debugging imo it does make sense to have a better version of the dumpvariables macro

At the moment “dumpvariables” a big mess and it does not show the definition. It only shows the “body”. IMO It should be possible to rebuild the “macro signature” from the parse-tree and display it together with the “body” for documentation purpose.

dumpvariables should also have a filter similar to your custom-dumpvariables macro using the $reveal-widget instead of the HTML DETAILS element.

#8

@pmario I think it’s possible @TW_Tones’ use of the transclusion variable may have thrown you off – it did throw me off for a moment. Then I switched to [<currentTiddler>] and saw the same wacky HTML production (Tony pointed out the same in a subsequent post).

The confounding thing is, the parse and widget trees look fine but HTML raw is :exploding_head:

#9

Hi @TW_Tones all that’s happening here is that most of the ways that you’ve tried to view the filter expression do so by interpreting it as wikitext or evaluating it instead of displaying it.

The filter syntax and the wikitext syntax are different – the wikitext syntax doesn’t support writing “freestanding” filters; they have to be used in context: as an attribute value, or within the {{{}}} syntax.

The best way to display a raw filter from a variable is to use the <$text text=<<variable>>/> syntax. I agree that the syntax is clumsy, and we have considered adding a shortcut, but we haven’t decided on a syntax.

#10

@jeremyruston my examples shows how it cant be used, I did this to illustrate the issue. I can also come up with a number of other ways it can’t be used.

That is not the point. How can I store such filters as text, for later use without an evaluation?

  • I do not want a freestanding filter, only the ability to represent a filters full syntax as plain text when desired and in a parameter.
  • We document a range of syntaxes such as << >> [[ ]] {{ }} and {{{ }}} to illistrate a few.
  • The following is certianly not one of these [..<anyame>..] yet in wikitext the parser;
    • Attempts to evaluate it and fails
    • Views it as an arbitary html tag and closes it

But that is not the problem, yes I can use the text widget, or backticks to stop evaluation. The problem is if I am using strings of text that fit this pattern, the parser interfears making their use almost impossible.

Why am I trying to use strings of text that look like filters?

  • I am trying to write TiddlyWiki script to handle filters, not an unexpected desire given filters are pivotal to tiddlywikis operations.
  • I have a good example at the bottom of how I want to use it.

May I suggest a solution?

I would even be happy with an optional patch or a wikitext rule we can turn on/off.

  • Or perhaps a way to “escape this parsing”, when used

When the parser comes across an apparent filter in wikitext, that is not itself valid, like [[tiddler name]] (is valid), but do not attempt to parse it if it contains <>.

For example the following is valid in wikitext, or as text within a macro or procedure, as simple text;

[all[]]
[note]
[all{tiddler}]
[all{!!field}]

But as soon as we use <anything> inside such a filter it breaks eg [all<var>] as it also does in Discourse.

  • In wikitext <anything> is a valid html tag, but inside [ ] it is not really.
  • If we extend the parser to recognise [ ] as delimiting a simple string when it is not a link etc…

I was writting a custom widget and wanted to pass a string that defined a filter;

  • In this case I want to generate a JSON array of parameter value pairs.
\widget $jsonify.params()
\function parameters.value() [<params>jsonget<param>]
<$parameters $params=json-params>

# JSON: <<json-params>>
</$parameters>
\end $jsonify.params

<$jsonify.params my-filter="[<varname>]"/>
  • Any variable (macro or procedure) referenced within the value of the parameters fails.

This fails with the output reading;

   1. JSON: {"my-filter":"[]"}

HTML
<ol><li>JSON: {"my-filter":"[<transclusion>]"}</transclusion></li></ol></p>
  • I want to exclude the parameter value from being evaluated as a html attribute
  • I can use backticks or the text widget in prose, if nessasary, but in this case [ ... <var> ...] is an exception.
#11

What does it break if you use the text widget? You’ve shared what the actual results are (1. JSON: {"my-filter":"[]"}), but what is your desired result? I’m guessing maybe 1. JSON: {"my-filter":"[<varname>]"}. Using the text widget like the following gives that desired result:

\widget $jsonify.params()
<$parameters $params=json-params>

# JSON: <$text text=<<json-params>>/>
</$parameters>
\end $jsonify.params

<$jsonify.params my-filter="[<varname>]"/>

I’m guessing using the text widget in this way breaks something else for you? Could you share the test cases which work and don’t work along with the actual and desired results for those that don’t work? I didn’t really understand your post.

#12

In the example from your last post, you are already storing the filter string without evaluation into the attribute my-filter. If it wasn’t stored without evaluation (the "" quotes around the attribute’s value helps prevent evaluation), then the $text widget wouldn’t be able to get it out again.

I’m having trouble understanding what you mean by “storing filters without evaluation”.

#13

Fair enough but I just dont want the use of text, or attribute values containing a string like [something<varname>] [<varname>] returning “corrupt values”, because it considers it half of a html tag.

  • I think this is a fair enough desire.
  • A pretty basic one really.

But why?

If I want to pass a string containing an unevaluated filter, to a procedure or custom widget etc… it works unless it contains <varname>, I want this fixed.

Using the text widget is only useful to show the content, I expect to take such filters and make use of them but also possibly passing the unevaluated filter on.

Since filters are a key building block of tiddlywiki we should be able to pass and share filter syntax for subsequent use without it failing.

Specific case;

I can store keyword=value pairs in a JSON array. Consider ;

{"one":" filter ","no":"  filter ","pending":"  filter "}

I have a custom widget/procedure that can generate the above but I can not pass a filter into this tool if it contains <varname> of any type.

#14

The only evaluation that currently occurs is the erroneous handling of the <name> and it should know better when it is inside [ ] eg [.. <name> ..]

#15

I’ll just leave this here and leave, quietly…

\procedure stored-ct() [<currentTiddler>]

\procedure receiver(f)
  <$text text=<<f>> />
\end

Store a filter...

<$let stored-filter="[<currentTiddler>]">

Pass stored-filter to a proc:

<$transclude $variable=receiver f=<<stored-filter>> />

Pass stored-ct to a proc:

<$transclude $variable=receiver f=<<stored-ct>> />

Now Tony's list...

# <<stored-filter>>

# <<stored-ct>>

</$let>

Output:

image

#16

I haven’t seen anything in the thread to indicate that storing filters is causing them to be evaluated. However, rendering is an issue?

If I understand correctly the complaint is that these filter-looking strings

* [''bold'']
* [~~strikethrough~~]
* [<i>italics</i>]
* [<ILookLkeAnHtmlTagAndWillBeStripped>]

Render as:

  • [bold]
  • [strikethrough]
  • [italics]
  • []

and you’d rather everything inside the braces to remain untouched? (I guess the first two are bad examples because those aren’t actually valid filter syntax. The last two are, though)

As @jeremyruston said, filter syntax and wiki syntax are different. Rendering wikitext involves the wikitext parser and it doesn’t know anything about filter syntax.

#17

We are on the same page; As I asked;

#18

Here is an example of what I would like fixed, or at the least a workaround.

  • This is getting in my way of making some helpful solutions wiothout having to deal with this anoying exception. Specificaly writing documentation on filters.

Consider this custom widget to display code inline with its result;

\widget $show.code(code tooltip:"Inline code and result, click to copy" label result-label)
<$button tag=span tooltip=<<tooltip>> message="tm-copy-to-clipboard" param=<<code>> >
<<label>><span  style="background-color: #f2f2f2">"""<$text text=<<code>>/>"""</span>
</$button>&nbsp;<<result-label>>&nbsp;<span  style="border: 1px solid grey; border-style: dotted;"><<code>></span>
\end $show.code

This works Quite well with most small code snipits

# some thing <$show.code code="""My inline ''code'' <<transclusion>>"""/> something else

Snag_301d2cf5
Snag_301d2cf51877×51 8.05 KB

How ever when I do this

# <$show.code code="""[<transclusion>]"""/>
# [<transclusion>]

Snag_30234d32

This also screws up subsequent content.

  • The point here is [<transclusion>] is not valid wikitext, and only partial html

The solution add a parse rule to not interpret the content between [ ] as html and concider it a simple string that is used to represent filter strings. This wikirule could be toggled on/off if nessasary.

#19

Tones, @btheado made a great point up-thread: make it easier for us by showing your “desired output”, the results you would like to see. Parsing your verbose explanations is kinda hard and difficult to follow.

#20

This is probably your best bet as a solution and it might be quite tricky to get it right and not mess up link parsing.

Your new rule would need to be able to match these:

  1. [[1 2 3 4]enlist-input[]join<currentTiddler>]
  2. [[1 2 3 4]enlist-input[]join<currentTiddler>addsuffix[hello]]
  3. [all[current]prefix<myprefix>suffix<mysuffix>]
  4. [<myvariable>]
  5. [<myvariable>get{mytiddler}]
  6. [<myvariable>get[myfield]]

without matching these valid links

  1. [[this is a link]]
  2. [[this is|also a link]]
  3. [[maybe <anglebrackets> here|my tiddler]]
  4. [[<anglebrackets> maybe here|my tiddler]]

And #2 from the first list is debatable. That currently parses as a valid link.

1 Like