Consider changing variable markers in documentation

[etardiff]

@Springer

I don’t like the idea of actually using «» characters in a way that can be seen / copied though … because I think that whatever code we have should be copy-pasteable and should just … “work”.

Absolutely. This is my top concern as well.

@Springer

Is there a smart way to get tw-com to overlay a kind of “smart syntax” about where what’s being shown is a filter-operator parameter and somehow make a tooltip there?

I think procedures are the simplest way to do this without adding any new syntax to the parser. For instance, I can quickly tweak the code I shared above to add some (ugly) highlighting to my example:

\procedure .docs-param(label)
	\function .param-type() [<.docs-param-config>get[text]] ~string
	\function .param-display()
		[<.param-type>match[variable]then[<XX>]]
		~[<.param-type>match[transclusion]then[{!!XX}]]
		~"[XX]"
		+[search-replace[XX],<label>]
	\end .param-display
@@<<.param-display>>@@
\end

Of course, I’d want something a little less eye-searing for the actual docs! But I do like the idea of using CSS classes to provide some additional visual cues; it emphasizes that the brackets are part of the parameter. And if we were using <span> instead of @@, it’d be easy to add tooltips and aria-labels, too.

[etardiff]

I said I wasn’t going to attempt a custom widget, but apparently I couldn’t help myself. Here’s a self-contained example (raw code follows at the end of the post): Sample $docs.filters.tid (2.0 KB) (Now revised as discussed in post #55, below)

It converts code like this…

<$docs.filters>
[range«end»]
[range«begin»,«end»]
[range«begin»,«end»,«step»]
</$docs.filters>

into output like this:

I have to confess that I had no idea how to type guillemets myself. (For my fellow Windows users, it’s Alt-174 for « and Alt-175 for »; the numbers must be typed on your numpad.) I’d really prefer a more standard character for use on TW-com, but I copy-pasted a few of them anyway to use as quick delimiters; they are, if nothing else, easy to read, and unlikely to appear in sample filters.

Other notes:

• I used $wikify to capture the value of <$slot $name=ts-raw /> for use in filtering, but surely (hopefully?) this isn’t best practice…?
• I used <<color primary>> to add some quick, palette-responsive color to the parameters. For a more polished version, we might prefer a more thoughtfully chosen palette. (Perhaps a different color for each type of parameter?)
• I didn’t put a lot of effort into styling the $select widget, but IMO, it would look better either outside the bounds of the <pre> block or inline with the first line of code.
• I added a quick tooltip to the $select to help clarify its use, but obviously, a <<lingo>> macro would be preferable. We could also include more explanatory text before or after the codeblock; here, I was trying to keep the display as clean as possible.

Full sample code:

\widget $docs.filters()
\function .docs-param-config() [[$:/temp/docs-config/]] [<storyTiddler>] +[join[]]

\procedure .docs-param(label)
	\function .param-type() [<.docs-param-config>get[text]] ~string
	\function .trim-label() [<.param-type>!match[string]] :then[<label>trim[]] ~[<label>]
	\function .param-display()
		[<.param-type>match[variable]then[<XX>]]
		~[<.param-type>match[transclusion]then[{!!XX}]]
		~"[XX]"
		+[search-replace[XX],<.trim-label>]
		:map[<.param-type>match[variable]then{!!title}search-replace:g[ ],[-]else{!!title}]
	\end .param-display
<span class="tc-param"><<.param-display>></span>
\end .docs-param

\procedure .docs-param-select()
<$select
	tiddler=<<.docs-param-config>>
	default="string"
	class="tc-float-right"
	tooltip="Select a type of parameter to see it used in the following filters.
You may mix and match parameter types within a single filter run."
>
	<option>string</option>
	<option>variable</option>
	<option>transclusion</option>
</$select>
\end .docs-param-select

\whitespace trim

<style>
.tc-param { color: <<color primary>>; }
</style>

<<.docs-param-select>>
<pre>
	<$wikify name=content text="<$slot $name=ts-raw />">
	<$list
		variable="output"
		filter="""[<content>search-replace:g[«],[<<.docs-param "]search-replace:g[»],[">>]]"""
	>
		<<output>>
	</$list>
	</$wikify>
</pre>
\end

<$docs.filters>
[range«end»]
[range«My Tiddler»]
[range«begin»,«end»]
[range«begin»,«end»,«step»]
</$docs.filters>

[Springer]

Wow wow wow!

The educational value of being able to select among the three kinds of parameter will certainly help folks grasp the flexibility of parameter syntax very quickly. AND whatever choice is active, the text is selectable, and is valid as-is without any new mystery characters.

People still have to understand that there can be a mix, and that the indirect reference doesn’t have to be a field of the current tiddler. But this is powerful stuff!

I’m tinkering with adding a hover-effect — maybe even a tooltip. But this is already inspiring!

[Springer]

Quick note, in case this limitation wasn’t already noticed: this proof-of-concept doesn’t yet parse spaces within a parameter value, as for example happens in tw-com’s examples for documenting addprefix and addsuffix operators:

https://tiddlywiki.com/#addprefix%20Operator%20(Examples)

[pmario]

IMO there is no need to create new doc-macros. I did test UPPERCASE placeholders and it looks good.

Preview: TiddlyWiki v5.3.8 — a non-linear personal web notebook

[etardiff]

I did very little testing, so all corrections, enhancements, and other feedback are much appreciated!

Here’s an updated version that should correct the issue: Sample $docs.filters.tid (2.0 KB)
I’ll update the code in my previous post as well.

In addition to correcting to account for multi-word values (just a matter of adding quote marks around the parameter), I added a couple of additional lines to my .param-display function:

\function .trim-label() [<.param-type>!match[string]] :then[<label>trim[]] ~[<label>]
\function .param-display()
[<.param-type>match[variable]then[<XX>]]
~[<.param-type>match[transclusion]then[{!!XX}]]
~"[XX]"
+[search-replace[XX],<.trim-label>]
:map[<.param-type>match[variable]then{!!title}search-replace:g[ ],[-]else{!!title}]
\end .param-display

• The final :map run replaces any spaces in the parameter with hyphens when displaying variable parameters. I didn’t make the same substitution for {!!field}-type parameters since field names can theoretically include spaces, but we certainly could do so there as well.
• .trim-label removes any leading or trailing space from non-literal parameters — and here, I did trim fields as well. You could have a field name with a trailing space, but IMO it’s asking for trouble, and not something we want to promote.

As a side note, I’m not entirely happy with my use of search-replace to substitute the param name; I chose it primarily because it felt more efficient than using addprefix and addsuffix (which can also be problematic when used with brackets, particularly [ — which merely looks like a mistake — and ] — which actually disrupts the filter parsing). If anyone has a more elegant solution, I’d love to hear it.

I’ll also note that while I do see the appeal of adding this extra functionality to all the sample filters, I really designed it while thinking of the syntax illustration “filters” currently used for filter operators like range (https://tiddlywiki.com/#range%20Operator) and search-replace (https://tiddlywiki.com/#search-replace%20Operator). The addprefix tiddler doesn’t seem to have one at present (perhaps because it’s harder to abstract?), but if I were writing one, I might do something like

Cat Garden [[Favourite Armchair]] +[addprefix<prefix>]

or the following, in my proof-of-concept syntax:

<$docs.filters>
Cat Garden [«Favourite Armchair»] +[addprefix«prefix»]
</$docs.filters>

[pmario]

@pmario

IMO there is no need to create new doc-macros

OK – That’s interesting and should be explored further. – But transclusions need to be more generic. We do use field transclusions for every filter. So those examples can be confusing too.

[etardiff]

I think the uppercase placeholders are somewhat clearer than the current “syntax” filters, and certainly lower-impact than a dynamic display in terms of macros required. But we would need to add additional tables illustrating dynamic usage (as you’ve done here) for every operator tiddler, so I’m not sure I’d call it easier as quick solutions go.

[pmario]

The problem I see with “fake code blocks” that consist of <pre> only will probably be screen readers.

We need to make sure that accessibility is part of our improvements, if we need to touch a lot of tiddlers anyway.

[etardiff]

@pmario

We need to make sure that accessibility is part of our improvements, if we need to touch a lot of tiddlers anyway.

Absolutely. My demo was really just a proof-of-concept; I’d hope and expect that it would be significantly improved before it got anywhere near the live TW-com code. Unfortunately, I haven’t studied accessible web design, so I don’t feel qualified to make recommendations on that front.

[TW_Tones]

Note: I have as yet not tried the code examples above of @etardiff but I would like to see a blend of active tooltips/popup, copy to clipboard, « » and UPPERCASE.

@Greyed

Couldn’t we just use all-capital letters for the variable markers, such as BEGIN / END / STEP?

Worth considering, but I don’t think it addresses the fact that such values can be replaced with [ ] { } and < >. As in Scotts reply @Scott_Sauyet

• If copying syntax from the documentation you don’t need to enter « » you need to replace/delete them.

@Greyed

Double-chevron characters like « » aren’t part of all standard keyboard layouts and may be difficult/annoying for some people to insert.

I contest this for the following reasons;

• Just find out how to do so and you can enter them from the keyboard
• We can provide an editor toolbar and other tools to use this and other select characters
• This is only needed when writing documentation and could even be applied through a macro.

@Scott_Sauyet

Not to mention the possibilities of mix-and-match:

And

@pmario

IMO there is no need to create new doc-macros. I did test UPPERCASE placeholders and it looks good.

This is in part what establishing « » or other unused brace as a standard does. Handle the alternative braces.

@etardiff

@Springer

I don’t like the idea of actually using «» characters in a way that can be seen / copied though … because I think that whatever code we have should be copy-pasteable and should just … “work”.

Absolutely. This is my top concern as well.

The popup could include the ability to copy the different forms without the «» however as I said these need only be replaced, and decision to be made when writing code as to which braces are appropriate. To get a working but possibly inappropriate syntax is not that helpful.

[nemo]

@Scott_Sauyet

@nemo

(seriously, I remain a fan of « » in part because they’re trivial to type under linux at least (compose+<+<)

I’m less concerned about ease-of-typing here than some others. Documentation is meant to be read much, much more than it’s written. It feels like a fairly minor thing.

I 100% agree. But others have raised it as a concern

However, the demo widget is inspiring. That sort of interactive documentation feels like it’s using the power of TW to tell about TW.

[twMat]

I think this is excellent:

@Springer

a hover effect on parameters might be very helpful, where

[range[18],[65]]

is what you see, but if you hover over [18] the expression [18] gets a background-highlight effect, and the tooltip shows

[18] (hard parameter)
{!!minimum-age} (indirect)
<changecount> (variable)

This and possibly some color or bold, to indicate there’s something special about it.

This would keep the immediately seen doc fully clear and readable - and the tooltips/popups allow the user to get more details on a need basis and without cluttering the immediate presentation.

How could the addition of such tooltips/popups be automated?

…and assuming it can be automated, then perhaps the full filter could even show instead of just the operand? I think beginners are often confused by how to apply things to get exactly the correct syntax, e.g

[range{!!minimum-age},{!!maximum-age}]

[Springer]

This is pretty much what @etardiff has accomplished in the proof-of-concept above.

I imagine it would be fairly straightforward to have a version that simply puts dummy placeholders into existing Examples (strings that don’t have an intuitive connection to the nature of the filter operator, but which show how the syntax works for the three main kinds of parameter).

Starting with a few key and vital operators, having semantically meaningful parameter names (like {!!minimum-age} and <selectedContact>) could be a follow-up step.

I think the Days of the Week tiddler is a great model. It’s a real tiddler (not just test-case) with fields that are useful for Example documentation.

I’d like to see a few more semantically intuitive tiddlers — maybe adding some non-sensitive biographical/trivia fields for JeremyRuston. Or adding a caption field (“TiddlyWiki mascot”) and bunch of biographical details to a MotovunJack tiddler (not yet a tiddler name, despite the image and robot variants). What else? Eiffel Tower? Tardis? Esperanto? Or a tiddler for Elements and for a few sample elements… with the Elements main text field pointing people to the Periodic Table demo by @Scott_Sauyet?) I recall the demo for the Tour plugin had some tiddlers with Planets. They would also be great for having semantically meaningful fields.

With an international world of learners out there, it’s hard to strike the balance of being culturally accessible with all examples, and tw-com shouldn’t be cluttered with too many extra tiddlers… But a few well-chosen ones would really allow for intuitive examples, I think!

[TW_Tones]

@Springer

it’s hard to strike the balance of being culturally accessible with all examples

I agree we should do our best on this, so one way of avoiding the complexities with cross cultural references is to resort to human/global and universal information. The reason I published the periodic table, Sometimes which are even common by reference to ancient languages eg Latin and Greek.

That is scientific bodies of knowledge including dinosaurs, geological epochs, mathematics, atoms/molecules, evolutionary. Although not perfect these are often where language differences are less because we all resort to a shared cannon of words.

[Scott_Sauyet]

@Scott_Sauyet

I’m wondering if the best bet would be to introduce new widgets that would allow us to parse and format filters in a universal way.

I’ve started playing around with a highlighter for filter syntax. It looks like this:

There are many different hover behaviors, which I won’t try to capture here. You can just take a look at it at http://scott.sauyet.com/Tiddlywiki/WIP/FilterFormatDemo.html. If you want to play with styles, the stylesheet can be found in the sidebar. So is the JS macro I used. You can try them in your own wiki using: FilterFormatDemo.json (50.8 KB)

This is all throw-away, proof-of-concept code. I wouldn’t expect to keep a line of it for a production implementation. I’ll discuss it more below, but the main question is, would something like this be useful for the docs site, to be used wherever we demonstrate filters?

Implementation Details

I built this atop a Parser Expression Grammar (PEG) that I wrote custom for this, using the dingus from Peggy.js. This allows us to write declarative descriptions of expressions in our grammar alongside JS code to generate a result, in this case an Abstract Syntax Tree (AST). Tiddlywiki already has a function to do this, $tw.util.parseExpression, but the tree it generated seemed to me to lack details I wanted to have. I would definitely revisit this choice. But even better would be to incorporate the Highlight Plugin, and add a mode for our filters. I don’t know precisely how to do this, but if we go this route, that would seem most likely.

The PEG grammar I used looks like this:

Filter 
  = _ head:Run tail:(_ Run)* _ 
   {return [head, ...tail.map(e => e[1])]}

Run
  = PrefixRun
  / StaticRun

PrefixRun
  = prefix:Prefix run:FilterRun
    {return {prefix, run}}
  / run: FilterRun
    {return {prefix: '', run}}

StaticRun
  = "\"" chars: [^"]* "\""
    { return {prefix: '', run: {steps: [{text: chars.join('').trim(), quotes: '"'}]}}; }
  / "'" chars: [^']* "'"
    { return {prefix: '', run: {steps: [{text: chars.join('').trim(), quotes:"'"}]}}; }
  / chars: [^'"\[]+
    { return {prefix: '', run: {steps: [{text: chars.join('').trim(), quotes: ''}]}}; }

Prefix
  = NamedPrefix
  / ShortcutPrefix
  
NamedPrefix
  = ":all" / ":and" / ":cascade" / ":else" / ":except" / ":filter"
  / ":intersection" / ":map" / ":or" / ":reduce"  / ":sort" / ":then"
  
ShortcutPrefix
  = "+" / "-" / "~" / "="
  
FilterRun
  = "[" steps:FilterSteps "]"
    {return {steps}}

FilterSteps
  = head:FilterStep tail:(_ FilterStep)* 
   {return [head, ...tail.map(e => e[1])]}

FilterStep
  = negated: ("!")? op:Operator suffixes:Suffixes? params:Params?
    {return {negated: !!negated, op, suffixes: suffixes || [], params: params || []}}
  / negated: ("!")? params:Params
    {return {negated: !!negated, op: '', params: params || []}}

Suffixes
  = head:Suffix tail:(Suffix)*
    {return [head, ...tail]}

Suffix
  = ":" name:[^:\[\{\< ]+
    {return {parts: name.join('').split(',')}}
  / ":"
    {return {parts: [""]}}

Params
  = head:Param tail:("," Param)*
    {return [head, ...tail.map(e => e[1])]}
 
Param
  = "[" hard:[^\]]+ "]"
    {return {type: 'hard', text: hard.join('')}}
  / "{" textRef:[^!}]+ "}"
    {return {type: 'textRef', tiddler: textRef.join('')}}
  / "{" textRef:[^!}]+ "!!" field:[^}]+ "}"
    {return {type: 'textRef', tiddler: textRef.join(''), field: field.join('')}}
  / "{" "!!" field:[^}]+ "}"
    {return {type: 'textRef', field: field.join('')}}
  / "<" varRef:[^<>]+ ">"
    {return {type: 'varRef', text: varRef.join('')}}
 
Operator
  = op: [^\[\]\:\<\{ ]+
    {return op.join('')}

_ "whitespace"
  = [ \t\n\r]*

This gets turned into JS parsing code, which is included in that macro.

For the filter [range[5],<max-val>,{!!step}] -[[20]] +[[9999]], this generates the tree:

[
  {
    prefix: '',
    run: {
      steps: [
        {
          negated: false,
          op: 'range',
          suffixes: [],
          params: [
            {
              type: 'hard',
              text: '5'
            },
            {
              type: 'varRef',
              text: 'max-val'
            },
            {
              type: 'textRef',
              field: 'step'
            }
          ]
        }
      ]
    }
  },
  {
    prefix: '-',
    run: {
      steps: [
        {
          negated: false,
          op: '',
          params: [
            {
              type: 'hard',
              text: '20'
            }
          ]
        }
      ]
    }
  },
  {
    prefix: '+',
    run: {
      steps: [
        {
          negated: false,
          op: '',
          params: [
            {
              type: 'hard',
              text: '9999'
            }
          ]
        }
      ]
    }
  }
]

That is then passed to some custom code, which converts that to the following HTML (without all the indentation and other white space):

<tt class="filter">
  <span class="complex run">
    <span class="punc sq-bracket">[</span>
    <span class="step">
      <span class="operator">range</span>
      <span class="param">
        <span class="punc sq-bracket">[</span>
        5
        <span class="punc sq-bracket">]</span>
      </span>
      <span class="punc comma">,</span>
      <span class="param">
        <span class="punc ang-bracket">&lt;</span>
        max-val
        <span class="punc ang-bracket">&gt;</span>
      </span>
      <span class="punc comma">,</span>
      <span class="param">
        <span class="punc curly-bracket">{</span>
        <span class="punc bangs">!!</span>
        <span class="field">step</span>
        <span class="punc curly-bracket">}</span>
      </span>
    </span>
    <span class="punc sq-bracket">]</span>
  </span> 
  <span class="prefix">-</span>
  <span class="complex run">
    <span class="punc sq-bracket">[</span>
    <span class="step">
      <span class="operator"></span>
      <span class="param">
        <span class="punc sq-bracket">[</span>
        20
        <span class="punc sq-bracket">]</span>
      </span>
    </span>
    <span class="punc sq-bracket">]</span>
    </span> 
    <span class="plain run"><span class="step">+</span>
  </span> 
  <span class="complex run">
    <span class="punc sq-bracket">[</span>
    <span class="step">
      <span class="operator"></span>
      <span class="param">
        <span class="punc sq-bracket">[</span>
        9999
        <span class="punc sq-bracket">]</span>
      </span>
    </span>
    <span class="punc sq-bracket">]</span>
  </span>
</tt>

Creating a module from this is just a matter of properly combining the parser generated by PEG alongside the AST->HTML code, and wrapping it in at HTML module,

Again, note that this is not production-ready, not even close. It’s a quickly hacked-together tool meant only to show the idea. But please play with it.

[TW_Tones]

Interesting demo Scott

Your example could help the debugging process.

I wonder if it could ever be possible to parse a filter into a plain language representation ?

eg [tag[example]has:field[caption]]

list all tiddlers (without shadows) with the tag example and has the field caption (even if empty) dominatly appended.

[tag[example]] [has:field[caption]]

list all tiddlers (without shadows) with the tag example and also;
list all tiddlers (without shadows) that have the field caption (even if empty) and dominatly appended.

[Scott_Sauyet]

@TW_Tones

I wonder if it could ever be possible to parse a filter into a plain language representation ?

It might be possible to start building that out of the information that’s in the main site (op-purpose might help, for instance), but only for filters all of whose operators are built-in ones. Custom operators would need some convention we could use to document them.

It would take changes to the core, though, to incorporate any such information into regular wikis.

And worse, there would be substantial complications for suffixes and for operators that have very different behaviors depending upon the number of parameters they take, such as this thread’s pet operator, range.

[TW_Tones]

@Scott_Sauyet

It would take changes to the core, though, to incorporate any such information into regular wikis.

Why do you think this? I am just thinking of an extension to the documentation, a plugin even. one has to intentionally give the filter to it like your syntax.

Perhaps an LLM could help but I still believe in systematic analysis.

perhaps we start with a collection of working found and submitted filters, with a community effort to provide one or more translations. it could be a kind of game.

• A good project for a crowd sourcing wiki. using a review read only wiki submission system I have being toying with.
• even just collecting working filter examples, then later testcases against a public example data set
• allowing others to submit third party filter operators

[Scott_Sauyet]

@TW_Tones

@Scott_Sauyet

It would take changes to the core, though, to incorporate any such information into regular wikis.

Why do you think this? I am just thinking of an extension to the documentation, a plugin even. one has to intentionally give the filter to it like your syntax.

You’re right. This could be a plug-in. But it would need to capture the information about each operator as stored now in the main site.

Could one use an LLM for this? Yes, although finding/creating enough training data would be difficult. Could you trust the results? No, not at all. The amount of confident bullshit that these tools generate is overwhelming.

While I’d love to be proven wrong, I also am sceptical about your crowd-sourced suggestion. I don’t think coming up with suggested filter explanations is the hard part. I think the hard part is choosing between competing texts and synthesizing the results.