# Introduction
This PR introduces a number of improvements and new features r…elated to some of TiddlyWiki's most fundamental components: macros, widgets, operators and transclusion.
The motivation is to fix one of TiddlyWiki 5's early design flaws: the reliance on macros using textual substitution as the primary way to modularise and reuse wikitext and filters.
Experience has shown that while macros are a good match for a small number of tasks, they are brittle and error prone for many common operations. Over the years we have introduced mitigations for the worst problems but these have come at a cost of increased complexity.
The changes in this PR provide powerful new ways to achieve common tasks, and unlock completely new capabilities that were previously impossible in wikitext.
* **Procedures**, which are essentially what macros should have been; they work in exactly the same way except that parameters are exposed as simple variables (without the double underscores) and no textual substitution takes place
* **Custom widgets**, allowing the creation of widgets in wikitext, and the redefinition of built-in widgets
* **Functions**, a new way to encapsulate filter expressions with named parameters
* **Custom Filter Operators**, allowing functions to be used as custom filter operators
* **Parameterised transclusions**, allowing strings and wikitext trees to be passed to transclusions
Some smaller improvements are also needed to enable these new features:
* ~~New filter operators for **reading JSON data**~~ – superseded by #6936
* ~~A new **`<$genesis>`** widget that permits any other widget to be dynamically created, with dynamic attributes~~ – cherrypicked to #6961
* ~~A new **`<$error>`** widget for displaying core error messages~~ – cherrypicked into #6970
* ~~More effective protection against infinitely recursive functions and procedures~~ – cherrypicked into #6970
All of these changes are intended to be backwards compatible, and should not affect existing functionality. While they represent a new field of opportunities for wikitext authors, equally it is entirely possible for authors to ignore all these new features and continue to use TiddlyWiki 5 in the way that they have always done.
# Background
TiddlyWiki 5 macros were originally based on the technique we call "textual substitution": the string values of the parameters provided when calling a macro would be plugged into the macro definition before it was wikified in the usual way.
A typical example of the approach in early versions of TiddlyWiki 5:
```
\define mymacro(title)
<$codeblock code={{$title$}}/>
\end
```
The technique worked well enough to get the basics of the TiddlyWiki 5 user interface up and running, but it was clear from the start that it was annoyingly brittle. For example, the macro above would fail with tiddler titles containing double closing curly braces. Trying to use it with the title `foo}}bar` would lead to the macro being expanded to the following invalid syntax:
```
<$codeblock code={{foo}}bar}}/>
```
As a result, for a long time, the TiddlyWiki 5 user interface failed if a variety of combinations of special characters were found in tiddler titles. Long time users will remember a warning that popped up in the edit template whenever a potentially troublesome character was detected.
Over the years we've mitigated almost all of these issues, particularly by providing access to the macro parameters as variables. For backwards compatibility, this was done without affecting the existing syntax, which required us to adopt the clumsy protocol of wrapping the parameter name in double underscores to get the name of the corresponding variable.
This has all worked well enough for us to fix the UI issues with special characters in tiddler titles, but is very inconsistent and complex, requiring users to grasp multiple mutually exclusive conceptual models for what is going on.
# New Features and Improvements
The approach taken by this PR is to add new functionality by extending and augmenting the system without disturbing existing functionality.
This lays the groundwork for macros and related features to be deprecated, which is the point at which users are advised not to use old features, and instead given clear pointers to the equivalent modern functionality.
The new transclusion architecture is not by itself sufficient to enable us to fully deprecate macros yet. To handle the remaining use cases we propose a new backtick quoted attribute format that allows for the substitution of variable values. See https://github.com/Jermolene/TiddlyWiki5/issues/6663 for details.
# Procedures
Procedures are the modern replacement for macros. The key difference is that the parameters are only available as wikitext variables (without requiring double underscores), and that textual substitution does not occur.
The syntax for defining a procedure closely resembles the existing syntax for defining a macro:
```
\procedure hello(when:"afternoon")
<$list filter="[<when>match[morning]]">
Good morning!
</$list>
<$list filter="[<when>!match[morning]]">
Good afternoon or evening!
</$list>
\end
```
Procedures are invoked using the same shortcut syntax that is used for macros:
```
<<hello>>
<<hello "morning">>
<<hello when:"afternoon">>
```
Procedures can also be invoked as transcluded variable attributes using the syntax `<div class=<<hello>>>`. Note that the plain, unwikified text of the procedure will be used, without any parameter processing.
# Custom Widgets
Custom widgets are defined with the `\widget` pragma, which works analogously to the `\procedure` pragma.
The names of user defined widgets must start with the prefix `$$`.
Built-in widgets can be overridden by using a single `$`. Note that it is not possible to create new widgets named with a single `$`, only to override existing built-in widgets. This is to prevent users creating widgets whose names clash with future core widgets.
Invoking a custom widget transcludes the contents of the variable containing the definition. The widget definition can transclude the content of the calling widget by using the construction `<$slot $name="ts-raw"/>`.
For example:
```
\widget $$hello(name)
<div class="greeting">
<h1>Hello <$text text=<<name>>/></h1>
<$slot $name="ts-raw"/>
</div>
\end
```
The custom widget is invoked in the usual way:
```
<$$hello name="Jeremy">
This is the greeting
</$$hello>
```
That example would render as:
```
<div class="greeting">
<h1>Hello Jeremy</h1>
This is the greeting
</div>
```
# Functions
Functions are analogous to procedures except that they encapsulate filter expressions rather than wikitext. They are defined in the same way:
```
\function add-tax(rate:"10")
[multiply<rate>]
\end
```
Functions can be invoked via the new `function` operator:
```
Total: <$text text={{{ [<total>function[add-tax],[12.5]] }}}/>
```
Functions can also be invoked as transcluded variable attributes. For example:
```
<div class=<<myfunction "foobar">>>
...
</div>
```
# Custom Filter Operator Functions
If a user defined function name starts with a period then it can also be invoked directly as a custom filter operator:
```
\function .add-tax(rate:"1.10") [multiply<rate>]
...
Total: <$text text={{{ [<total>.add-tax[1.125]] }}}/>
```
# Transclude Widget Updates
Many of these improvements depend for their implementation on a significantly upgraded `<$transclude>` widget. To accommodate the new functionality without breaking backwards compatibility, the widget now operates in two modes:
* Modern mode enables all the new functionality. It is engaged when at least one attribute name starts with a $ character such as `$tiddler`, `$field`, `$index` etc.
* Legacy mode is fully backwards compatible, using attribute names such as `tiddler`, `field`, `index`. It is engaged when none of the attributes start with a $ character
Like macros, procedures are implemented as a special type of variable. The syntax for invoking procedures using the `<$transclude>` widget is as follows:
```
<$transclude $variable="hello" when="morning"/>
```
Note how the attributes `$variable`, `$tiddler`, `$field`, `$index` etc. that start with a single dollar sign are used to define what to transclude, while the plain attributes are used as parameters to be passed to the procedure.
The transclude widget can be used to pass named parameters to the transclusion. The transcluded content can reference the parameters via the `<$parameters>` widget.
```
<$transclude $tiddler="MyTiddler" param1="HelloThere"/>
```
## Parameterised Transclusion
The transclusion shortcut syntax has been extended to allow parameters to be passed to all types of transclusion, not just transclusions of macros/procedures/custom widgets.
The parameters are passed according to their position in the corresponding parameters declaration, not by name.
For example, with the text below in a tiddler titled "FooBar":
```
\parameters (a,b,c,d)
(definition of the transclusion goes here...)
```
The shortcut syntax for invoking the parameterised transclusion uses a single vertical bar to separate them from the title of the tiddler being transcluded:
```
{{FooBar|first|second|third|fourth}}
{{FooBar||template|first|second|third}}
```
Parameters can be omitted to skip them. For example:
```
{{FooBar|first||third|fourth}}
```
Note that omitting the first parameter `{{FooBar||second|third}}` creates an ambiguity because the resulting double vertical bar causes it to be interpreted as `{{title||template|param1}}`.
### Slotted Parameters
"Slots" provide a way to pass complex structured data to a transclusion instead of the simple string values supported by string parameters.
Slots are named areas that are defined within transcluded text using the `<$slot>` widget. They are replaced with custom content provided by a corresponding `<$fill>` widget inside the transclusion invocation. For example:
```
\procedure hello
<div class="frame">
<$slot $name="greeting"/>
</div>
\end
```
The syntax for invoking the procedure using the `<$transclude>` widget uses the `<$fill>` widget to pass the content for the slot:
```
<$transclude $variable="hello">
<$fill $name="greeting">
<h1>A heading</h1>
<p>A paragraph</p>
</$fill>
</$transclude >
```
The result would be equivalent to:
```
<div class="frame">
<h1>A heading</h1>
<p>A paragraph</p>
</div>
```
The contents of the `<$slot>` widget provide the default contents that are used in case a value is not provided for a named slot. For example:
```
\procedure hello
<div class="frame">
<$slot $name="greeting">
Default text for the named slot "greeting"
</$slot>
</div>
\end
```
The entire contents of the transclude widget invocation is available as the special slot fill value `ts-raw`.
### Specifying Content to Display When Target is Missing
Up until now, the content of the transclude widget has been used as the fallback content to display when the target of the transclusion is missing.
To allow room for the new `<$fill>` widget, the new behaviour is to use the special slot value `ts-missing` if present, and if there are no `<$fill>` widgets present, then falling back to the content of the transclude widget.
# Open Questions
These need to be settled before merging:
- Do custom widgets really need their own type of definition – they could just be implemented as procedures
- Review naming of "procedures" and "functions"
- Consider adding support for custom filter run prefixes. While it could be added later there's a chance that the design might influence the design of custom filter operators
- Is `$$` the best prefix for custom widgets? We have a (very) loose guideline that `$` stands for "system", and it is used as a kind of warning sign for system-y stuff that generally doesn't need to bother casual users. But we don't have an equivalent prefix for a user defined namespace. Perhaps `.`, as we're using with custom operators?
- Is `<$fill>` a good name? The rationale is that it pairs logically with `<$slot>` (which in turn reflects the W3C `<slot>` element). The trouble is that end users won't see the `<$slot>` widget until they start writing their own relatively complex procedures/widgets, and until that point `<$fill>` won't make much sense
- Consider making transcluding tiddler text fields understand fields like _parameters, _is_procedure etc
# Future Possibilities
These can be addressed after merging:
- Now that the entire parse tree of the contents of the transclude widget is passed to the transclusion as a JSON blob, it might be useful to be able to render a parse tree. In particular, a new flag for the transclude widget that causes it to interpret the target as a JSON parse tree
- Consider adding custom commands, analogous to procedures but for action strings. We may be able to retcon the existing commands into wikitext implementations too
- Refactor core icons to parameterise the width and height (the calendar icon can also parameterise the date that is shown)
# Progress
This PR is fully functional, and very, very nearly complete.
Incomplete:
- [x] Disable widget overrides in safe mode
- [x] Function operator should return the input list if the function is missing
- [x] Review performance impact of these changes
- [x] Documentation (see below)
- [ ] Refactor isFunctionDefinition/isProcedureDefinition/isWidgetDefinition flags into a single field?
- [x] Review whether other operators that have a fake widget.getVariable() handler (eg $:/core/modules/filters/filter.js) should also implement `widget.getVariableInfo()`
- [ ] Check that transclude widget refresh optimisations still work
- [ ] Review and rename test cases
- [ ] Extend set widget to set the hidden parse tree properties like isprocedure, and to be able to specify parameters
- [ ] Finish (or defer) visible transclusion demo
<p><details>
<summary>Completed</summary>
- [x] Recursion detection for functions
- [x] ubertransclude widget
- [x] parameters widget
- [x] slot widget
- [x] values widget
- [x] genesis widget
- [x] widget -> ubertransclusion mapping
- [x] use ubertransclude widget for wikitext transclude syntax
- [x] improved framework for wiki-based rendering tests
- [x] extend wikitext transclusion syntax with positional parameters
- [x] wikitext parser rule for new-style `\function` definitions
- [x] wikitext parser rule for `\parameters` construction
- [x] importvariables should skip the parameters widget
- [x] parse trees for transcluded variables should be cached
- [x] custom filter operators
- [x] genesis widget shouldn't evaluate attributes of child widget, leave it to the child widget
- [x] improve efficient of recursion detection (currently the $transclusion variable gets excessively long)
- [x] Support for `$:/tags/Global` alongside `$:/tags/Macro`
</details></p>
# Documentation progress
- [x] Procedure Definitions
- [x] Function Definitions
- [x] Widget Definitions
- [x] Macro Definitions retcon
- [x] Transclusion in WikiText update to add parameters
- [x] SlotWidget
- [x] ParametersWidget
- [x] MacroCallWidget updates – to note that the transclude widget is preferred now
- [x] ImportVariablesWidget updates to skip parameters widgets
- [x] FillWidget
- [x] ErrorWidget
- [x] SetWidget updates to add `conditional` attribute
- [x] Unknown Operator
- [x] Function Operator
- [x] TranscludeWidget updates
- [x] Format:Json Operator
- [x] JSON Operators Docs
- [x] Visible Transclusion How To
- [x] LetWidget update noting that dollars are now allowed
- [x] GenesisWidget
# Notes for documentation
- [ ] What we've previously referred to as "JavaScript Macros" are actually now better thought of as "JavaScript functions", because they work more like functions than macros
- [x] Note the difference between the subfilter and function operators (the former interprets the operand as the filter while the latter interprets the operand as the name of the variable containing the filter)
- [ ] Note that first parameter of user defined filter operator functions cannot be missing, because the empty double square brackets are required by the operator syntax (in other words, the operator syntax doesn't permit operators to be called with zero operands, just one or more)
- [ ] Note that macros, procedures, widgets and functions are all variables, and all share the same namespace. Definitions with the same name will overwrite one another
- [x] Fill widget $name attribute must be specified as a literal string, not a transcluded attribute
# Other tickets that can be closed when this is merged
- #3750
