Jermolene:master ← Jermolene:parameterised-transclusions
opened 02:48PM - 26 Apr 22 UTC
# Introduction
This PR introduces a redesigned architecture for transclusion …in TiddlyWiki with some significant new capabilities:
* The ability to pass parameters to transclusions, with default values for parameters that are not provided
* A simple, clean way to pass wikitext fragments as parameters into transclusions, without having to use triple double quotes
* The ability to define (or redefine) widgets entirely in wikitext
The cost of these improvements is that it may not be possible to take full advantage of these new features without breaking backwards compatibility.
NOTE: This PR currently uses the term “ubertransclude” as a placeholder pending decisions that are discussed in detail below.
# History
TiddlyWiki 5 macros were originally based on the technique we call "textual substitution": the string values of the parameters provided when calling a macro would be plugged into the macro definition before it was wikified in the usual way.
It led to a lot of wikitext that looked like this:
```
\define mymacro(title)
<$codeblock code={{$title$}}/>
\end
```
The technique worked well enough to get the basics of the TW5 user interface up and running, but it was clear from the start that it was annoyingly brittle. For example, the macro above would fail with tiddler titles containing double closing curly braces. Trying to use it with the title `foo}}bar` would lead to the macro being expanded to the following invalid syntax:
```
<$codeblock code={{foo}}bar}}/>
```
As a result, for a long time, the TW5 user interface failed if a variety of combinations of special characters were found in tiddler titles. Long time users will remember a warning that popped up in the edit template whenever a potentially troublesome character was detected.
Over the years we've mitigated almost all of these issues, particularly by providing access to the macro parameters as variables. For backwards compatibility, this was done without affecting the existing syntax, which required us to adopt the clumsy protocol of wrapping the parameter name in double underscores to get the name of the corresponding variable.
This has all worked well enough for us to fix the UI issues with special characters in tiddler titles, but is very inconsistent and complex, requiring users to grasp multiple mutually exclusive conceptual models for what is going on.
Since about 2015, I've been intermittently thinking and making notes to figure out a better approach. Building on the progress we've made in other areas of TiddlyWiki, a new architecture has gradually taken shape that unifies transclusions, macros and widgets into a single, coherent mechanism.
# New Transclusion Architecture
The new architecture includes the following features:
* A new "ubertransclude" widget that can transclude variables as well as tiddler fields/indexes/subtiddlers, subsuming both the existing transclude and macrocall widgets, and the associated wiki text syntax rules
* The ability to pass named string parameters to an ubertransclusion. The transcluded content can access the parameters as ordinary variables, and can also optionally specify default values to be used if the parameter is not provided. The default values are not restricted to fixed strings, any computed value can be used
* The ability to cleanly pass blocks of wikitext as parameters to macros, without having to pack the wikitext into a triple quoted string
* The ability to invoke macros using widget syntax (in other words, the ability to define custom widgets in wikitext)
* The ability to override built-in JavaScript widgets. For example, one could render a tiddler while providing a custom way to render the `<$link>` widgets within it
If we had a blank slate, the ubertransclude widget would replace the existing transclude and macrocall widgets, but for the moment it is implemented as a separate widget to make things easier to understand and explain.
For reasons discussed below, I think that in the end we may well need to make incompatible changes to our existing wikitext syntax to get the best out of the new architecture. That raises lots of questions about the future road map for TiddlyWiki, explicitly raising the possibility of a future version that intentionally breaks backwards compatibility in order to regain simplicity and consistency.
## Parameterised Transclusion
The new syntax for defining a macro closely resembles the existing syntax. The chief difference is that the parameters are only available as wikitext variables, and textual substitution cannot be used.
```
\macro hello(when="afternoon")
<$list filter="[<when>match[morning]]">
Good morning!
</$list>
<$list filter="[<when>!match[morning]]">
Good afternoon or evening!
</$list>
\end
```
The syntax for invoking the macro using the `<$ubertransclude>` widget is as follows:
```
<$ubertransclude $variable="hello" when="morning"/>
```
Note that the attributes `$variable`, `$tiddler`, `$field`, `$index` etc. that start with a dollar sign are used to define what to transclude, while the plain attributes are used as parameters to be passed to the macro.
As at present, defining a macro is really a special syntax for assigning the text of the macro to a variable. The special part is that the parameter declaration is handled by wrapping the content of the macro with a `<$parameters>` widget. So, the following is equivalent to the macro definition given above:
```
<$let hello="""
<$parameters when="afternoon">
<$list filter="[<when>match[morning]]">
Good morning!
</$list>
<$list filter="[<when>!match[morning]]">
Good afternoon or evening!
</$list>
</$parameters>
""">
<$ubertransclude $variable="hello" when="morning"/>
</$let>
```
## Slotted Parameters
"Slots" provide a way to pass complex structured data to a transclusion instead of the simple string values supported by string parameters.
Slots are named areas that are defined within transcluded text using the `<$slot>` widget. They are replaced with custom content provided by a corresponding `<$value>` widget inside the transclusion invocation. For example:
```
\macro hello
<div class="frame">
<$slot $name="greeting"/>
</div>
\end
```
The syntax for invoking the macro using the `<$ubertransclude>` widget uses the `<$value>` widget to pass the content for the slot:
```
<$ubertransclude $variable="hello">
<$value $name="greeting">
<h1>A heading</h1>
<p>A paragraph</p>
</$value>
</$ubertransclude>
```
The result would be equivalent to:
```
<div class="frame">
<h1>A heading</h1>
<p>A paragraph</p>
</div>
```
The contents of the `<$slot>` widget provide the default contents that are used in case a value is not provided for a named slot.
The special named value `missing` is used if the target of the transclusion is missing. For example:
```
\macro hello
<div class="frame">
<$slot $name="greeting">
Default text for the named slot "greeting"
</$slot>
</div>
\end
```
## Invoking Macros via Widget Syntax
Macros whose names conform to a special format can also be invoked via the standard widget syntax. When invoking a widget called `foo` the core looks for a variable called `<$foo>` and if it finds it, transcludes the contents of the variable instead of invoking the widget in the usual way. It also passes the content of the widget as the named slot value `body`.
For example:
```
\macro <$hello>(name)
<div class="greeting">
<h1>Hello <$text text=<<name>>/></h1>
<$slot $name="body"/>
</div>
\end
```
The custom widget is invoked in the usual way:
```
<$hello name="Jeremy">
This is the greeting
</$hello>
```
Would render as:
```
<div class="greeting">
This is the greeting
</div>
```
# Backwards Compatibility
The current implementation carefully avoids breaking backwards compatibility, but there are challenges to maintaining that position as the work progresses:
The most obvious difficulty is that we need a shortcut syntax for invoking a parameterised transclusion from a variable. This is the equivalent operation to our current `<<macroname>>` syntax, but with the new parameter handling. Options include:
* Reusing the existing `<<macroname param=value>>` syntax
* Adopting a new shortcut syntax. For example, we could use double parenthesis: `((macroname param=value))`
A second difficulty is what to call the new `<$ubertransclude>` widget. The only incompatibility with the existing `<$transclude>` widget is that the attributes `tiddler`, `field` etc. have been renamed `$tiddler`, `$field` etc to avoid clashing with any parameters of the same name. So there are again two options:
* Reuse the existing `<$transclude>` name, thus breaking backwards compatibility
* Adopt a new name for the new widget, making the existing `<$transclude>` widget be an alias with different attribute naming and without the ability to pass parameters
The new transclusion architecture is not sufficient to enable us to fully deprecate the existing textual substitution mechanism for parameter passing. To handle the remaining use cases I propose a new attribute form quoted with backticks that allows for the substitution of variable values. See https://github.com/Jermolene/TiddlyWiki5/issues/6663 for details.
# Progress
This PR is fully functional but incomplete. Some of the examples above use wikitext shortcut syntaxes that have not yet been implemented; the tests included in the PR show those same examples using the underlying raw widget syntax.
- [x] ubertransclude widget
- [x] parameters widget
- [x] slot widget
- [x] values widget
- [x] widget -> ubertransclusion mapping
- [x] use ubertransclude widget for wikitext transclude syntax
- [x] improved framework for wiki-based rendering tests
- [ ] fix and verify recursion detection
- [ ] extend wikitext transclusion syntax with positional parameters
- [ ] wikitext parser rule for new-style `\macro` definitions
- [ ] wikitext parser rule for `\parameters` construction
- [ ] importvariables should skip the parameters widget
- [ ] parse trees for transcluded variables should be cached
- [ ] check that refresh optimisations still work
