How to read about documentation regarding Parameterised transclusions

arunnbabu81 · September 26, 2022, 5:07pm

From GitHub notifications, I can see that @jeremyruston is constantly adding some commits regarding Parameterised transclusions including some documentation. I would like read those documentation about Parameterised transclusions. How to do that ?

jeremyruston · September 26, 2022, 8:20pm

Hi @arunnbabu81 the docs for the work on parameterised transclusion are not quite complete.

There is an overview of the changes at the top of the pull request:

github.com/Jermolene/TiddlyWiki5

Parameterised transclusions

Jermolene:master ← Jermolene:parameterised-transclusions

opened 02:48PM - 26 Apr 22 UTC

Jermolene

+3127 -439

# Introduction This PR introduces a number of improvements and new features r…elated to some of TiddlyWiki's most fundamental components: macros, widgets, operators and transclusion. The motivation is to fix one of TiddlyWiki 5's early design flaws: the reliance on macros using textual substitution as the primary way to modularise and reuse wikitext and filters. Experience has shown that while macros are a good match for a small number of tasks, they are brittle and error prone for many common operations. Over the years we have introduced mitigations for the worst problems but these have come at a cost of increased complexity. The changes in this PR provide powerful new ways to achieve common tasks, and unlock completely new capabilities that were previously impossible in wikitext. * **Procedures**, which are essentially what macros should have been; they work in exactly the same way except that parameters are exposed as simple variables (without the double underscores) and no textual substitution takes place * **Custom widgets**, allowing the creation of widgets in wikitext, and the redefinition of built-in widgets * **Functions**, a new way to encapsulate filter expressions with named parameters * **Custom Filter Operators**, allowing functions to be used as custom filter operators * **Parameterised transclusions**, allowing strings and wikitext trees to be passed to transclusions Some smaller improvements are also needed to enable these new features: * ~~New filter operators for **reading JSON data**~~ – superseded by #6936 * A new **`<$genesis>`** widget that permits any other widget to be dynamically created, with dynamic attributes * A new **`<$error>`** widget for displaying core error messages * **Conditional definitions** that only occur if the target variable does not already have a value * More effective protection against infinitely recursive functions and procedures All of these changes are intended to be backwards compatible, and should not affect existing functionality. While they represent a new field of opportunities for wikitext authors, equally it is entirely possible for authors to ignore all these new features and continue to use TiddlyWiki 5 in the way that they have always done. # Background TiddlyWiki 5 macros were originally based on the technique we call "textual substitution": the string values of the parameters provided when calling a macro would be plugged into the macro definition before it was wikified in the usual way. A typical example of the approach in early versions of TiddlyWiki 5: ``` \define mymacro(title) <$codeblock code={{$title$}}/> \end ``` The technique worked well enough to get the basics of the TiddlyWiki 5 user interface up and running, but it was clear from the start that it was annoyingly brittle. For example, the macro above would fail with tiddler titles containing double closing curly braces. Trying to use it with the title `foo}}bar` would lead to the macro being expanded to the following invalid syntax: ``` <$codeblock code={{foo}}bar}}/> ``` As a result, for a long time, the TiddlyWiki 5 user interface failed if a variety of combinations of special characters were found in tiddler titles. Long time users will remember a warning that popped up in the edit template whenever a potentially troublesome character was detected. Over the years we've mitigated almost all of these issues, particularly by providing access to the macro parameters as variables. For backwards compatibility, this was done without affecting the existing syntax, which required us to adopt the clumsy protocol of wrapping the parameter name in double underscores to get the name of the corresponding variable. This has all worked well enough for us to fix the UI issues with special characters in tiddler titles, but is very inconsistent and complex, requiring users to grasp multiple mutually exclusive conceptual models for what is going on. # New Features and Improvements The approach taken by this PR is to add new functionality by extending and augmenting the system without disturbing existing functionality. This lays the groundwork for macros and related features to be deprecated, which is the point at which users are advised not to use old features, and instead given clear pointers to the equivalent modern functionality. The new transclusion architecture is not by itself sufficient to enable us to fully deprecate macros yet. To handle the remaining use cases we propose a new backtick quoted attribute format that allows for the substitution of variable values. See https://github.com/Jermolene/TiddlyWiki5/issues/6663 for details. # Procedures Procedures are the modern replacement for macros. The key difference is that the parameters are only available as wikitext variables (without requiring double underscores), and that textual substitution does not occur. The syntax for defining a procedure closely resembles the existing syntax for defining a macro: ``` \procedure hello(when:"afternoon") <$list filter="[<when>match[morning]]"> Good morning! </$list> <$list filter="[<when>!match[morning]]"> Good afternoon or evening! </$list> \end ``` Note that unlike macros, the parenthesis can be omitted when there are no parameters: ``` \procedure name value ``` Procedures are invoked using the same shortcut syntax that is used for macros: ``` <<hello>> <<hello "morning">> <<hello when:"afternoon">> ``` Procedures can also be invoked as transcluded variable attributes using the syntax `<div class=<<hello>>>`. Note that the plain, unwikified text of the procedure will be used, without any parameter processing. # Custom Widgets Custom widgets are defined with the `\widget` pragma, which works analogously to the `\procedure` pragma. The names of user defined widgets must start with the prefix `$$`. Built-in widgets can be overridden by using a single `$`. Note that it is not possible to create new widgets named with a single `$`, only to override existing built-in widgets. This is to prevent users creating widgets whose names clash with future core widgets. Invoking a custom widget transcludes the contents of the variable containing the definition. The widget definition can transclude the content of the calling widget by using the construction `<$slot $name="ts-raw"/>`. For example: ``` \widget $$hello(name) <div class="greeting"> <h1>Hello <$text text=<<name>>/></h1> <$slot $name="ts-raw"/> </div> \end ``` The custom widget is invoked in the usual way: ``` <$$hello name="Jeremy"> This is the greeting </$$hello> ``` That example would render as: ``` <div class="greeting"> <h1>Hello Jeremy</h1> This is the greeting </div> ``` # Functions Functions are analogous to procedures except that they encapsulate filter expressions rather than wikitext. They are defined in the same way: ``` \function add-tax(rate:"10") [multiply<rate>] \end ``` Functions can be invoked via the new `function` operator: ``` Total: <$text text={{{ [<total>function[add-tax],[12.5]] }}}/> ``` Functions can also be invoked as transcluded variable attributes. For example: ``` <div class=<<myfunction "foobar">>> ... </div> ``` # Custom Filter Operator Functions If a user defined function name starts with a period then it can also be invoked directly as a custom filter operator: ``` \function .add-tax(rate:"1.10") [multiply<rate>] ... Total: <$text text={{{ [<total>.add-tax[1.125]] }}}/> ``` # Transclude Widget Updates Many of these improvements depend for their implementation on a significantly upgraded `<$transclude>` widget. To accommodate the new functionality without breaking backwards compatibility, the widget now operates in two modes: * Modern mode enables all the new functionality. It is engaged when at least one attribute name starts with a $ character such as `$tiddler`, `$field`, `$index` etc. * Legacy mode is fully backwards compatible, using attribute names such as `tiddler`, `field`, `index`. It is engaged when none of the attributes start with a $ character Like macros, procedures are implemented as a special type of variable. The syntax for invoking procedures using the `<$transclude>` widget is as follows: ``` <$transclude $variable="hello" when="morning"/> ``` Note how the attributes `$variable`, `$tiddler`, `$field`, `$index` etc. that start with a single dollar sign are used to define what to transclude, while the plain attributes are used as parameters to be passed to the procedure. The transclude widget can be used to pass named parameters to the transclusion. The transcluded content can reference the parameters via the `<$parameters>` widget. ``` <$transclude $tiddler="MyTiddler" param1="HelloThere"/> ``` ## Parameterised Transclusion The transclusion shortcut syntax has been extended to allow parameters to be passed to all types of transclusion, not just transclusions of macros/procedures/custom widgets. The parameters are passed according to their position in the corresponding parameters declaration, not by name. For example, with the text below in a tiddler titled "FooBar": ``` \parameters (a,b,c,d) (definition of the transclusion goes here...) ``` The shortcut syntax for invoking the parameterised transclusion uses a single vertical bar to separate them from the title of the tiddler being transcluded: ``` {{FooBar|first|second|third|fourth}} {{FooBar||template|first|second|third}} ``` Parameters can be omitted to skip them. For example: ``` {{FooBar|first||third|fourth}} ``` Note that omitting the first parameter `{{FooBar||second|third}}` creates an ambiguity because the resulting double vertical bar causes it to be interpreted as `{{title||template|param1}}`. ### Slotted Parameters "Slots" provide a way to pass complex structured data to a transclusion instead of the simple string values supported by string parameters. Slots are named areas that are defined within transcluded text using the `<$slot>` widget. They are replaced with custom content provided by a corresponding `<$fill>` widget inside the transclusion invocation. For example: ``` \procedure hello <div class="frame"> <$slot $name="greeting"/> </div> \end ``` The syntax for invoking the procedure using the `<$transclude>` widget uses the `<$fill>` widget to pass the content for the slot: ``` <$transclude $variable="hello"> <$fill $name="greeting"> <h1>A heading</h1> <p>A paragraph</p> </$fill> </$transclude > ``` The result would be equivalent to: ``` <div class="frame"> <h1>A heading</h1> <p>A paragraph</p> </div> ``` The contents of the `<$slot>` widget provide the default contents that are used in case a value is not provided for a named slot. For example: ``` \procedure hello <div class="frame"> <$slot $name="greeting"> Default text for the named slot "greeting" </$slot> </div> \end ``` The entire contents of the transclude widget invocation is available as the special slot fill value `ts-raw`. ### Specifying Content to Display When Target is Missing Up until now, the content of the transclude widget has been used as the fallback content to display when the target of the transclusion is missing. To allow room for the new `<$fill>` widget, the new behaviour is to use the special slot value `ts-missing` if present, and if there are no `<$fill>` widgets present, then falling back to the content of the transclude widget. # Open Questions These need to be settled before merging: - Do custom widgets really need their own type of definition – they could just be implemented as procedures - Review naming of "procedures" and "functions" - Consider adding support for custom filter run prefixes. While it could be added later there's a chance that the design might influence the design of custom filter operators - Is `$$` the best prefix for custom widgets? We have a (very) loose guideline that `$` stands for "system", and it is used as a kind of warning sign for system-y stuff that generally doesn't need to bother casual users. But we don't have an equivalent prefix for a user defined namespace. Perhaps `.`, as we're using with custom operators? - Is `<$fill>` a good name? The rationale is that it pairs logically with `<$slot>` (which in turn reflects the W3C `<slot>` element). The trouble is that end users won't see the `<$slot>` widget until they start writing their own relatively complex procedures/widgets, and until that point `<$fill>` won't make much sense - Consider making transcluding tiddler text fields understand fields like _parameters, _is_procedure etc # Future Possibilities These can be addressed after merging: - Now that the entire parse tree of the contents of the transclude widget is passed to the transclusion as a JSON blob, it might be useful to be able to render a parse tree. In particular, a new flag for the transclude widget that causes it to interpret the target as a JSON parse tree - Consider adding custom commands, analogous to procedures but for action strings. We may be able to retcon the existing commands into wikitext implementations too - Refactor core icons to parameterise the width and height (the calendar icon can also parameterise the date that is shown) # Progress This PR is fully functional, and very, very nearly complete. Incomplete: - [x] Disable widget overrides in safe mode - [x] Function operator should return the input list if the function is missing - [x] Review performance impact of these changes - [ ] Documentation (see below) - [ ] Refactor isFunctionDefinition/isProcedureDefinition/isWidgetDefinition flags into a single field? - [ ] Review whether other operators that have a fake widget.getVariable() handler (eg $:/core/modules/filters/filter.js) should also implement `widget.getVariableInfo()` - [ ] Check that transclude widget refresh optimisations still work - [ ] Review and rename test cases - [ ] Extend set widget to set the hidden parse tree properties like isprocedure, and to be able to specify parameters - [ ] Finish (or defer) visible transclusion demo <p><details> <summary>Completed</summary> - [x] Recursion detection for functions - [x] ubertransclude widget - [x] parameters widget - [x] slot widget - [x] values widget - [x] genesis widget - [x] widget -> ubertransclusion mapping - [x] use ubertransclude widget for wikitext transclude syntax - [x] improved framework for wiki-based rendering tests - [x] extend wikitext transclusion syntax with positional parameters - [x] wikitext parser rule for new-style `\function` definitions - [x] wikitext parser rule for `\parameters` construction - [x] importvariables should skip the parameters widget - [x] parse trees for transcluded variables should be cached - [x] custom filter operators - [x] genesis widget shouldn't evaluate attributes of child widget, leave it to the child widget - [x] improve efficient of recursion detection (currently the $transclusion variable gets excessively long) - [x] Support for `$:/tags/Global` alongside `$:/tags/Macro` </details></p> # Documentation progress - [ ] Procedure Definitions - [ ] Function Definitions - [ ] Widget Definitions - [x] Macro Definitions retcon - [ ] Transclusion in WikiText update to add parameters - [x] SlotWidget - [x] ParametersWidget - [x] MacroCallWidget updates – to note that the transclude widget is preferred now - [x] ImportVariablesWidget updates to skip parameters widgets - [x] FillWidget - [x] ErrorWidget - [x] SetWidget updates to add `conditional` attribute - [x] Unknown Operator - [x] Function Operator - [x] TranscludeWidget updates - [x] Format:Json Operator - [x] JSON Operators Docs - [x] Visible Transclusion How To - [x] LetWidget update noting that dollars are now allowed - [x] GenesisWidget # Notes for documentation - [ ] What we've previously referred to as "JavaScript Macros" are actually now better thought of as "JavaScript functions", because they work more like functions than macros - [x] Note the difference between the subfilter and function operators (the former interprets the operand as the filter while the latter interprets the operand as the name of the variable containing the filter) - [ ] Note that first parameter of user defined filter operator functions cannot be missing, because the empty double square brackets are required by the operator syntax (in other words, the operator syntax doesn't permit operators to be called with zero operands, just one or more) - [ ] Note that macros, procedures, widgets and functions are all variables, and all share the same namespace. Definitions with the same name will overwrite one another - [x] Fill widget $name attribute must be specified as a literal string, not a transcluded attribute # Other tickets that can be closed when this is merged - #3750

You can see the raw documentation tiddlers starting here:

github.com/Jermolene/TiddlyWiki5

Parameterised transclusions

Jermolene:master ← Jermolene:parameterised-transclusions

opened 02:48PM - 26 Apr 22 UTC

Jermolene

+3127 -439

# Introduction This PR introduces a number of improvements and new features r…elated to some of TiddlyWiki's most fundamental components: macros, widgets, operators and transclusion. The motivation is to fix one of TiddlyWiki 5's early design flaws: the reliance on macros using textual substitution as the primary way to modularise and reuse wikitext and filters. Experience has shown that while macros are a good match for a small number of tasks, they are brittle and error prone for many common operations. Over the years we have introduced mitigations for the worst problems but these have come at a cost of increased complexity. The changes in this PR provide powerful new ways to achieve common tasks, and unlock completely new capabilities that were previously impossible in wikitext. * **Procedures**, which are essentially what macros should have been; they work in exactly the same way except that parameters are exposed as simple variables (without the double underscores) and no textual substitution takes place * **Custom widgets**, allowing the creation of widgets in wikitext, and the redefinition of built-in widgets * **Functions**, a new way to encapsulate filter expressions with named parameters * **Custom Filter Operators**, allowing functions to be used as custom filter operators * **Parameterised transclusions**, allowing strings and wikitext trees to be passed to transclusions Some smaller improvements are also needed to enable these new features: * ~~New filter operators for **reading JSON data**~~ – superseded by #6936 * A new **`<$genesis>`** widget that permits any other widget to be dynamically created, with dynamic attributes * A new **`<$error>`** widget for displaying core error messages * **Conditional definitions** that only occur if the target variable does not already have a value * More effective protection against infinitely recursive functions and procedures All of these changes are intended to be backwards compatible, and should not affect existing functionality. While they represent a new field of opportunities for wikitext authors, equally it is entirely possible for authors to ignore all these new features and continue to use TiddlyWiki 5 in the way that they have always done. # Background TiddlyWiki 5 macros were originally based on the technique we call "textual substitution": the string values of the parameters provided when calling a macro would be plugged into the macro definition before it was wikified in the usual way. A typical example of the approach in early versions of TiddlyWiki 5: ``` \define mymacro(title) <$codeblock code={{$title$}}/> \end ``` The technique worked well enough to get the basics of the TiddlyWiki 5 user interface up and running, but it was clear from the start that it was annoyingly brittle. For example, the macro above would fail with tiddler titles containing double closing curly braces. Trying to use it with the title `foo}}bar` would lead to the macro being expanded to the following invalid syntax: ``` <$codeblock code={{foo}}bar}}/> ``` As a result, for a long time, the TiddlyWiki 5 user interface failed if a variety of combinations of special characters were found in tiddler titles. Long time users will remember a warning that popped up in the edit template whenever a potentially troublesome character was detected. Over the years we've mitigated almost all of these issues, particularly by providing access to the macro parameters as variables. For backwards compatibility, this was done without affecting the existing syntax, which required us to adopt the clumsy protocol of wrapping the parameter name in double underscores to get the name of the corresponding variable. This has all worked well enough for us to fix the UI issues with special characters in tiddler titles, but is very inconsistent and complex, requiring users to grasp multiple mutually exclusive conceptual models for what is going on. # New Features and Improvements The approach taken by this PR is to add new functionality by extending and augmenting the system without disturbing existing functionality. This lays the groundwork for macros and related features to be deprecated, which is the point at which users are advised not to use old features, and instead given clear pointers to the equivalent modern functionality. The new transclusion architecture is not by itself sufficient to enable us to fully deprecate macros yet. To handle the remaining use cases we propose a new backtick quoted attribute format that allows for the substitution of variable values. See https://github.com/Jermolene/TiddlyWiki5/issues/6663 for details. # Procedures Procedures are the modern replacement for macros. The key difference is that the parameters are only available as wikitext variables (without requiring double underscores), and that textual substitution does not occur. The syntax for defining a procedure closely resembles the existing syntax for defining a macro: ``` \procedure hello(when:"afternoon") <$list filter="[<when>match[morning]]"> Good morning! </$list> <$list filter="[<when>!match[morning]]"> Good afternoon or evening! </$list> \end ``` Note that unlike macros, the parenthesis can be omitted when there are no parameters: ``` \procedure name value ``` Procedures are invoked using the same shortcut syntax that is used for macros: ``` <<hello>> <<hello "morning">> <<hello when:"afternoon">> ``` Procedures can also be invoked as transcluded variable attributes using the syntax `<div class=<<hello>>>`. Note that the plain, unwikified text of the procedure will be used, without any parameter processing. # Custom Widgets Custom widgets are defined with the `\widget` pragma, which works analogously to the `\procedure` pragma. The names of user defined widgets must start with the prefix `$$`. Built-in widgets can be overridden by using a single `$`. Note that it is not possible to create new widgets named with a single `$`, only to override existing built-in widgets. This is to prevent users creating widgets whose names clash with future core widgets. Invoking a custom widget transcludes the contents of the variable containing the definition. The widget definition can transclude the content of the calling widget by using the construction `<$slot $name="ts-raw"/>`. For example: ``` \widget $$hello(name) <div class="greeting"> <h1>Hello <$text text=<<name>>/></h1> <$slot $name="ts-raw"/> </div> \end ``` The custom widget is invoked in the usual way: ``` <$$hello name="Jeremy"> This is the greeting </$$hello> ``` That example would render as: ``` <div class="greeting"> <h1>Hello Jeremy</h1> This is the greeting </div> ``` # Functions Functions are analogous to procedures except that they encapsulate filter expressions rather than wikitext. They are defined in the same way: ``` \function add-tax(rate:"10") [multiply<rate>] \end ``` Functions can be invoked via the new `function` operator: ``` Total: <$text text={{{ [<total>function[add-tax],[12.5]] }}}/> ``` Functions can also be invoked as transcluded variable attributes. For example: ``` <div class=<<myfunction "foobar">>> ... </div> ``` # Custom Filter Operator Functions If a user defined function name starts with a period then it can also be invoked directly as a custom filter operator: ``` \function .add-tax(rate:"1.10") [multiply<rate>] ... Total: <$text text={{{ [<total>.add-tax[1.125]] }}}/> ``` # Transclude Widget Updates Many of these improvements depend for their implementation on a significantly upgraded `<$transclude>` widget. To accommodate the new functionality without breaking backwards compatibility, the widget now operates in two modes: * Modern mode enables all the new functionality. It is engaged when at least one attribute name starts with a $ character such as `$tiddler`, `$field`, `$index` etc. * Legacy mode is fully backwards compatible, using attribute names such as `tiddler`, `field`, `index`. It is engaged when none of the attributes start with a $ character Like macros, procedures are implemented as a special type of variable. The syntax for invoking procedures using the `<$transclude>` widget is as follows: ``` <$transclude $variable="hello" when="morning"/> ``` Note how the attributes `$variable`, `$tiddler`, `$field`, `$index` etc. that start with a single dollar sign are used to define what to transclude, while the plain attributes are used as parameters to be passed to the procedure. The transclude widget can be used to pass named parameters to the transclusion. The transcluded content can reference the parameters via the `<$parameters>` widget. ``` <$transclude $tiddler="MyTiddler" param1="HelloThere"/> ``` ## Parameterised Transclusion The transclusion shortcut syntax has been extended to allow parameters to be passed to all types of transclusion, not just transclusions of macros/procedures/custom widgets. The parameters are passed according to their position in the corresponding parameters declaration, not by name. For example, with the text below in a tiddler titled "FooBar": ``` \parameters (a,b,c,d) (definition of the transclusion goes here...) ``` The shortcut syntax for invoking the parameterised transclusion uses a single vertical bar to separate them from the title of the tiddler being transcluded: ``` {{FooBar|first|second|third|fourth}} {{FooBar||template|first|second|third}} ``` Parameters can be omitted to skip them. For example: ``` {{FooBar|first||third|fourth}} ``` Note that omitting the first parameter `{{FooBar||second|third}}` creates an ambiguity because the resulting double vertical bar causes it to be interpreted as `{{title||template|param1}}`. ### Slotted Parameters "Slots" provide a way to pass complex structured data to a transclusion instead of the simple string values supported by string parameters. Slots are named areas that are defined within transcluded text using the `<$slot>` widget. They are replaced with custom content provided by a corresponding `<$fill>` widget inside the transclusion invocation. For example: ``` \procedure hello <div class="frame"> <$slot $name="greeting"/> </div> \end ``` The syntax for invoking the procedure using the `<$transclude>` widget uses the `<$fill>` widget to pass the content for the slot: ``` <$transclude $variable="hello"> <$fill $name="greeting"> <h1>A heading</h1> <p>A paragraph</p> </$fill> </$transclude > ``` The result would be equivalent to: ``` <div class="frame"> <h1>A heading</h1> <p>A paragraph</p> </div> ``` The contents of the `<$slot>` widget provide the default contents that are used in case a value is not provided for a named slot. For example: ``` \procedure hello <div class="frame"> <$slot $name="greeting"> Default text for the named slot "greeting" </$slot> </div> \end ``` The entire contents of the transclude widget invocation is available as the special slot fill value `ts-raw`. ### Specifying Content to Display When Target is Missing Up until now, the content of the transclude widget has been used as the fallback content to display when the target of the transclusion is missing. To allow room for the new `<$fill>` widget, the new behaviour is to use the special slot value `ts-missing` if present, and if there are no `<$fill>` widgets present, then falling back to the content of the transclude widget. # Open Questions These need to be settled before merging: - Do custom widgets really need their own type of definition – they could just be implemented as procedures - Review naming of "procedures" and "functions" - Consider adding support for custom filter run prefixes. While it could be added later there's a chance that the design might influence the design of custom filter operators - Is `$$` the best prefix for custom widgets? We have a (very) loose guideline that `$` stands for "system", and it is used as a kind of warning sign for system-y stuff that generally doesn't need to bother casual users. But we don't have an equivalent prefix for a user defined namespace. Perhaps `.`, as we're using with custom operators? - Is `<$fill>` a good name? The rationale is that it pairs logically with `<$slot>` (which in turn reflects the W3C `<slot>` element). The trouble is that end users won't see the `<$slot>` widget until they start writing their own relatively complex procedures/widgets, and until that point `<$fill>` won't make much sense - Consider making transcluding tiddler text fields understand fields like _parameters, _is_procedure etc # Future Possibilities These can be addressed after merging: - Now that the entire parse tree of the contents of the transclude widget is passed to the transclusion as a JSON blob, it might be useful to be able to render a parse tree. In particular, a new flag for the transclude widget that causes it to interpret the target as a JSON parse tree - Consider adding custom commands, analogous to procedures but for action strings. We may be able to retcon the existing commands into wikitext implementations too - Refactor core icons to parameterise the width and height (the calendar icon can also parameterise the date that is shown) # Progress This PR is fully functional, and very, very nearly complete. Incomplete: - [x] Disable widget overrides in safe mode - [x] Function operator should return the input list if the function is missing - [x] Review performance impact of these changes - [ ] Documentation (see below) - [ ] Refactor isFunctionDefinition/isProcedureDefinition/isWidgetDefinition flags into a single field? - [ ] Review whether other operators that have a fake widget.getVariable() handler (eg $:/core/modules/filters/filter.js) should also implement `widget.getVariableInfo()` - [ ] Check that transclude widget refresh optimisations still work - [ ] Review and rename test cases - [ ] Extend set widget to set the hidden parse tree properties like isprocedure, and to be able to specify parameters - [ ] Finish (or defer) visible transclusion demo <p><details> <summary>Completed</summary> - [x] Recursion detection for functions - [x] ubertransclude widget - [x] parameters widget - [x] slot widget - [x] values widget - [x] genesis widget - [x] widget -> ubertransclusion mapping - [x] use ubertransclude widget for wikitext transclude syntax - [x] improved framework for wiki-based rendering tests - [x] extend wikitext transclusion syntax with positional parameters - [x] wikitext parser rule for new-style `\function` definitions - [x] wikitext parser rule for `\parameters` construction - [x] importvariables should skip the parameters widget - [x] parse trees for transcluded variables should be cached - [x] custom filter operators - [x] genesis widget shouldn't evaluate attributes of child widget, leave it to the child widget - [x] improve efficient of recursion detection (currently the $transclusion variable gets excessively long) - [x] Support for `$:/tags/Global` alongside `$:/tags/Macro` </details></p> # Documentation progress - [ ] Procedure Definitions - [ ] Function Definitions - [ ] Widget Definitions - [x] Macro Definitions retcon - [ ] Transclusion in WikiText update to add parameters - [x] SlotWidget - [x] ParametersWidget - [x] MacroCallWidget updates – to note that the transclude widget is preferred now - [x] ImportVariablesWidget updates to skip parameters widgets - [x] FillWidget - [x] ErrorWidget - [x] SetWidget updates to add `conditional` attribute - [x] Unknown Operator - [x] Function Operator - [x] TranscludeWidget updates - [x] Format:Json Operator - [x] JSON Operators Docs - [x] Visible Transclusion How To - [x] LetWidget update noting that dollars are now allowed - [x] GenesisWidget # Notes for documentation - [ ] What we've previously referred to as "JavaScript Macros" are actually now better thought of as "JavaScript functions", because they work more like functions than macros - [x] Note the difference between the subfilter and function operators (the former interprets the operand as the filter while the latter interprets the operand as the name of the variable containing the filter) - [ ] Note that first parameter of user defined filter operator functions cannot be missing, because the empty double square brackets are required by the operator syntax (in other words, the operator syntax doesn't permit operators to be called with zero operands, just one or more) - [ ] Note that macros, procedures, widgets and functions are all variables, and all share the same namespace. Definitions with the same name will overwrite one another - [x] Fill widget $name attribute must be specified as a literal string, not a transcluded attribute # Other tickets that can be closed when this is merged - #3750

The plan is:

complete the documentation
publish a build of the PR to tiddlyhost to make it easier for the community to review it
release v5.2.4 and merge the PR as v5.3.0-prerelease

Mohammad · September 27, 2022, 4:01am

I see Parameterized Transclusion adds a lot of new features and open new windows for scripting in Tiddlywiki

Is 5.3.0 still full backward compatible?
Is there any plan to remove some old/inefficient features?

TW_Tones · September 27, 2022, 4:27am

I admit I am concerned about the “Parameterised transclusions” being challenging for most users to make use of, I am no genius but do know my way around much of tiddlywiki, and I find “Parameterised transclusions” difficult. Perhaps this is due to the brevity used to describe them in GitHub, but there are new terms, selected because they need to differentiate between current terms but are close synonyms of current terms. There are also new conceptual models and even ways of using syntax that is illegal or inoperative in current tiddlywiki.

I don’t doubt the power in the new features and it is knocking down long standing barriers many of which I have voiced over the years, but I am genuinely concerned its going to be hard to learn, to teach and likely to generate code novices just can’t understand at all, and may scare people.

The brevity and collaboration between developers during the development of this solution was not easy for even a super user to participate in, both technically and emotionally. We must acknowledge, in many ways there is no “user acceptance testing” in the current upgrade process, at least for every day users.

Constructively, if we want this to proceed graciously we really need develop good documentation, in plain language, and even train the key influencers in talk.tiddlywiki so they too can help introduce these super revolutionary changes to tiddlywiki.

It takes time to develop the language and comparisons between current syntax and new syntax to help people to deal with change and decide when or why the new methods / syntax is in use.

Mark_S · September 27, 2022, 3:09pm

The user defined widgets seem almost identical to procedures. What am I missing?

TiddlyTweeter · September 27, 2022, 4:43pm

TBH, I thought it a remarkable bit of writing. Compact but brilliant …

… starts it.

TT

TW_Tones · September 28, 2022, 3:06am

Perhaps here or in additional threads we could start a discussion about key elements of these changes so the community gets a heads up on this substantial development of tiddlywiki features. First I propose a discussion then we can create some wiki posts for a collated explanation.

What do you think - I already have some content.

pmario · September 28, 2022, 5:51am

As it stands now they are optional. So nobody is forced to use them.

TW_Tones · September 28, 2022, 6:07am

That is importiant but my concern is if anyone wants to use it we need it well documented.

jeremyruston · October 2, 2022, 6:14pm

Yes, v5.3.0 is fully backwards compatible, and no, there are no plans to remove/deprecate older features at this point. That’s something we’ll come back to in a subsequent release.

@TW_Tones I think you mean “development process” rather than “upgrade process”?

The parameterised transclusion work is not yet complete, in particular the documentation is not complete. The work on these features is taking place on GitHub which I appreciate is not accessible to everyone. But you can at least see that there are a number of community members participating in the thread who do have a working understanding of the changes.

User defined widgets are effectively a syntactic sugar for procedures; the value they provide is that they make it much easier to pass structured content (ie the content of the widget doesn’t need to be quoted), and that they allow the core widgets to be overridden.

TW_Tones · October 2, 2022, 11:06pm

Yes, TiddlyWiki Release and features Development, that is reflected in upgrades to the core.

My comments here are not a criticism or complaint of anyone or anything just a desire to improve the quality of the outcomes, and support the community in which I participate.

I agree, but this is a necessary minimum. Also “The current community members participating” are self selecting and only those with a Github account and particular skill sets, not very representative of the whole community. I have contributed to this process and can see changes that reflect my comments, even if not acknowledged as such, but even now it’s harder for me to contribute.

Perhaps in the future we may find ways to engage a larger audience before large irreversible changes take place in the core.

A few of us, myself included, deal on a daily or weekly basis, communicating how to use TiddlyWiki with a broad audience of different skills levels. Some changes cause either more time and effort to be demanded of community volunteer’s, or causes them not respond or support because they don’t have the time available.

In many other cases we insist that a plugin be released before changes to the core, this makes something reversable? I expect its not likely but what if;

“Parametrised transclusions” first came out as a Plugin, even if it must overwrite core tiddlers, allowing to it mature before it was committed to the core and thus backwards compatibility obligations?.
This would give a wider audience a chance to test, experience and contribute.

jeremyruston · October 3, 2022, 8:56am

GitHub PRs like this is where we do development work collaboratively. Everything we discuss there is based around the code that makes up TW (anything that doesn’t touch the code gets discussed here). So there is a knowledge threshold for useful participation in those threads.

It’s hard to see a way around that. It would be impossibly inefficient to discuss everything in the simplified terms that would make sense to all end users. I think we have to accept that the coding process is not conducive to broader involvement.

I don’t see “irreversible” here. Everything is provisional until release.

The prerelease process is very much oriented for the needs of end users, and is the opportunity for broader discussion within the community.

That would make the development enormously more complicated. These are fundamental changes to TiddlyWiki’s core code.

Can you give examples of where that has happened?

The essence of open source software is continuous improvement. I can see that it can cause anxiety for end users because it can be disorientating to encounter change, but what’s the alternative? Software rusts over time, without those continuous improvements it dies.

TW_Tones · October 3, 2022, 10:58pm

Jeremy, thanks for your comprehensive response.

Some simple responses;

I never suggested this, but I do believe we need to get more interaction and involvement.

Perhaps it is not but “addressing user needs, understanding the gaps they see and the areas they have difficulty understanding” is essential in any solution that is intended to satisfy the needs of a broad audience. I am not saying how, not asking current systems to change, I am pointing out I think there is an unnecessarily large gap between these two, and as a community I think we should seek to improve in this area.

Perhaps, but we tend to finally see everything in pre-release just before a new version release. The documentation which is essential to being able to evaluate a new feature is usually the last thing to arrive, meaning It is hard for others to even play with a new feature let alone contribute. We could have more engagement here.

If that is the case then that is the case, but the point is about engagement and contributions from a broader audience when posible.

There are a range of subject areas where users are often asking for help in the GG in the past and here. You only have to look at where we tend to spend most of the time, perhaps some illustrative examples are

The MAP and REDUCE filters which I don’t reject but then there may have being other more accessible words we could use.
The menu bar plugin is not as strait forward to use and customise as it could have being and I speak from the perspective of having developed a number of menu solutions.

You clearly misunderstand what I was saying to respond this way, It is not about change itself, or coping with change, but engaging people as broadly in the change process as practical. The more perspectives, ideas, input and criticism’s typically the better the results and I am pointing out where this is somewhat limited.

Please understand this is about constructive observations that I think need to be acknowledge and incorporated in future thinking. Sure we may dispute the degrees and examples but the points remain valid.

Yours sincerely, Tony

TW_Tones · October 3, 2022, 11:00pm

In the act of responding here one idea that may help that comes to mind is separating releases containing fixes and those that introduce features so fixes arrive quicker and features get more input and documentation before release.

I also think this could help make new features even better, more innovative and valuable.
An example may be a seperate demonstration wiki with Parameterised Transclusions before release.

saqimtiaz · October 4, 2022, 4:53am

I think it important to highlight here that:

the above next steps indicate that the new features under discussion are at least two TW releases away which in all likelihood means 6+ months given past release cycles.
despite how far the new features are from release, the reference documentation is already close to being complete thanks to Jeremy putting in countless hours on this front throughout the development of this PR to allow others to engage with it. Writing extensive documentation this early in the development process always leads to a high overhead in terms of having to redo it, sometimes almost entirely, as the code changes and matures.
the release plan provided above indicates a longer than usual pre-release period for v5.3.0 and that the intention is to potentially publish a build of the PR as a TiddlyWiki everyone can experiment with and explore even in advance of that with complete reference documentation, giving ample opportunity for community engagement and feedback.
the changes under discussion are completely backwards compatible and that existing code and knowledge of how to use it will still suffice if it does so today.

I also find this previous comment by Jeremy relevant here in terms of what is and is not realistic in terms of both development and user engagement for a small community like ours with limited resources. If anything I believe the end user engagement by TiddlyWiki developers is significantly higher than you will find in most open source communities.

Could we simplify the PR process?

The PR-based development process is almost universal amongst open source projects. That’s really crucial: we’re not in the business of inventing new ways for open source projects to run. A small community like ours can only exist by riding on the shoulders of all the other open source projects, and doing things in broadly the same way. The sharing philosophy that we see within our own community also exists at a higher level, with communities of communities sharing tools and processes.

So, I think it’s important to cast this not as a problem that is unique to our community, but one that is shared with other projects. That’s important because we have little choice but to build our own infrastructure for things that are unique to our project. But it’s not economic for us to build infrastructure for problems that are universal to all (or most) projects.

That means that to solve this I think we need to look outside of our own project, and explore what solutions other projects use.