Parameterised Transclusion -- Macros vs Procedures and Functions

Developers
, , , ,
#1

Introduction

Since the release of the first beta at 15th December 2013 and the first version 5.1.0 of TiddlyWiki5 at 20th September 2014 we have gone a long way.

These examples should show the difference between the current macro implementation designed back then and upcoming features procedures and functions.

Dynamic URL Example

(All of the code snippets in the first post have issues because of the current implementation details.)

A very common usecase is the dynamic creation of URLs, where our users always have problems and the naive implementation shows several technical problems

Code with Current Macro Definition

This macro definition uses the simple text substitution mechanism.

\define dynURL(protocol:"https", host, path, tiddler) $protocol$://$host$/$path$#$tiddler$

Eg: create a dynamic link to the HelloThere tiddler at tiddlywiki.com

<<dynURL host:"tiddlywiki.com" tiddler:"HelloThere">> -> Works OK

image

First Technical Problem

Now let’s say we want to create a list of clickable URLs for tiddlers tagged HelloThere

<$list filter="[tag[HelloThere]]" /> gives us the list.

The intuitive code for the dynamic URL list would look like this:

<$list filter="[tag[HelloThere]]" variable="tiddler">
<$macrocall $name="dynURL" host="tiddlywiki.com" tiddler=<<tiddler>> /><br>
</$list>

BUT … as you can see it breaks down because of the spaces in the titles

image

First attempt … to fix that problem we introduce triple double quotes

The macro definition would need to look like this

\define dynURL(protocol:"https", host, path, tiddler) """$protocol$"""://"""$host$"""/"""$path$"""#"""$tiddler$"""

BUT that does NOT work.

Second attempt use <<__var__>> for macro parameters

\define dynURL(protocol:"https", host, path, tiddler) <<__protocol__>>://<<__host__>><<__path__>>#<<__tiddler__>>

BUT that does NOT work either

Third attempt … use the transcluded filter trick {{{ [<__var1__>][<__var2__>] +[join[]] }}}

\define dynURL(protocol:"https", host, path, tiddler) {{{ [<__protocol__>][[://]][<__host__>][<__path__>][[#]][<__tiddler__>] +[join[]] }}}

Looks promising BUT does not work

WTF … It should be simple … Go to the forum and tell them they are @"!§$%&

4 Likes
Parameterised Transclusion Introduction
#2

Working Exampes

The following code snippets work with the Parameterised Transclusion Preview wiki

Dynamic URL Example

A very common usecase is the dynamic creation of URLs, where our users always have problems but it should be straight forward.

String Concatenation with Function Definition

\function dynURL(protocol:"https", host, path, tiddler) [<protocol>] [[://]] [<host>] [[/]] [<path>] [[#]] [<tiddler>] +[join[]]

Eg: create a dynamic link to the HelloThere tiddler at tiddlywiki.com

<<dynURL host:"tiddlywiki.com" tiddler:"HelloThere">> -> Creates a concatenated string

image
image693Ă—41 2.05 KB

But it should be an external link.

  • We need to create external links using the A HTML tag
  • The parameters are moved from the \function f-URL() to the \procedure extURL(...)
    • Avoids a lot of parameter handling
\function f-URL() [<protocol>] [[://]] [<host>] [[/]] [<path>] [[#]] [<tiddler>] +[join[]]

\procedure extURL(protocol:"https", host, path, tiddler)
<a class="tc-tiddlylink-external" href=<<f-URL>> rel="noopener noreferrer" target="_blank"><<f-URL>></a>
\end

<<extURL host:"tiddlywiki.com" tiddler:"HelloThere">>

image

Example List of Links

<$list filter="[tag[HelloThere]]" variable="tiddler">
<$macrocall $name="extURL" host="tiddlywiki.com" tiddler=<<tiddler>> /><br>
</$list>

image

Bonus

The new \procedure pragmas give us the possibility to have consistent references to our variables.

Make Everything Dynamic

<!-- Comment: at the top of the tiddler add 
\procedure myHost() tiddlywiki.com
\procedure filter() [tag[HelloThere]]
-->

<$list filter=<<filter>> variable="tiddler">
<$macrocall $name="extURL" host=<<myHost>> tiddler=<<tiddler>> /><br>
</$list>

image

Conclusion

Only variable syntax is needed!

  • No tripple double quotes """$var$"""
  • No underlined variables <<__var__>>
  • No filtered transclusion trick {{{ [<__var__>] }}}
2 Likes
#3

Bonus Bonus

We will be able to create our own widgets.

Make extURL a Widget

The cool thing is – It will not clash with the extURL procedure, because it has a different name.

Definitions

\procedure globalHost() tiddlywiki.com
\procedure globalFilter() [tag[HelloThere]]

\function f-URL() [<protocol>][[://]][<host>][<path>][[#]][<tiddler>] +[join[]]

\widget $$extURL(protocol:"https", host, path, tiddler)
<a class="tc-tiddlylink-external" href=<<f-URL>> rel="noopener noreferrer" target="_blank"><<f-URL>></a>
\end

Usage

<$$extURL host=<<globalHost>> tiddler="GettingStarted" />

image

<$list filter=<<globalFilter>> variable="tiddler">
<$$extURL host=<<globalHost>> tiddler=<<tiddler>> /><br>
</$list>

Copy paste it to the TW preview and it should work.

Have fun!
Mario

For those, who prefer copy / paste :wink:
macros-vs-procedures-and-functions.json (7.4 KB)

2 Likes
#4

In current TW I would use this:

<$list filter="[tag[HelloThere]]" variable="tiddler">
<$let url={{{ [[https://tiddlywiki.com/#]addsuffix<tiddler>] }}}>
<a class="tc-tiddlylink-external" rel="noopener noreferrer" target="_blank" href=<<url>>><$text text=<<url>> /></a><br>
</$let>
</$list>

It doesn’t use a macro to concatenate the URL (I don’t know if this was a stipulation for your example), but it also avoids the text substitution that we’ve been tought is the red-hot hopping devil…

If you’d absolutely want some macro definition in there, one could outsource the filtering to a macro-defined subfilter. I hardcoded the URL path above, but it may be equally simply constructed with variables inside the filtered transclusion.

Don’t get me wrong, I’m absolutely looking forward to the coming functionality. I’m already a big fan of its herald $genesis. And I also know that finding simple examples for these complex new mechanisms is not easy. So, while there may be ways to solve this particular problem with current functionality, it is an absolute pain in the sphincter to use macros with variable parameters as an attribute to a widget:
$macrocall doesn’t work there, and variables cannot be used as parameters inside <<invocations>>.

You’ll almost always have to construct a $let around it like above. Once one gets his head around the new pragmas, this should become relatively straightforward.

1 Like
#5

Sure, we could solve those problems in the past, otherwise it wouldn’t have been possible to dynamically create and concatenate URLs but it should be more consistent now.

I needed an example that is easy to understand, somewhat useful and still shows the problems we faced from user feedback. There are plenty of help requests here in the forum, that look similar to the first post. … So I used them for this thread.

I do have other examples that use $reveal-widgets and show the challenges to create dynamic but predictable state tiddlers. … Those examples are way to complex to be useful as “old vs new” …

They may be useful for the future docs to describe best practices how to use “procedures” and “functions”, where we do not need to bother with the “old code”

#6

Isn’t the use of function a filter transclusion trick in disguise? To me it doesn’t look at all intuitive that string concatenation should be with a filter. In most languages the way to do this is to create a local variable which uses string interpolation to build the URL and then use it when building the link which again can be done with string interpolation.

#7

Thanks for this “expose” @pmario and a great practical example as well

  • Perhaps later when we write documentation start with the new way and place the history after.
  • I would personally also set the target to the host, so multiple links open in the same tab/window. In most browser r-click allows to to choose “_blank” if you want.
  • I understand there are good arguments to avoid substitution, but there are also good argument’s to still use it in particular cases.
    • As impressive these examples are, they are they are much more verbose than a simple substitution.
  • I feel we should explain these performance issues further and consider if it is, always valid to throw the “substitution method” away in all cases.
  • Since I personally never wrote something using substitution that caused a noticeable performance issue, it would “appear” not to be justified.
  • Perhaps rewriting the way substitutions work to resolve performance issues would make more sense than forcing the way the user has to do it.
    • Or perhaps a new substitution method for the simple cases so a more concise method is available.

I also agree;

and

  • The more seperate definitions you need to accomplish something, the higher the “cognitive load”, and harder to read, the less understandable and maintainable it is.
  • As cool as being able to make a widget is, in Marios example, you need 4 definitions even before you call the widget.
  • Even if it is not a trick, filters are tantamount to regular expressions (to learn), I know filters, I can read them and its easy, but others will not find this so simple.
  • I use the triple curly braces a lot, “Filtered Transclusions”, using filters to construct output, so it is not “too long a shot” to make use of filters this way. but I do “feel” for new and potential users.
#8

Yours is the first mention of “performance issues” in this thread. What do you mean by it?

#9

Perhaps performance is not the best word, it’s not explicitly mentioned here but it is part of the reason for this new code, that substitutions in macros are apparently less “performant”.

#10

There primary issue with textual substitution in macros is that it is brittle. The classic symptom is the long string of bugs that we had around the presence of double quotes within tiddler titles. We fixed it by introducing the “double underscore” notation for accessing macro parameters as variables. But the fix comes at the cost of more complexity for users, who need to understand and differentiate between two different ways of accessing parameters.

There are performance considerations, but they are marginal and pale into insignificance behind the kind of intermittent, content-sensitive bugs that textual substitution gives rise to.

I’d urge you to look again at my post at the top of the original PR which makes all these same points.

#11

I did understand the previous points. For much of my own code these were rarely a problem. However the main problem has always being when we are forced to wikify a macro to make use of its results. I don’t think that is an issue with simple substitution.

  • otherwise using $param$ $(vaname)$ in a whole suite of common use cases is easy and reliable unless the parameters are complex in some way.
  • For this reason I can see a subset of these cases remaining quite valid.

Don’t get me wrong, I appreciate the advances proposed for the other cases including more sophisticated parameter and content handling.

So I conclude from your list of symptoms there are no other compelling reasons to avoid simple substitution with macros going forward?

#12

@TW_Tones

I’m not sure what you mean. I pointed out the problems with textual substitution. Those are the compelling reasons for avoiding textual substitution with macros in many situations.

Your position seems to be to deny that there is ever anything wrong with textual substitution. The development history of TW5 suggests otherwise. Textual substitution has caused significant bugs in the core. The double underscore thing was a hack to get around that problem, and procedures are a properly engineered solution.

Again, I don’t quite understand. Nobody is saying that there aren’t valid uses for textual substitution, just that there are significant dangers in many situations.

#13

With respect I don’t say that at all, that is a serious exaggeration. I get fearful when I question the “received wisdom” around here.

  • More often than not I am only reflecting what others have recently said, that is, a perspective evident in the forums by the nature of discussions and questions that arise.
  • Well that is good, it has definitely appeared otherwise, not only in your words but others. This is exactly what I am seeking clarification on.

When it comes to the broader community we need to clarify these things. I am dependant on yours and others perspective’s to throw light on issues behind the obvious, issues that exist that are not visible.

  • This is particularly important when undertaking substantial change.

You must be aware that I help a lot of people on the forums, I see get to see when there is discrepancies in public statements and how tiddlywiki can be used. The more I can clear up the better.

I don’t want to spend unnecessary time needing to explain the new methods when the existing ones are adequate, or clearing up misunderstandings as a result of other peoples statements, gaps in the documentation or the best reason to use a new feature.

  • In a similar way to how I depend on others for the hidden, it would be wise to consider my view point as reflecting how these things appear to a broader audience.
  • Also I have developed a deep understanding of TiddlyWiki from a user/super user perspective, a perspective that is not always easy for someone with deep core knowledge to put themselves in.

I have done my best here to describe things from my perspective. We all have different perspectives and need to respect those differences. Please respect mine.

P.S. I may be offline for a lot of this weekend as I am celebrating my birthday

#14

Congratulations, I am sure everyone will join me in wishing you a happy birthday, and an enjoyable celebration.

There’s no lack of respect, but there is an expectation that those taking part in these discussions will do the groundwork of reading and understanding what I have already written on the subject. Lack of engagement with the substance of those writings makes it very difficult for me to respond constructively.

You don’t use those words, but you have consistently questioned the view that there are problems with textual substitution. For example, you said the following:

That’s a very direct statement that you do not believe that the parameterised transclusion work is justified because you personally have never run into problems with textual substitution. That’s consistent with my summary that you deny there is anything wrong with textual substitution.

As far as my own words are concerned, that is not true.

There’s a section in the #6666 PR that says this:

The new transclusion architecture is not by itself sufficient to enable us to fully deprecate macros yet. To handle the remaining use cases we propose a new backtick quoted attribute format that allows for the substitution of variable values. See #6663 for details.

#15

I need to respond hurriedly, as I am off, but please understand I would not reply about details, without reading the details, possibly multiple times. My suspicions are correct the problems with substitution is mostly functional not performance.

Simply you are wrong in your assessment of my words. I said;

  • And you agree with me.
  • This is about a future where you deprecate macros, you may see this “future” as explaining the present, but this requires your perspective.
  • This is somewhat distant to my questions which was trying to simply establish which aspects of the current macros remain the best way to do things and which we should migrate to the new solutions the moment they are available.
  • I now have more clarity and happy with my current understanding.
  • But I am not happy with the way my words are being misinterpreted or miss represented. I am happy for my words to be questioned, requests for clarification etc… even disagreed with, no ones language is perfect. But questioning if I have read the thread or documentation, or I have something against these changes other than a critical eye, is hurtful.

Have a nice week end. :nerd_face:

#16

While the brittleness of textual substitution is far more critical an issue than performance considerations, that does not mean the performance impact cannot be substantial in complex and recursive code where the same macros are called repeatedly times.

For instance, the early versions of Streams really struggled with slow rendering with a large number of nodes in a single tiddler such as the one that imported Dave’s dynalist collection of TW links. Removing macros with textual substitution alone increased the render time of such streams by 2.5-4 times depending on the hardware.

Overall the problem with textual substitution as implemented in macros is that it works well most of the time, until it doesn’t. The goal for the TW core is to move towards primitives and mechanisms that can be relied on to be robust every time. Part of that will likely involve providing an alternative and more robust means for textual substitution or string concatenation.