"Custom widgets" vs "Procedures"

Developers
#1

With custom widgets, do we need procedures (and macros) at all? Is there anything CWs can not do that Ps can?

I must be missing something obvious because the only downsides I can find with CWs vs Ps are syntactic:

  • C.widget names need attention to not overwrite core widgets (include a dot .)
  • Invoking <$a.a.../> is uglier and less smooth than invoking <<aa...>> and only the latter allows for conveniently throwing in unnamed parameter values.

What am I missing? Why are procedures (and macros) needed?

IF CWs indeed can do everything that Ps can, then for TWX I wish the latter will be omitted and the former should perhaps reuse the smoother syntax from the latter.

Confusing discussion about backwards compatibility
#2

The main building block in TW are widgets.

Widgets came first and then macros have been implemented, later in the process. We are talking about TW versions prior to 5.0.0-alpha.11, which is the first one published.

Macros have always been syntactical sugar for the <$macrocall widget. Which means, for the TW core internally everything is converted into widgets.

Syntactic sugar

In computer science, syntactic sugar is syntax within a programming language that is designed to make things easier to read or to express. It makes the language “sweeter” for human use: things can be expressed more clearly, more concisely, or in an alternative style that some may prefer. Syntactic sugar is usually a shorthand for a common operation that could also be expressed in an alternate, more verbose, form: The programmer has a choice of whether to use the shorter form or the longer form, but will usually use the shorter form since it is shorter and easier to type and read.

Wikipedia

With TW v5.3.0 macros have been superseded by

  • procedures
  • functions
  • custom-widgets

TW docs Custom Widgets

[New in v5.3.0] A custom widget is a special kind of procedure that can be called using the same syntax as widgets.

  • It allows users to create their own wikitext widgets which follow the core widget refresh cycle
  • It allows users to overwrite core widgets.

TW docs Procedures

[New in v5.3.0] A procedure is a named snippet of text. They are typically defined with the Pragma: \procedure:

  • Procedures basically replace macros.
  • They are more robust and most of the times more performant.

TW docs Functions

New in v5.3.0 A function is a named snippet of text containing a Filter Expression. Functions can have named parameters which are available within the function as variables.

  • Functions are a new mechanism to work with filter expressions.
  • Filter expressions can be used like variables

Conclusion

Procedures and Functions are still syntactical sugar. But now for the transclude-widget, which internally is used to “display” things.

Custom-Widgets are also “syntactical sugar” – “not sweet”, because they looks like a widgets.

In the end all of them are converted into transclude-widgets.

Hope that helps
Mario

#3

There is nothing wrong, if users “wrap” custom widgets within procedures. But we will never get rid of widgets and the widgets syntax. Widgets are the main building block. For consistency widgets will need to be called in the same way.

The macro parameter syntax may be “relaxed” in the next version v5.4.0. So macro parameters can use x=0 and x:0 in the same way as widgets.

\widget $my.widget(attribute:"Default value")
This is the widget, and the attribute is <<attribute>>.
\end $my.widget

\procedure test()
<$my.widget/>
\end

<<test>>

Which gives you this.

image
image1506×453 25.6 KB

#4

Thank you @pmario - appreciated!

So the advantages with procedures are, in deed, limited to syntax. But then, are there any drawbacks with procedures?

Regarding…

But we will never get rid of widgets and the widgets syntax. Widgets are the main building block. For consistency widgets will need to be called in the same way.

…I was asking about TWX so I don’t think consistency is a criteria (compare TWC vs TW5) (but, obviously, this is up to Big J or other superhuman that implements it :wink: My point is merely that IF things are fully started over then why not use the more elegant syntax if it works as well.

#5

Because there are different usecases for abstractions and verbose code.

“Less verbose” syntax works until it fails. It’s a bit of a chicken / egg problem. We will always need a verbose syntax, that has access to most of the internal JavaScript functions.

Later in the process “abstractions” with “less parameters” can be implemented. But we still need to rely on the basic building blocks, that have “all the parameters”.

Think about the tabs macro. Which is a “heavy abstraction” using “views” and “buttons”.

  • v5.1.0 tabs macro.
    • It has 5 parameters. tabList, default, state, class, template
  • v5.3.6 tabs macro
    • It has 9 parameters: tabList, default, state, class, template, buttonTemplate, retain, actions, expliciteState

Using the <<tabs ...>> macro gives us a “nicer” interface for complex code. So abstractions hide complexity

But over time we may find out, that it is not good enough. We need more parameters. eg: buttonTemplate, which is not there in v5.1.0. (That’s why I wrote. “less verbose” syntax works until it fails)

The advantage of the “tabs abstraction” is, that we can keep the first 5 parameters in place and add a 6th one, without breaking existing code.

So there is a different usecase for procedures and widgets. Procedures define “abstractions” that can make the users live easier. But they will also limit them, to the function provided by the abstraction.

I hope that makes sense
-m

#6

Great point! Thank you @pmario !

I’m drifting from the OP and turning somewhat philosophical but since we’re talking about TWX and the future:

For a complete TW rewrite, maybe it’d be worth questioning things from first principles. I mean, we currently have these abstraction layers:

    1. procedures
    1. widgets
    1. js

So, presumably there’s a verbose layer in js - so maybe that middle layer can be cut omitted?

My interpretation is that we, historically, intended for the procedure layer (at that time only “macros”) to be the primary user-facing layer, perhaps leaving widgets to developers. But in practice it was not so clear cut and beginners have had to confront widgets very early, even if occasionally, and average+advanced users need to jump up and down between layers regularly.

My experience is that to make very advanced stuff or integrations in TW one typically needs to know js anyway. I.e we can’t totally abstract away the js layer anyway, so maybe it can be the layer for all the verbosity?

#7

Nope. – Security, backwards compatibility and JS feature parity.

Even if TWX is not compatible with TW5.x.y, it will start its own backwards compatibility-chain.

The elephant in the room: Security. If we would allow users to directly use JavaScript like wikitext, we would need to parse and sanitise it, to avoid misuse. That’s slow and super complex.

JS feature parity: The JS language specification advances every year. So in 2025 there is a specification ES2025, which defines the new features, allowed in JS. Browsers implement these specs in their own pace. That’s why we need pages like caniuse.com to see what’s possible, to be used.

For the TW v5.4.0 core we defined our target to be ES2017. So if users would want to use newer features, which they will, we would need to implement a transpiler, to make their code compatible. — We will not want to do that. ** no – never ever - eiks **

So what you suggest is:

    1. procedures
    1. sanitised (for security reasons) and transpiled JS
    1. JS

Where no. 2 is a 1000 times more complex than handling widgets / wikitext.

You are right TW widgets are an abstraction. Widgets look very similar to HTML elements. So if someone knows HTML they should be somewhat familiar with the syntax.

TW wikitext, which contains plain text, HTML elements and widgets, is a so-called DSL (Domain specific language) – If you insert text or HTML code into your tiddler, we parse, sanitise and then prepare it for the browser, so it can “draw” it.

→ The only purpose of TW wikitext is to allow ordinary users, to create interactive hypertexts, without the need to know HTML, CSS and JS. If they do know it, that’s a plus but it is no requirement to start.

It can become a necessity if the users want to modify the appearance of TW. …

    1. procedures
    1. widgets
    1. js

Is an extremely simplified view.

Procedures and widgets are TW wikitext. To transform TW wikitext into html-code, that the browser can handle there are many more steps. All of them are some kind of “abstractions” for different usecases.

eg: This is ''bold'' text

A) For the user looks like this:

image
image1472×142 6.43 KB

B) For the browser it looks like this

image
image1467×176 7.42 KB

To be able to create the code for the browser we need:

C) A Parser - 1st step it looks like this:

We call it: “parse-tree”. The parse tree is mainly needed for performance optimisations. Since parsing wikitext is the most time consuming operation, the parse-tree is cached. It is only recreated, if a tiddler is changed. It is used to find “back-links”, “back-transclusions” and some other stuff.

It “describes” the “structure” of the text in a unified syntax, that we can use for the next step D)

image
image1513×811 11.3 KB

D) parse-tree is converted to the widget-tree

If you have a closer look at the widget-tree, you can see, that it mainly is a description of HTML elements to be used, so the browser can make sense of our text.

image
image1531×678 9.54 KB

E) Renderer

The widgets render() functions converts the widget-tree into HTML elements, so the browser can make sense of it.

How does all of this correlate with JavaScript. → Javascript it the general purpose language the browser understands. It is used to manipulate and manage our TiddlyWiki’s.

And here is the point, where the core JavaScript journey starts.

More precisely:

Looks simple - right :wink: – But that’s a completely different story.

#8

You are right. But I think that’s a resource problem. We do have 35 core macros, where half of them are directly useful for end-users. Like:

TOC, some tag-related, some tab-related, copy-to-clipboard and a view others. They where created because everyone needs them.

If we would have 20 full time core programmers, we would probably have a lot more core macros for many more specific usecases. - But - We do not have the resources.

What we have, are some talented community plugin authors, like yourself, that provide the community with plugins of all kinds :wink:

1 Like
#9

Your posts are highly enlightening - brilliant stuff @pmario

OK, so, what can be done to not “force” the users to jump between abstraction layers i.e between widgets and procedures?

Or, actually, in what way do widgets have to be verbose? What exactly are you saying is verbose in them? Contrasting procedures with widgets I note these:

  • the <$ vs << characters: I’d think the former can be exchanged for the latter if we’re starting over.
  • the <$foo> encapsulation </$>vs <<foo "encapsulation">>: Again, syntactic sugar, I’m guessing either could be implemented
  • invocations; must a widget invocation specify all param values? I’d guess not.

…so why would a (rewritten) widget construct have to be any more verbose than procedures? Put another way; instead of removing the widget layer in my list we remove the procedures layer - yes, doable?

#10

The main difference to procedures is, that widgets have to have named parameters. There are no “position” related params.

No. Usually all widgets have sensible default values. eg: <$link/> or <$transclude/> work just fine, if the <<currentTiddler>> variable is defined in the right way.

This may be possible, but we will get naming conflicts between core widgets and user defined procedures. If something starts with <$ it is immediately obvious, that it is a widget.

  • eg: The core would not be allowed to use <<button>> as a widget name, because there is a 100% chance that users would name a custom widget or a procedure <<button>>. So all core widgets would have to have a unique and longer name :confused:

  • For the core we try to use names that have a lower chance to cause naming conflicts.

    • eg: <<list-links macro or <<list-links-draggable … So <<list and <<list-draggable are free, for users

Overloading the same start- and end-patterns, creates its own type of problems. Namely “variable naming” problems. eg:

  • Can you tell me if <<toc-getTooltip>> is a function or a procedure?

You can’t. So I have to name it <<tf.toc-getTooltip>> where the tf. prefix stands for “tiddlywiki function”. Similar to tc- for tiddlywiki-class.

User function can be named eg: <<f.something>> or my prefix is wltf.something … wl - WikiLabs

So having the same start- and end-patterns cause their own problems, for “non trivial” procedures.

That’s more a technological than a syntax problem.

<$list> ... </$list> is a block pattern, where the parser sees the first <$list> pattern and searches for the last </$list> pattern. Like for HTML elements.

It takes the content between and parses again and again … until it found all the elements that builds the parse-tree.

The angle-brackets are inline <<inline delimiters>> used for variables.

So the parser sees the first opening << brackets and searches for the first closing brackets >>. It takes the content in between and sends it to the “procedure” parser.

This is the reason why <<test <<asdf>> >> is not possible.
It has to be inline because we need <$list filter=<<test>> variable=<<aaa>> > ... to work.

1 Like
#11

But that would not necessarily be so if what a widget is is totally redefined from scratch, right? I mean, I must presume Jeremy defined what a widget is by writing a JS code for it which he (or anyone) could redefine for TWX… right?

…try to avoid naming conflicts…

I can imagine several decent ways to deal with that, e.g core widgets could hold special status so they cannot be overwritten (e.g special tag), i.e they are always prioritized unless user does “special thing” (e.g \proc button|yesoverwritedammit!()) What does JS, and other coding languages, do to protect reserved words? Surely it can be considered a solved problem.

As for…

<<list-links macro or <<list-links-draggable … So <<list and <<list-draggable are free, for users

I did not realize that was the intention (which also explains why I always need to test if it was list-links or link-list or links-list etc :sweat_smile: )

Thanks for your informative input!

#12

As an end-user, I hate this suggestion! :stuck_out_tongue: I think of widgets as building blocks providing the underlying functionality, whereas macros/procedures are largely shortcuts — many of which also contain one (or many!) widgets. I try to convert any bit of code I’m likely to reuse into a procedure, so I use the <<... syntax all the time, and I find that bit of syntactic sugar hugely helpful for identifying, at a glance, that a bit of code is a reference to a procedure defined elsewhere, not a purely local construction. I also really appreciate that the shortcut syntax lets me omit the parameter names; I’ll sometimes define a relatively simple widget as a procedure just so I can use the << syntax to reference it. I’m really looking forward to the “relaxed” parameter syntax of 5.4.0; I’m hopeful it will let me eliminate a lot of wordy $macrocall widgets.

On the flip side, though I use the core widgets all the time and have redefined a core widget or two (like $link), I’ve yet to find a single scenario in which it made more sense to use a fully custom widget than a procedure…

1 Like
#13

Yeah, it’d take some getting used to but I think that it’d greatly simplify learning TW which probably is necessary if we want more users. The steep learning curve is a recurring complaint.

Obviously, we’re talking hypotheticals, but if my idea was to be implemented you could still interpret a distinction; instead of “widgets vs procedures” you’d have “core widgets vs user defined widgets”.

Besides… in another thread there was some reference to the year 2036 for TWX… by then we’ll be in a very different world me thinks…

#14

I think it may be importiant to point out that most widgets are where content goes to be displayed, although one widget can contain other widgets as a rule, we tend not to use widgets as inputs to something (lest one use $wikify). The exceptions being things like edit text that update a tiddler rather than the flow of TiddlyWiki Script.

  • So I use custom widgets to display or generate interactive components.

Procedures, functions and variables behave more like typical programing or macro languages. However I think functions point to where we should move towards because they “evaluate” before rendering thus provide greater value in coding activity relating to tiddlers.

  • See also the new back tick parameters
  • There is a hack around this with custom widgets and $parameters widget. But as a rule yes.

Also;

  • $macrocall can still be used for functions except where it does not support new features.
  • I would like to see the ability for meta programing such as building variables full of code then being able to change into and use as a procedure. A little more towards a Quine :nerd_face:
#15

So true!

I hope someone is saving this discussion beyond the forum so it doesn’t disappear in it’s prolix volumes.

@pmario’s comments taken together are a superb presentation/intro of the most important activities of TW’s Backroomboys.

1 Like
#16

What a fabulous thread! :heart:

But, going back to the OP @twMat …

Widgets, unlike procedures, support encapsulation at the point of use via slots.

<$my.widget …>
 All manner of shenanigans (html, wikitext) not known at declaration time.
</$my.widget>

You can’t do that with any other mechanism, procedures or otherwise. Perhaps (though I find it highly unlikely), TWX could offer slots via…

<<my-proc … >>
 Stuff
<</my-proc>>

#17

Ah, thanx @CodaCoder , excellent point! That is a critical difference and a superior advantage with widgets over procedures.

2 Likes
#18

Also…

\define do(a:not, b:forget) $a$ $b$ $(this)$

#19

Or

\procedure do(a:not, b:forget) <<a>> <<b>> <<this>>
#20

Having recently discovered web components - I’d suggest we think whether we can do away with <$widget> in favour of just <widget> rather than just move to <<… Its all about the parsers and the element widget that sorts out that node in the parse tree …

Wouldn’t the solution be to establish the priority order for such collsions - such that user defined widgets take precedence over core widgets ( that maybe then take precedence over html elements )… Does’t the core do that already with <$button> syntax as a custom widget can override a core widget and then call the <$genesis> widget as needed ?

That is because the parser is fairly simplistic just searching for text using regex - there are more elegant ways to handle this using regression that is implemented in some existing parser libraries… These only return back to seek the closing brackets when the inner content has exhausted some grammar (e.g. in this case once parsing <<asdf for its matching >> then you allow <<test to match its end bracket…)

This lack of context aware parsing is a bit of a pain - tends to make some obvious code imposible to do.

:smiley: