Frustrations of Wikitext from a Software Dev

[GameDungeon]

A feeling I get a lot, “I’m a software dev. Why does wikitext of all things give me so much trouble.”

While it is wonderfully and amazingly powerful, wikitext is so painfully unintuitive. The feeling of, how on earth do I pass the current time into a macro and set the output of that to a variable. Oh, after hours of trying everything it doesn’t work because I didn’t “Wikify” it first. (Why does TW not have real functions ;( ) And even reading through grok isn’t enough to learn these edge cases. It feels like you have to have someone else teach you when you run into a new issue.

[pmario]

Sorry for the wall of text. It turned out to be much longer as I wanted it to be.

Anyway … I hope it helps

TLDR; There is a JSON attachment at the end of the post, you can use to play with.

---

@GameDungeon

(Why does TW not have real functions ;( )

For the same reasons, why browsers don’t allow to execute arbitrary code from different domains. For security reasons.

@GameDungeon

A feeling I get a lot, “I’m a software dev. Why does wikitext of all things give me so much trouble.”

Yes. Being a software developer is a bit of a problem, because we are used to think differently.

We think like:

@GameDungeon

how on earth do I pass the current time into a macro and set the output of that to a variable.

But that’s not the mindset wikitext was developed for.

So try to free your mind and try to be open. (for a bit of history

---

Wikitext is intended to convert human readable plain text into rendered HTML like so:

* element 1
* element 2
* element 3

to be seen as:

• element 1
• element 2
• element 3

---

Many users want to show this info more dynamically.

• So if there is a new tiddler the list should automagically grow.

That’s what the list-widget is for. Basically any UI element in TW uses lists in some form.

Since this is a very common usecase, a core developer created a list-links macro for this problem, which does a bit more. It also converts the elements into hyperlinks.

<<list-links filter:"[tag[myElement]]">>

So the “complexity” is completely hidden from the end-user.

---

BUT … But some users want to create such macros. … OK let’s dig a bit deeper.

For a SW developer we need a “macro specification” right?

• We want an unordered list <ul> that
  • shows users defined elements as list items <li>
  • a “user defined element” is specified by a tag
  • the tag is: myElement

So the initial implementation will look like this.

<ul>
  <$list filter="[tag[myElement]]">
    <li><<currentTiddler>></li>
  </$list>
</ul>

That’s it. … really … that is what wikitext rendering and variables like <<currenTiddler>> should do.

---

Everything that comes now adds additional possibilities to create more sophisticated lists using macros, widgets and transclusions

eg: I want to call it like so: <<myList filter:"[tag[anyTag]]">>

\define myList(filter)
<ul>
  <$list filter=<<__filter__>> >
    <li><<currentTiddler>></li>
  </$list>
</ul>
\end

<<myList filter:"[tag[myElement]]">>

Now think about it. That’s the only usecase that filters are designed for. To be executed by $list-widgets.

So which variables do we need here? … As shown in the example, we need 2 variables:

• the <<__filter__>> from a macro parameter
• the <<currentTiddler>> as the “loop variable”

<<__filter__>> comes from the macro call in the content area of the text.
(The __ double-underscore are used because of this)

<<currentTiddler>> … Is “just” a convention we (the core devs) use for “lists”

---

So what if we want to be much more flexible. … A new spec is needed.

• The code snippet <li><<currentTiddler>></li> is named: element
• The content of the element should be a template
• A developer creates a defaulListElement which is predefined
• The user should be able to redefine that template

To make the code readable and debuggable we need some more “variables” …

• The custom template should show
  • the tiddler title
  • and the description field if there is one
  • There should be a separator between title and description
  • The separator should only be there if needed

– spec end

---

New code

\define myList(filter template)
<$let elementTemplate={{{ [<__template__>is[blank]then{!!defaultListElement}else<__template__>]  }}}>
<ul>
  <$list filter=<<__filter__>> >
    <$let separator={{{ [all[current]get[description]then[--]] }}}>
      <<elementTemplate>>
    </$let>
  </$list>
</ul>
</$let>
The macro is defined at: <$link $to=<<currentTiddler>>/>
\end

<<myList filter:"[tag[myElement]]" >>

elementTemplate … is “the outer variable” only needed to be defined once.
separator … depends on the currenTiddler variable defined by the list-widget it needs to be defined every iteration

That’s how variables are defined … In a context of “lists” … In TW it’s always like this, because that’s what it was designed for.

There still are the <$set and the <$vars widget for backwards compatibility reason. And we don’t have a replacement for <$set name=xx filter="[[filter expression]]" yet.

---

This example contains the macro call a custom template as a string

<<myList filter:"[tag[myElement]]" template:"""
<li>
  <<currentTiddler>>
  <span class="tc-tiny-gap"><<separator>></span>
  <$transclude field=description/>
</li>
""">>

---

Conclusion: The wikify-widget is the last resort if no other way is possible. If $wikify is your standard way to create variables, you’ll get performance problems — You are doing it wrong

Links to docs of macros, widgets and operators used.

---

The code with the custom template will be more complex. That’s why I did add JSON file, with which interested users can play with.

It contains 3 example tiddlers and 3 tiddlers tagged myElement. I of them has a description field.

• simple-list
• filtered-list-macro
• macro-and-templates

creating-variables-and-list-mindset.json (2.1 KB)

[GameDungeon]

@pmario

For the same reasons, why browsers don’t allow to execute arbitrary code from different domains. For security reasons.

I take issue with this. I too often see arbitrary things listed under “security reasons” with no explanation.

• First, I can already execute arbitrary JS. That can do far more damage than some wikitext. It’s also my wiki, what damage would I be doing to my wiki?

• When I said functions, I literally mean a macro runs and replaces itself with a returned variable instead of its entire code.

• Js macros already do this, why can I not do it in wikitext?

• This literally has nothing to do with security.

[telumire]

I find wikitext to be a lot similar to the template feature of Angular (transclusion, list, the way events are handled …) - but I’m just starting to learn that so maybe I’m missing something

[pmario]

@GameDungeon

• When I said functions, I literally mean a macro runs and replaces itself with a returned variable instead of its entire code.

As I did show in my examples what you probably want is this construction, which we also use in the core UI code.

Pseudo code

<$let variable={{{ [<filter expression>]  }}}>

If {{{ [<filter expression>] }}} is used stand alone in a line that construction is a shortcut for

<$list filter="[<filter expression>]]"/>

If used in a filter-string it is a shortcut for

<$list filter="[<filter expression>nth[1]]"/>

That’s not optimal, but it has been implemented that way.

---

Working code that can be copy / pasted to tiddlywiki.com to play with.

!! one element

{{{ [tag[HelloThere]nth[1]] }}}

<$list filter="[tag[HelloThere]nth[1]]"/>

!! different behaviour

{{{ [tag[HelloThere]] }}}

<$let test={{{ [tag[HelloThere]] }}}>     <!-- IMO that's the behaviour you talk about -->>
<<test>>
</$let>

[jeremyruston]

Hi @GameDungeon the security concerns here are that we want people to be able to freely exchange TiddlyWiki content without fear that it contains malicious or unexpected behaviours. That would be hard if we allowed unfettered access to the underlying JavaScript environment intermingled with ordinary content. Meanwhile, of course we want developers to be able to access JavaScript in order to extend TiddlyWiki.

So, the compromise we arrived at was to restrict executable JavaScript content to tiddlers explicitly marked as JS modules. That makes it possible to sanitise incoming wikitext tiddlers, which would be much more difficult if we allowed JS inline within wikitext.

These security considerations become particularly important in a multi-user TiddlyWiki environment, but are also relevant to single file users if they are working with untrusted content from other users.

Another part of the conceptual hurdle for people coming from a developer background is that TiddlyWiki only superficially resembles an ordinary procedural programming language. It actually incorporates two distinct declarative programming languages:

• Wikitext itself (the signature programming feature being loops via the <$list> widget), which is some kind of dynamic execution tree
• The filter language with its monad-like syntax, which resembles some modern functional languages like Haskell or (particularly interestingly) Verse

[pmario]

I think you are right. TW wikitext is a very powerful templating engine that is able to reproduce itself.

As I wrote the problem for existing programmers is the different mindset, that wikitext uses. Once it makes “click” it is much easier to use as a conventional programming environment that creates UIs.

[Charlie_Veniot]

I speak both English and French. As I speak in either of those languages, I’m also thinking in the language I am speaking. Otherwise, I find myself translating in my head, and the results can be pretty screwed up.

Same for TiddlyWiki. You have to think in that language. Or a language that is very similar.

When it comes to TiddlyWiki programming, you will have better success if you think as you would writing SQL (and thinking relational algebra), than if you are thinking C/C++/javascript/java/fortran/rust/whatever.

If not thinking as you would when writing SQL, then thinking as you would when writing APL.

Think in a series of transformations from the raw stuff you have at the beginning to the end result you want.

I can’t explain it beyond that. Too complicated to explain.

[Scott_Sauyet]

@Charlie_Veniot

When it comes to TiddlyWiki programming, you will have better success if you think as you would writing SQL (and thinking relational algebra), than if you are thinking C/C++/javascript/java/fortran/rust/whatever.

Is the distinction you’re making here between an imperative style (do this, then do that, then do the other thing) and a declarative one (give me output like this, however you need to implement it?)

I write mainly in JS these days, front and back end. I write mostly in a functional paradigm, which is much closer to the declarative mindset. But I still find myself facing the same issues @GameDungeon describes. That is improving with experience, but comes back frustratingly often.

Or did you mean something else more specific to SQL, something that might not apply to things like XSLT or Haskell or LISP?

[GameDungeon]

In this case, then, I think some confusion came from the word “Function” I originally asked why WikiText didn’t have something like a function instead of just macros. That’s where the security reasons thing came up. I was not referring to JS, I meant within wikitext itself.

Having the ability to run some code and return a value is surprisingly valuable. Just as the first example that comes to mind, in a macro-only language, you lose entirely the ability to use recursion.

Honestly the fact that someone has to show me to use wikify is a failure on the language’s part. It’s entirely unintuitive. Maybe I do have the wrong mindset, but it’s a mindset that comes from experience with software, and as far as I can tell, one not at all challenged by the docs. Above charlie says I have to think in the language. I can’t think in a language I don’t understand. The docs give me a vocabulary without teaching me grammar. It just comes out as a word soup.

Wikify shouldn’t exist. This post partially exists because of me. The author was finally pushed to write it, by my complete failure to understand what it actually does. I still don’t really understand what it does. Why can’t I use “the output of widgets in a filter expression?” Wikify feels like a patch over a hole in a language, and a patch I’m told not to use, as it’s “slow.” It’s also a patch that without someone explicitly telling me about it, I would never even think to use, I’d probably never even guess it exists. Honestly, short of bashing your head against the wall for a while, it’s hard to even tell that a hole is there. “Maybe I’m just typing this wrong?”

Maybe this just sounds crazy, trying to catch an emotion of frustration, and then discontent upon hearing the solution and putting it into words.

[TW_Tones]

@GameDungeon

Having the ability to run some code and return a value is surprisingly valuable. Just as the first example that comes to mind, in a macro-only language, you lose entirely the ability to use recursion.

This is curious to me because it is just incorrect, unless you are using one of these terms differently to me, or you are referring to the specific case where if you don’t wikify something it breaks.

@GameDungeon

Honestly the fact that someone has to show me to use wikify is a failure on the language’s part

• Frankly I totally agree, I have stumbled on this a number of times and just in time wrapping something with a widget so that a variable is treated as we expect it to, that is, to return its a value, not the code that determines value, is next to silly even if there is a historical justification. This is however an “edge case”.
• I have endeavoured to have this at least simplified down to some custom delimiters without success.
• As you say, the other problem is we are readily discouraged from using wikify whilst at the same time there are some cases where it is the only option.

Although not withstanding my above agreement with you on wikify @GameDungeon there are a few reasons in my view why you may have problems initially becoming proficient in tiddlywiki, some that effected me as well.

• If you are familiar with procedural programming you may come unstuck because the way tiddlywiki is implemented is more like a 4th generation language, List Processor.
• If you are familiar with object oriented programming you will get a little stuck because the objects are outside the code, they are tiddlers.
• Both procedural and Object oriented programmers get stuck somewhat because the rendering is handed over to the tiddlywiki core to handle.
  • You need only do part of the work, tiddlywiki does the rest.
• Another thing is how somethings need a trigger and others handled automatically.

I personally had the advantage of starting with TiddlyWiki Classic v2.x , where to do anything sophisticated, you had to understand the architecture and the place of tiddlywiki’s role in keeping the whole wiki up to date (in a very efficient manner) and the final render process.

• A technical summary of this, although aging (2014) can be found here

Where tiddlywikis architecture is difficult for programmer’s to come to terms with, I believe it is much easier for non-programmer’s to come to terms with tiddlywiki than most program languages. Remember we did not know programming until we learned it.

The truth is what we already know can make an “Ass out of U an ME” - ASSUME, assuming and part of tiddlywiki is unlearning some things.

• Pity we cant use Spock’s mind melding technique.

What you see as an end result in a tiddler, is the sum of things that take place as a result of a number of different and sometimes sophisticated parsing processes that TiddlyWiki does for you. Not the same as what you are accustomed to in many languages. In part this is because its all built inside a HTML document.

I hope this helps

[JanJo]

Just a short statement: I love Wikitext.

[EricShulman]

@GameDungeon

…in a macro-only language, you lose entirely the ability to use recursion.

This is not true!

Recursion is used in the TWCore toc-* macros (see $:/core/macros/toc), which does a depth-first recursive “tree walk” based on tags. Another example can be seen in the TWCore tree macro (see $:/core/macros/tree), which produces a depth-first recursive tree walk of system tiddlers by splitting tiddler titles using “/” as a separator.

Note that neither of the above examples use the $wikify widget since they produce direct output in a wikitext context.
It is only when we need to “capture” the output for further processing that $wikify comes into play.

For example, I’ve written a recursive <<toc-list>> macro that works similarly to the TWCore toc-* macros, but produces a flattened list of tiddlers, rather than outputting an indented tree view. Let’s suppose we then want to navigate through this flattened list using first/previous/next/last buttons.

Note that I previously posted an example of this type of navigation using a filter that finds all tiddlers that have a field named seq that contains a “sequence number” that defines the order of the tiddlers to be viewed. (see Next / Last functionality using fields - #2 by EricShulman).

Here’s a variant of that wikitext navigation code, using a list generated by my recursive <<toc-list>> macro to perform sequential navigation, starting from a “root tag” tiddler (e.g., “TableOfContents”):

<!-- RECURSIVE TREE-WALK TO GENERATE A FLAT LIST OF TIDDLERS -->
<!-- USE OPTIONAL "max" PARAM TO LIMIT THE DEPTH OF THE RECURSION -->
<!-- "exclude" IS AUTOMATICALLY COMPUTED TO AVOID INFINITE RECURSION CAUSED BY "TAG LOOPS" -->
\define toc-list(here,max,exclude,level:"1")
<$list filter="""[tag[$here$]] $exclude$ -[[$here$]]""">
   <$text text="[["/><<currentTiddler>><$text text="]]"/><br>
   <$reveal default="$level$" type="nomatch" text="$max$">
      <$macrocall $name="toc-list" here=<<currentTiddler>> max="$max$" exclude="""$exclude$ -[[$here$]]""" level={{{ [[$level$]add[1]] }}}/>
   </$reveal>
</$list>
\end
<!-- UTILITY MACRO TO OUTPUT THE BUTTONS -->
\define go(label) <$button> $label$ <$action-navigate/></$button>
<!-- GET THE FLAT LIST -->
<$wikify name="tids" text="<<toc-list 'TableOfContents'>>">
<!-- MAKE SURE THE CURRENT TIDDLER IS IN THE LIST -->
<$list filter="[enlist<tids>match<currentTiddler>]" variable="curr">
<!-- OUTPUT THE FIRST/PREVIOUS/NEXT/LAST BUTTONS -->
<$list filter="[enlist<tids>first[]!match<curr>]"><<go first>></$list>
<$list filter="[enlist<tids>before<curr>]"       ><<go prev>></$list>
<$list filter="[enlist<tids>after<curr>]"        ><<go next>></$list>
<$list filter="[enlist<tids>last[]!match<curr>]" ><<go last>></$list>

To see this code in action on https://TiddlyWiki.com, just copy/paste the above code into a tiddler (e.g., “TOCNav”), and tag it with $:/tags/ViewTemplate. For each tiddler in the TableOfContents, navigation buttons will appear at the bottom of the tiddler.

[twMat]

@GameDungeon

in a macro-only language, you lose entirely the ability to use recursion.

As Eric notes, it works fine.

\define recursion(string) 
<$list filter='$string$ +[split[]butlast[]join[]]'>
{{!!title}}
<$macrocall $name=recursion string={{!!title}} />
</$list>
\end

<<recursion recursion>>

[pmario]

@GameDungeon

Having the ability to run some code and return a value is surprisingly valuable. Just as the first example that comes to mind, in a macro-only language, you lose entirely the ability to use recursion.

Recursive macros work perfectly fine … One of the famous examples for recursive code is the Fibonacci series.

So the spec is: Print all Fibonacci numbers below 1000.

The code looks as follows. I call it “macronacci” in short “m” since it does not force us to start with 0, 1, 1, 2 … It allows to start with any number n1 and applies the same rules as the Fibonacci series does.

It also has a sep-parameter to define the separator between the numbers. This comes in handy if we want to make the list more readable. eg. linebreaks instead of spaces

\define m(n1:0 n2 sep:" ")
<<__n1__>><<__sep__>>
<$let n1={{{ [<__n1__>match[0]then[1]else<__n1__>] }}}>
  <$let n1={{{ [<n1>add<__n2__>] }}}>
    <$list filter="[<n1>subtract[1000]sign[]match[-1]]">
      <$macrocall $name=m n1=<<n1>> n2=<<__n1__>> sep=<<__sep__>> />
    </$list>
  </$let>
<$let>
\end

<<m>>

The output is

This works perfectly fine. The $macrocall widget is used to call the “m” macro recursively.

Instead of only calculating the numbers with the line <$let n1={{{ [<n1>add<__n2__>] }}}> we could also calculate the quotient for n1/n2 which works by replacing the old formula with the following lines. The quotient should converge against the “golden ratio” which is 1.618033

  <$let n1={{{ [<n1>add<__n2__>] }}} quotient={{{ [<__n1__>divide<__n2__>precision[6]] }}}>
| <<quotient>><br>

As I wrote TW is all about creating lists to “draw” content.

BTW we use recursion often eg: The table of content and the tree macros use recursive code.

[xcazin]

Hi @GameDungeon,

Let me offer my understanding of Wikitext after many years of using TiddlyWiki.

I believe that your frustration comes from overlooking that a wikitext program is mainly aimed at asynchronously displaying HTML code in the user’s browser, in adequate response to user actions.

• Widgets are responsible for storing this code into tiddlers (usually upon interaction with the user), some of which may get displayed immediately, others just holding the code lazily, some just recording a state.
• Filters are the closest things to functions. They are the most powerful way to compute results in an ever refreshing environment (once again due to unpredictible user interactions). Before being synched into tiddlers, these results are usually assigned to variables or widget attributes.
• Macros are primarily a way to store text or code snippets to be reused at will by widgets or filters.

What about the <$wikify/> widget? It is a very special widget that can be used to capture the text that would otherwise be computed by the display engine, in order to produce immediately usable code. For instance, say you like metaprogramming and you want to compute a filter string based on specific user interactions, and use this filter right away in your code : in such cases, it is handy to store the “ready to print” result of your computation into a variable, thanks to <$wikify/> and reuse this as a subfilter. Certainly not for the everyday use.

[Scott_Sauyet]

@GameDungeon

I can’t think in a language I don’t understand. The docs give me a vocabulary without teaching me grammar. It just comes out as a word soup.

I’ve been thinking about that a lot. I’m asking myself how I would document TW so that it would be comprehensible to someone who thinks like me. So now I’m asking you, @GameDungeon. I’m assuming you’ve made some progress and it’s not all still mysterious. How would you document what you know?

I’m hoping to start writing such documentation in six or eight weeks when some of my other work should slow down. I have no idea how to present it, or if what I eventually do might be useful on tiddlywiki.com, but I would start entirely separate. If you have interest in contributing, please let me know.

(And feel free to reach out to me out-of-channel. scott@sauyet.com.)

[linonetwo]

Acturally we have write many tutorials in Chinese tw-cn.netlify.app/ to cover conercases like:

1. using transclusion in filter
2. using filter in transclusion
3. using variable in filter
4. using xxx in yyy

and

1. get wikitext variable in js side
2. send message from wikitext to js
3. call wikitext widget from js side
4. xxx wikitext from/to js side

I think with these knowledge, wikitext is just like simple JSX in React.

[Charlie_Veniot]

Well if I could easily describe it with labels, I would have described it with labels.

None of the labels describe it.

The difference is like the difference between one spoken language and the other, and the ability to fully transition from one to the other such that you are thinking in the same language you are speaking.

Imagine somebody speaking Spanish with French accents on everything instead of Spanish accents, and English sentence structure all over the place instead of Spanish sentence structure.

TiddlyWiki programming, the thinking is more like SQL programming for the person who has mastered SQL programming, and thinks SQL programming as per the math behind it.

So many great Zen stories that would fit, like:

The Zen teacher’s dog loved his evening romp with his master. The dog would bound ahead to fetch a stick, then run back, wag his tail, and wait for the next game. On this particular evening, the teacher invited one of his brightest students to join him – a boy so intelligent that he became troubled by the contradictions in Buddhist doctrine.

“You must understand,” said the teacher, “that words are only guideposts. Never let the words or symbols get in the way of truth. Here, I’ll show you.”

With that the teacher called his happy dog.

“Fetch me the moon,” he said to his dog and pointed to the full moon.

“Where is my dog looking?” asked the teacher of the bright pupil.

“He’s looking at your finger.”

“Exactly. Don’t be like my dog. Don’t confuse the pointing finger with the thing that is being pointed at. All our Buddhist words are only guideposts. Every man fights his way through other men’s words to find his own truth.”

A Taoist story tells of an old man who accidentally fell into the river rapids
leading to a high and dangerous waterfall. Onlookers feared for his life.
Miraculously, he came out alive and unharmed downstream at the bottom of
the falls. People asked him how he managed to survive. “I accommodated
myself to the water, not the water to me. Without thinking, I allowed myself
to be shaped by it. Plunging into the swirl, I came out with the swirl. This is
how I survived.”

[Charlie_Veniot]

If you are getting frustrated, then the depth and breadth of your software development experience might be getting in the way.

It could be you have fallen into something akin to getting upset at the green bean plant for not producing a tomato despite you doing all of the things that would help grow a tomato.

Know what it is, and do as it needs. Think as it is.

Detach yourself from all you know that is software dev. Detach yourself from expecting TiddlyWiki programming to behave like anything you’ve known.

Get rid of your expectations. You are not going to get tomatoes out of that green bean plant.

Exception: if you can think SQL correctly (i.e. you are at one with the math of SQL), then think SQL and that will be helpful.

Learn to walk before you run. Learn to crawl before you walk.

There are some very tiny things from your software development experience that will be of use. You’ll know when they are of use as you know TiddlyWiki programming.