Filter Syntax History

#1

Thank you for the high praise.

For context, before TiddlyWikIi, I only had practical experience of a very small number of languages: machine code, assembly language, BASIC, FORTH, C, C++ and of course JavaScript. I had a smattering of Java, Pascal and one or two other ancient languages but no experience of actor based languages like Erlang, or modern functional languages.

The story starts with the double square bracket syntax used in wikitext for links. This was already an established usage in wikis. I switched the order of pretty links because I thought Wikipedia’s [[link address|link text]] was the wrong way around because it broke up sentences: The file is [[https://site.com/thing|here]] seems less readable than The file is [[here|https://site.com/thing]]. Obviously now I wish I hadn’t been so big headed, and had just gone with Wikipedia’s established usage.

Anyhow, one way to look at the double square bracket link syntax is that it establishes a way to quote page/tiddler titles so that they may contain spaces, and don’t have to use CamelCase.

Thus, in TiddlyWiki Classic, when I was implementing the DefaultTiddlers feature it was natural to use double square brackets to quote titles containing spaces.

Then, when I was starting to think about TiddlyWiki 5 I wanted to extend the implementation of DefaultTiddlers so that more complex logic could be expressed while retaining backwards compatibility. A trick that I am apt to use in such situations is to try to engineer things so that the current behaviour is re-interpreted as a shortcut syntax for a new, richer syntax that provides more flexibility. In this case, the idea was that in filters we would interpret [[mytiddler]] as a shortcut for [title[mytiddler]]. Then we could put any keywords we like in place of “title”, giving us an infinitely extensible syntax. A similar example is the way that we implemented filter run prefixes by retrospectively defining the absence of a prefix as implying a default prefix.

We implemented a very, very simple first version of the filter syntax in TiddlyWiki Classic that supported things like [tag[mytag]], but it wasn’t until TiddlyWiki 5 that it evolved into something approaching a programming language. As others have probably expressed much more eloquently, a characteristic of the programming languages that I love is that they start with a small number of principles that are consistently applied and combined. In the case of TiddlyWiki, the list would be very roughly:

  • Double square brackets for linking and quoting
  • Curly braces for transclusion
  • Angle brackets for macros (which evolved in variables)
  • Double exclamation marks to indicate fields {{!!myfield}}
  • Double hashes to indicate indexes {{##myindex}}
  • Smashing together adjacent filter operations by removing the combining ][
  • The dollar sign as a rough signifier of data owned by the system rather than the user

As I have written about elsewhere I was privileged to know Joe Armstrong, the co-inventor of Erlang, in the last few years of his life – we were working together on a book about TiddlyWiki when he passed away in 2019. Joe had contacted me out of the blue ten years before to express his admiration for TiddlyWiki, and we had developed a friendship. He was actually a big fan of TW5’s filter syntax, and used to make me feel better about it by joking that I had (re-)invented the monad, which sounded impressive to me. That doesn’t make the filter language any easier to learn, but it does mean that it is worth learning: it’s a real language, based on the same principles as other languages.

I find it pleasing that the TW5 filter language has its roots in decisions that were taken in the TWC days. It’s still hard to learn, but that’s an ongoing paradox of programming: people want to do complicated things, and complicated things are complicated. It’s hard to see how we could have made filters any simpler without depriving users of the possibility of doing complicated things.

13 Likes
Filter out low-intent users
#2

fwiw, I think wikipedia has it wrong (wikipedia’s syntax origins are in UseModWiki, which has worse decisions than this even), and TW and markdown have it the right way around. It also makes it easier to refactor content from TW’s wikitext to markdown, and vice versa

Interesting to read the history of the current filter syntax - I think that’d go well in an “Introduction To…” type guide (I can’t be the only one who finds knowing the logic behind the creation of something, to be of great assistance in then understanding and learning it)

4 Likes
#3

I agree with @nemo that you were right about this. Wikipedia’s way is maybe more “logical” - you are creating a link, and the alias is the icing on the cake. But TiddlyWiki’s way is more user content-oriented - the sentence first, and the link as the addendum.

6 Likes
Filter out low-intent users
#4

Same! Even before I discovered TW (and Markdown), I always naturally type the title first before the URL. I just can’t get myself to do it the MediaWiki way. :sweat_smile:

It’s more natural that way. I think the most important also is a11y, readers will read the title first before the link.

1 Like
#5

That’s interesting. Although I’ve used TW for years (first TW was in 2005, first serious use in 2010), I’ve only paid real attention to the filter syntax in the past few years, as I’ve gotten more involved in the community. But I’ve never thought of a filter run in quite that way. I believe you’re suggesting that [tag[User]has[mastodon]] is best thought of as

        [tag[User]] [has[mastodon]]
<!--    `----+----' `------+------'
             |             `---- step 2 
             +----- step 1
                 compressed               -->
        [tag[User]][has[mastodon]]
<!--              `'--------- elided      -->
        [tag[User]has[mastodon]]

I’ve always thought of the outer [ ... ] as a pair, with its component parts looking like tag[User]:

      [ [tag[User]  has[mastodon]       ]
<!--  | `----+----' `-----+-----'       | -->
<!--  |      |            `---- step 2  |
      |      +----- step 1              |
      +------------ wrappers------------+ 
                   compressed             -->
        [tag[User]has[mastodon]]

My understanding matches the Railroad diagram1, and quite possibly the parser implementation, although I haven’t checked. But that doesn’t mean it matches the historical design. I’m trying to figure out of this is a potAYto / potAHto distinction, or a serious change in perspective I need to consider.


1 There seems to be a slight problem with the Filter Run diagram. I will try to see if it’s in our syntax or in the tool we’re using. The three bottom loops don’t actually allow smooth flow; their bottom halves are attached inside rather than outside. See Filter Step for comparison.

#6

I did create a PR with the OP: [Docs] Add Filter Syntax History by pmario · Pull Request #9201 · TiddlyWiki/TiddlyWiki5 · GitHub

Edit: Jeremy was faster than me :wink: https://tiddlywiki.com/#Filter%20Syntax%20History

#7

I feel like this should be in the official wiki, somewhere in relation with the Filter page.
Most of the explanation related to filter list the functions, show 1 or 2 example, and that’s it.
If you don’t already have the logic, it won’t help you that much.

Explaining your though-process behind the conception, the logic behind the language… This could help a lot of users to understand the filtering system you conceived, and thus allow them to use it more efficiently,

At least, I can say that it did gave me a better understanding of TiddlyWiki’s filters.

1 Like
#8

I’ve realised that I missed off a crucial step in the story: the first implementation of filters was actually in TiddlyWiki Classic back in 2007 (see this commit), but with a simplified syntax and only a single tag operator. That was just about sufficient for opening multiple tiddlers by tag, but didn’t allow for smashing operations together.

// Filter a list of tiddlers
//#   filter - filter expression (eg "tidlertitle [[multi word tiddler title]] [tag[systemConfig]]")
//# Returns an array of Tiddler() objects that match the filter expression

Provision to combine the filter operators had been on my mind from the beginning, but when TiddlyWiki 5 started in 2011 I reused the simple implementation from TiddlyWiki Classic. Smashing operators had been on my mind from the beginning, but was only implemented in May 2012. By the time of the launch the filter language had grown into pretty much what it is today – see the documentation for TiddlyWiki v5.1.0.

I’ll push an update shortly.

1 Like
#9

I will try to get to this soon, but vacation prep is taking most of my non-work/non-sleep time. And I can’t post a PR from my work machine, even with the Docs PR Maker. So if someone beats me to it, that’s great!

The fix is to swap the nesting of [: { ... } ] to { [: ... ] } in the three bottom clauses:

  <$railroad text="""
  \start none
  \end none
  ( "[" { [[<"Filter Step">|"Filter Step"]] } "]"
    |
-   [:{/"anything but [ ] or whitespace"/}]
+   {[:/"anything but [ ] or whitespace"/]}
  |    |
-   '"' [:{/'anything but "'/}] '"'
+   '"' {[:/'anything but "'/]} '"'
    |
-   "'" [:{/"anything but '"/}] "'"
+   "'" {[:/"anything but '"/]} "'"
  )
"""/>

Complete

<$railroad text="""
\start none
\end none
( "[" { [[<"Filter Step">|"Filter Step"]] } "]"
  |
  {[:/"anything but [ ] or whitespace"/]}
  |
  '"' {[:/'anything but "'/]} '"'
  |
  "'" {[:/"anything but '"/]} "'"
)
"""/>

Before:

BadRailroad

After

image

#10

This is an eloqant statement of something a lot of non-developer or non-information Technology professionals users discovering TiddlyWiki sometimes fail to grasp.

  • Perhaps @jeremyruston we can inform visitors to tiddlywiki of this sage advice?

Some complain because they want to design a complex algorithium but similtaniousely complain because it is not simple to do so.

  • Of course as a community we try and capture common algorithiums and put them (or parts to support them) in a plugin, edition or tiddlyWiki script, so they are available out of the box.
  • I often see this as one of my roles as a superuser, testing the limits of tiddlywiki out of the box without resorting to coding.
#11

It’s not a bad idea for the docs to directly address the FAQ “Why is TiddlyWiki so complicated?”, I’ll add it to my list…

#12

Thanks @Scott_Sauyet, I’ve committed your change here.

#13

IMO that’s wrong. Filter step needs to be fixed and this change should be reverted. You can not skip “anything” and then loop again. Either skip or loop