Why don't core tiddlers use indentation?

As simple as that - whenever I look at core tiddlers I always spend a lot of time just parsing through the nesting, trying to figure out what is what.

I am right now looking at $:/core/macros/tag-picker and I am at a loss. The macro tag-picker-inner() has a line that’s 637 characters long! Not to mention instead of using <$let/> it uses three nested <$vars/> instead.

What I am saying is, I love dabbling in legacy code and it seems like a lot of core tiddlers could use some love to make them readable and I’d be more than happy to spend some time from time to time just picking a tiddler at random and making it easier to parse for humans.

But perhaps there is a reason why they look like that? I know that adding indents could increase the size of tiddly wiki by a few percent and perhaps that’s the reason why?

Unlike most programming/scripting languages, TiddlyWiki wikitext syntax is, by default, sensitive to whitespace. As such, the presence of newlines and indentation can affect the rendered output. With regard to the TWCore shadow tiddlers, some of the (lack of) formatting in the “source code” is due to this sensitivity to whitespace.

One way to address this is to add the \whitespace trim pragma at the beginning of the content (either the start of the tiddler, or within a specific macro definition). However, this needs to be done carefully, as there are some cases where whitespace is still required in order to correctly render the output. For most (but not all) of these cases, it is possible to make small changes to the wikitext (such as surrounding content with <div>...</div>) to achieve the correct rendering without relying upon the whitespace parsing rules.

With regard to $set vs $vars vs $let:

$set was the original “assign a value to a variable” widget. Each assignment needed a separate $set widget, like this:

<$set name="a" value="5">
<$set name="b" value="2">
<$set name="c" value={{{ [<a>multiply<b>] }}}>

The $vars widget was then created to permit multiple variables to be set at the same time, but all assignments were done simultaneously, so if one variable depended upon another, you still needed to split them into separate $vars widgets, like this:

<$vars a="5" b="2">
<$vars c={{{ [<a>multiply<b>] }}}>

The newest widget, $let, also allows multiple assignments in a single widget, but they are evaluated in sequence, so that variables that are assigned later in the same $let widget can refer to the values of previously assigned variables, like this:

<$let a="5" b="2" c={{{ [<a>multiply<b>] }}}>

Note: while $let is now the preferred widget for simple variable assignements, $set has other forms that are still useful, such as capturing filter results (as a space-separated, bracketed list) into a variable, like this:

<$set name="tids" filter="[tag[sometag]]">

or performing "conditional assignment, like this:

<$set name="hastag" filter="[tag[sometag]]" value="tag found" emptyValue="tag not found">

Hope this helps,
-e

8 Likes

@EricShulman Thanks. I wouldn’t think whitespace would be a big deal in TW since it’s something that crops up in almost all web development and only rarely is an actual issue. That being said it is safer to just completely strip it all and never have to worry about it — safer but the code is very difficult to read.

As for differences between different ways to set a variable. I wasn’t aware that <$set/> can actually store the whole result of a filter while <$let/> can’t (or I forgot it). But the value and emptyValue can be simulated by using else and then, can’t it? That is, in your specific example:

<$set name="hastag" filter="[tag[sometag]]" value="tag found" emptyValue="tag not found">
<$set name="hastag" filter="[tag[sometag]then[tag found]else[tag not found]]">

Those two should be equal if I am not missing anything.


I am still curious if there is any worth and interest in someone going through the old core tiddlers and refactoring and restyling them.

Good question @Maurycy

With empty.html having 2,189 shadow tiddlers the introduction of white space to make the code more readable could add up quite a bit, contributing to the size of the wiki. So I can understand why the core may minimise it, however I too favour the ease of readability even self documentation within code (especially complex or novel cases). Indentation typically also includes a new line before it, and unless we are using a tab character the number of spaces add up.

First as a user/designer wanting to analyse the code there is a lot you can do to help yourself.

  • First install the code mirror plugin and display line numbers, and it allows you to use tabs.
  • Feel free to edit a core tiddler or a copy/clone and insert line breaks and indents - often this is surprisingly easy and quick to simplify reading.
  • You can also use the previews and compare differences to the shadow.

The big picture

  • If we could minimise the size of code tiddlers using a programmatic method we could reverse this to display content to the user?
  • We could have an improved wiki text Highlighter which also had the option to expand or add line breaks.
  • We could have an independent plugin containing expanded “none operating” versions of code in friendly formats with plenty of comments and whitespace/indentation- especially documenting possible customisations.

@TW_Tones I was thinking the physical space could be an issue. On the other hand I just realized - there is already indentation in JS tiddlers that’s not minimized in any way so there’s a precedent.

I don’t personally have much of an issue, I am used to reading much worse code and I can just copy-paste it into VSCode and have all the capabilities I can think of. But still, it’s time consuming regardless of how I approach this and I can only imagine how much more difficult it can be to TW newbies without any coding background.

If we could minimise the size of code tiddlers using a programmatic method we could reverse this to display content to the user?

The most optimal solution would be to just have an auto-indent option - you could run it for core tiddlers or for your own tiddlers as need. Otherwise the second best option (in terms of storage space) would be to use a single byte after each \n character that encodes the number of tabs/spaces to insert but that’d create a huge, huge mess that I don’t think anyone would want to get close to.

Whoopsie, clicked reply too early.

We could have an improved wiki text Highlighter which also had the option to expand or add line breaks.

I’ve been using CodeMirror for a while and newer before realized there’s no syntax coloring. That sounds like something easy to solve which means I’ll now waste time looking into it :).

We could have an independent plugin containing expanded “none operating” versions of code in friendly formats with plenty of comments and whitespace/indentation- especially documenting possible customisations.

I don’t like the idea of keeping and maintaining a copy of the code, unless it would happen automatically. But I also am not very fond of this solution either, it just adds friction to where no friction should be required.

My opinion is that with TW already being +2MB in size, adding another percent or two in whitespace is not a big deal :slight_smile: .

There is some working being done in Github by pmario, flibbles and others to improve the indentation and readability of core templates and introduce proper use of the whitespace pragma.

The catch is that changing whitespace can result in a different HTML output, and therefore tests need to be written to ensure that the output has not changed. This is less of a concern for macros or tiddlers only containing action strings.

From the release notes for TW 5.2.3:

2 Likes

The size increase is no concern. As Saq wrote we need tests to be sure the HTML output stays consistent. …

When I do some refactoring I also try to remove redundant HTML P-tags, remove redundant “double wrappers” around the reveal-widget. Eg use the <$reveal tag="div" parameter instead using <div><$reveal>..</$reveal></div> if and only if tc-class definitions allows it

So it’s not only adding indentation. We also need to make sure that there isn’t any unwanted side effects. Eg: templates are used with several UI elements. … Changing indentation in templates needs to make sure it works with all the core templates. … A big plus is to also have a look a some popular plugins. … They should also work after core refactoring.

It will be worth it. But be prepared that you need to make baby steps. You cannot add indentation and new code in 1 PR. It won’t get merged that way.

  • step 1: You need to add indentation and prove that the HTML output is still the same
  • step 2: Add new code that works (if possible) in a backwards compatible way.
    • 2.1 if not backwards compatible, there has to be a good reason. … eg: Fix an issue
2 Likes

@pmario Thanks! Is there any central hub I could announce I want to work on a specific macro/tiddler to avoid multiple people duplicating work?

@pmario it seems some tiddlers are more sencitive to white space than others such as templates vs macros. I often add formatting and line feeds when reverse engineering, researching or customising core code.

It certianly makes it easier to read, but the act of reformatting is part of understanding it. Inserting comments are importiant but I suspect there remain some hard to see interactions even bugs.

If we want this type of refactoring perhaps we could ask the community?

As far as I know, the only pull-requests that are open atm, which do code indentation are mine. Pull requests · Jermolene/TiddlyWiki5 · GitHub … Only some of them include tiddlers with indented wikitext code. … But you can have a look at how it works.

You can also open an issue, where you suggest to create new formatting for core tiddler “abc” . That would be less work for the beginning and you will get feedback if there is a chance to get it merged.

I have to disagree with the bolded part on a few grounds:

  1. People imitate what they see, if new TW users see core tiddlers that are an unreadable mess they’ll produce the same.
  2. The act of reformatting can be helpful in understanding something but it can just as well be a mindless drone work that only wastes time.
  3. A lot of people who didn’t have any experience with web development or programming before may not even realize indentation is a viable option.
  4. People inexperienced with programming or web development are terrible with things like proper naming, indentation, spacing, closing brackets and generally the small syntactic things that are either required or helpful.
  5. It’s a waste of time - after you properly indent some core module you have two options: a) save the changes and get issues down the line when updating TW; b) scrap the changes and if you need to understand the module again well tough luck, do it again.

As you can probably tell I’ve wasted many hours helping non-programmers and new programmers find that tiny bug which would’ve been obvious if they only actually indented their code and put spaces here and there :).

I suppose I should have said “I find it helpful”. I dont speak for others. However a point I would add is some of us find this a good way to reverse engineer just to get something done. Perhaps we can tap into this activity to rework such code prior to proposing core updates.

This is about community and crowd sourcing.

As I wrote. It does make sense to improve human readability for core tiddlers.

IMO it would make sense to start with top-level tiddlers. … That means tiddlers, where no other templates depend on them. … This will help to minimize side effects.

As I wrote. If core templates are changed indentation has to be the first step. Indentation only. So it’s easier to test that the resulting HTML stays the exact same way. …

In a second step other changes may be done eg: convert nested <$set widgets into <$let widgets. BUT be aware, that there are some cases eg: <$set name= filter=... which require to create a new logic. So those cases have to be well tested.

I think you might’ve accidentally read my old message thinking it’s newer :).

I’ll certainly try to work a bit on those changes.

Gotcha, sorry I didn’t mean to imply anything, I read it as “keeping it unindented is beneficial” and wanted to voice my disagreement. I totally get how doing the work of cleaning up the code can help understanding it, though at least in my case it’s still more time than just reading a clean code in the first place would’ve been.

1 Like
  • Perhaps we could document a position on the use of newline and tabs ie by virtue of code mirror (more efficient) and wehn reading code tiddlers without code mirror.
    • There may be a consistency gap.
  • Also do we thus introduce \whitespace trim at the same time.
  • This could easily recover the space used to add indentation
  • Nor I, I only wanted to insure clarity for now and future readers.

I suppose my key point at this moment in the conversations is can we write a reasonable set of guidelines to “code readability transformation” and thus encourage others to participate in an appropriate way.

There may be some guidelines, but the only way to be sure is testing. We use an automated test system in TW, but it is only useful for JS code.

To make “visual regressing testing” there is an open proposal at GitHub: [Proposal] It is proposed to implement UI tests using: https://playwright.dev/ · Issue #6845 · Jermolene/TiddlyWiki5 · GitHub

@Maurycy … IMO this would also be a very welcome contribution, because it would make testing template changes much much easier and automated. At the moment everyone avoids template changes, because a lot of manual testing is needed and it still can happen, that new bugs creep in.

I did read some of the framework docs, but I didn’t have time to actually test the system with TW.

1 Like