Relink macro backreferences not working in v5.3.0

#1

The backreferences in Relink plugin (used either through filter relink:backreferences, or the Relink References tab in tiddler info panel) based on macros seem not to work in v5.3.0, but were working in v5.2.7.

Steps to recreate, starting from empty TW v.5.3.0 or v5.2.7:

  1. Install latest Relink plugin v.2.3.4 and reload TW.
  2. Create two tiddlers:
title: One

<<tag Two>>

title: Two
  1. Open info panel of tiddler Two and to to “Relink References” tab.
  2. The backreference to One is seen under v5.2.7, but not under v5.3.0.

Screenshot on v.5.3.0:

5.3.0

Screenshot on v5.2.7:

5.2.7
5.2.7729×915 26.9 KB

Explicitly naming the parameter like <<tag tag:"Two">> does not solve the issue. I have only tested with tag, toc, and translink core macros so far, but I assume it applies to all (core?) macros in general.

The Relink backreferences work for custom fields under 5.3.0. I haven’t tested it with attributes or operators, as I wasn’t using this functionality before and I’m not really familiar with it.

Can anyone using Relink and its backreferences confirm this issue?

@Flibbles do you know something about it? It’s difficult for me to tell if the problem lies in the plugin or in the core update. There were surely some changes to macros in the core 5.3.0 version, but I guess they should be backwards compatible.

#2

Yeeeaaaaah,

TiddlyWiki v5.3.0 broke everything, not just Relink, but every plugin I’ve ever worked on.

I’m currently in the process of creating new versions of all my plugins. Relink is next on the list. Should take me a week or two.

There were surely some changes to macros in the core 5.3.0 version, but I guess they should be backwards compatible.

They were not.

1 Like
#3

Thanks for the quick reply, good to know that it’s already a known issue. Looking forward to the new version!

#4

@Flibbles thanks for your work, if possible do share, even at a high level what broke so others can see and perhaps learn from it, or inform the core dev team, since backward compatibility is a key value in tiddlywiki.

#5

I would submit issues, and I have for some, but frankly I don’t see the point.

It’s not a little thing that was changed. Macrocalls were completely redone. The <<macrocall syntax>> still exists, but now they’re just redressed transclusions. And this was very obviously intentional. Clearly, whoever made this change didn’t think any plugin developers used the parseTree. And unless Jermolene wants to straight up rollback the entire change, I don’t see how it’s going to be made backward compatible.

  • The macro.name property is gone. Instead, all “macrocalls” now have a “$variable” parameter that contains the name.
  • The params array is gone, and in its place are attributes and ordered attributes.
  • Unnamed parameters are gone. Instead, unspecified attributes have “integer assigned” names. There’s definitely a problem with that one, but so far no one seems to want to take action about it.

So it’s all completely different. It wouldn’t be a pain to change over existing plugins to use it, but it is a pain to have to support both the post-v5.3.0 and pre-v.5.3.0 versions of macrocalls.

Yes, I am a little salty about all this.

#6

Man, if you are salty, then I’m salty and sour.

I had mentioned in another post why I am staying on TW 5.2.3. Stability of Relink is another major reason for me to only ever upgrade TW when it hurts too much not to.

1 Like
#7

Hi @Flibbles it’s deeply frustrating, and I’d like to figure out how it could have been handled better.

The PR was first published in April 2022, and then it was extensively publicised on GitHub and here on the forum. It was merged a year later in April 2023, and then was in the prerelease for just over 10 weeks before it was released.

Ideally, we would have caught the problems at some point during that process. That didn’t happen, and nobody reported the problems until after the release.

The particularly galling thing is that I understand that you have extensive tests for the relink plugin. So, it does seem like it would be a good idea to run 3rd party tests like yours as part of our CI process. Perhaps it would be easier to manage running the tests if they were part of a centralised community plugin library

The other suggestion that has occurred to me is to introduce an intermediate set of parse tree node types that represent syntactic constructions (so there would be a distinction between all the different wikitext shortcuts for the transclude widget). These intermediate types might be named with a special prefix like “@inline-transclusion”, “@pragma-define” etc. Then we would need a process for mapping those syntactic node types to the actual rendered widget nodes that we use at the moment.

In any event, it’s a very unfortunate situation, and I’m sorry for the impact it has had on you. The challenge is that we need to be able to incrementally improve TiddlyWiki, but we also need to keep the enormous value locked up in the existing third-party plugins. Many of your plugins are pretty intricately tied to the internal implementation of TiddlyWiki which presents a particular challenge.

Much the same can be said for users like @Charlie_Veniot who are not on board with recent changes. We’d like to be able to keep everybody happy.

The one thing that I do know is that the moment we stop developing and improving TiddlyWiki is the moment that it dies. A common perspective that we’ve heard recently is that TiddlyWiki is old fashioned and hasn’t kept up with the competition.

The changes in v5.3.0 are designed to make TiddlyWiki simpler: simpler to explain, simpler to document. It’s part of a bigger process to cut out parts of TiddlyWiki that are now regarded as mistakes, and offer new users something better and simpler.

The v5.3.0 changes already give us better approaches for further improvements and modernisations to TiddlyWiki, allowing more expressive, understandable and reusable wikitext.

I am planning a v5.3.1 release which will revert a couple of the changes in v5.3.0, and I’m happy to consider going further. I’ll open a GitHub issue momentarily to discuss the details.

2 Likes
#8

Hey @jeremyruston. It looks like it’s not as bad as I made it out to be. I got fed up trying to migrate Relink and actually looked into what it would take to restore the macrocall’s backward compatibility, and it’s not much. I’ve made a PR for it here. With that change, Relink should be mostly operational again. (New features will still need to be implemented though.)

The only other problems were minor. The change to disable CamelCase Wikilinks hiccupped my tests a little, as did slight syntactical changes concerning the backtic and the new attribute substituted string feature. All of these weren’t serious.

Oh, but the “Global Assignment not allowed” definitely needs to be downgraded to a warning. Third Party javascript libraries like Vis use global assignment deliberately, and a whole slew of people using TiddlyMap are not upgrading because doing so causes TiddlyWiki not to boot up anymore. There’s a workaround, but very few users will know to find the secret unofficial versions I’ve been publishing only on github.

5 Likes
#9

I just wanted to say thank you for your continued work on TiddlyMap, @Flibbles! I have been keeping up with secret unofficial versions, and my 5.3.0 upgrade was totally painless on that front. I had no idea there was any issue at all.

1 Like
#10

Thank you, @etardiff. That means a lot.

Also, @jeremyruston, apparently the master branch is now sorting shadow tiddlers. This is breaking a lot of my tests.

Was there a super important reason why this was done?

#11

Nevermind, I sorted that out too.

#12

Thanks, I use parseTree in GitHub - tiddly-gittly/slate-write: A WYSIWYG editor for TiddlyWiki. (WIP) too, luckily I use typescript, so I can just update type in GitHub - tiddly-gittly/TW5-Typed: TypeScript type definitions for TiddlyWiki5. , and let compiler help me find where should change.

Also, I always (want to) target my plugins to target latest release of tw, because TidGi app always bundle latest release, just like it bundled latest Chromium browser engine. But I don’t really know what changed, if ts typing is not changed.

1 Like
#13

Good to hear, thank you.

I’ve made a ticket for the plans for v5.3.1:

2 Likes
#14

it does seem like it would be a good idea to run 3rd party tests like yours as part of our CI process.

This might be a good idea. My concern is that not all tests are created equal. For myself, I know I created a few bad tests for Relink before I got the hang of it. They’re tough to understand, unclear what they’re testing, and they break easily from changes. Community tests might be a mixed bag of good and bad tests.

That’s why if such a community hub existed, I wouldn’t necessarily hook it up to the build process. Just an extra step one might do before releases. This would require better standardization of testing frameworks, as well as collation of third-party plugins in a centralized source, both of these steps would be great. I’d love to see TiddlyWiki lean more on third party plugins.

an intermediate set of parse tree node types that represent syntactic constructions

But introducing such a step itself would be a breaking change. And it’d be an extra intermediate step during parsing that would probably confuse new developers as to its purpose. Outside of uglify, I don’t know why a plugin would want to care about what sort of wikitext language created a particular widget.

The changes in v5.3.0 are designed to make TiddlyWiki simpler

They do. The more I study them for their Relink implementation, the more I see their value. I know I’ve been giving everyone a hard time because I came back from a vacation to find about 12+ work hours of migrating and feature implementation waiting for me, but these features do improve. The general move away from placeholders (i.e. $(whatIsTHISgoingToBecome)$) is always healthy. And I’m sensing a trend with this release to make it so all wikitext exists in the main tiddler parse tree, instead of being hidden inside of attributes, parameters, and \define macros. This is a better design, even if I don’t fully understand it yet. (I still have no idea what the \function pragma is supposed to solve.)

#15

Yes, I can imagine that existing tests might need to be updated or rewritten to work well in this context.

It would be a breaking change, and so we’d need to coordinate that change with relink. The motivation would be to enable round tripping from raw wikitext to the parse tree and back again. Amongst other things, it would be useful if we were building a WYSIWYG editor…

Great, that’s very good to hear, thank you.

#16

@vilc, I’ve pushed Relink v2.3.5 which supports all pre-existing TiddlyWiki features.

Most new content is not yet supported. That means all the new pragma.

Especially \parameters. I am afraid of \parameters.

5 Likes
#17

Thank you a lot @Flibbles for the quick update! It works now as expected.