Tiddlywiki Upgrade Breaks my Wiki

, , ,
#1

Hi, wondering if I’m missing something. The Tiddlywiki Website says versioning works as follows:

MAJOR.MINOR.PATCH

MAJOR version when you make incompatible API changes,
MINOR version when you add functionality in a backwards-compatible manner, and
PATCH version when you make backwards-compatible bug fixes.

But if I try to upgrade my Tiddlywiki, every customization I have made is broken and the entire wiki fails to load. Furthermore, it seems the files are completely different and no longer editable by hand.

Does this NOT constitute a MAJOR, breaking change? Why is Tiddlywiki still on version 5 if major changes have been made to core that break plugins and customizations?

Developers like “pmario” seem to think that going from 5.2.XX->5.3.XX represents a major version change. See: Retro-upgrading aka downgrading - Is it possible? - #5 by pmario

I’m concerned little consideration is being given to breaking changes between minor versions.

#2

Hi, I’m sorry to read, that you have a problem with the upgrade.

From which version did you upgrade?

It is possible that 3rd party plugins and user customizations break with any update, but it should not happen, that the wiki is unable to load.

Which mechanism did you use to upgrade?

Do you use the browser or any other app to open your wikis?

#3

Hello. Thank you for responding to my post.

I am upgrading from 5.1.17. I have tried both dragging and dropping tiddlers as well as using the upgrade tool on the Tidddlywiki website.

But my larger concern is what I have posted in the OP. Namely, that it DOES fail to load, it DOES break customizations and plugins i.e. not backwards compatible, and that the HTML file now stores tiddlers completely inside of a JSON script tag instead of inside of html tags, which effectively removes the ability to edit the HTML file by hand, which I have done in the past when developing solutions using Tiddlywiki as well as debugging issues such as this, when the wiki fails to load entirely.

Can you please address the substance of my post, namely that versioning conventions are not being followed, and that inadequate consideration is being given to making breaking changes/ensuring backwards compatibility?

#4

I wonder, If it fails to load, how can you tell, that it breaks customizations. → It fails to load.

So that’s a problem we should try to solve first. Then we can tell if it also breaks used plugins. - Which it most likely does. - It “jumps” 2 major versions.

I think you are right. We need to update the TiddlyWiki Versioning tiddler.

TWs versions are 5.Major.Minor … Since TW is an app and not an API, there are no real patch-versions.

The TW prerelease page contains updates from the master branch. I think other projects would call them RCs (release candidates), which can be used by plugin devs for testing.

So you actually did “jump” 2 major versions.

  • v5.2.0 did introduce the change from DIV based data store to JSON based data-store.
  • v5.3.0 did introduce \procedures, \functions, custom \widget’s and parametrized transclusions

This step was necessary, because the DIV based data store caused a number of problems, that bocked the advancement of the app.

Using command line tools to externally create a tiddler-store is still possible – but different. We use JSON as a data-store format, because it is well understood, while the div-based store did confuse new developers.

Modifying the HTML by hand has never been a recommended way, but you are right JSON is harder to modify by humans.

#5

@Maladroit when you say “no longer editable by hand”, do you mean no longer editable with a text editor. If so, is this something you have done in the past? If so, could that have caused the upgrade failure?

Regardless, do you have backups of previous edits? If so, try upgrading any of those in an attempt to determine when the issue first presented itself.

If two files can be located (one that upgrades and one that doesn’t) then using a text editor (like notepad++) with a compare feature, compare the two files. It might highlight the reason.

#6

Hi @Maladroit welcome to the community.

I’m sorry to hear that you had problems upgrading from v5.1.17 to v5.3.4.

As you note, we indeed endeavour not to knowingly break backwards compatibility as we continue to improve TiddlyWiki.

Issues can still occur with upgrades, particularly in a case like this where v5.1.17 dates from 2018, and so there are 18 intermediate versions being skipped. There’s a non-zero chance of a problem with each upgrade, and jumping so many versions multiplies those chances.

The fundamental reason is that almost all meaningful changes are not fully backwards compatible. For example, last year we introduced <%if [<condition>]%> as a shortcut syntax for conditionals. Previously, those character sequences would have been parsed as plain text. It’s hard to think of a usage scenario where that would cause a problem with the upgrade, but it is undoubtedly not fully backwards compatible.

So, we try to strike a balance. We reject a great many changes on backwards compatibility grounds, but we have to accept that

When we do run into a problem as you have, we debug it and if necessary fix it.

Turning to the JSON store area, the motivation for that change was because previously the restrictions on HTML attribute names reflected into TiddlyWiki as restrictions on field names. That was very confusing for users, and exposed platform dependent features to users.

In cases like this, where we can, we provide a switch to revert to the old behaviour.

In this case, create a tiddler tagged $:/tags/Global with the following content:

\define storeAreaFormat() div

Subsequent “saves” will use the old DIV-based format.

#7

Thank you, @pmario, Jeremy, and @clsturgeon. I appreciate your responses.

@pmario, the reason why I say it fails to load and breaks customizations is because while importing by dragging and dropping in batches I ran into problems that broke the wiki and stopped it from loading, but other customizations had been loading (but not without with issues that would require fixes themselves). Whereas the upgrade tool basically spits out a completely broken wiki. I say “fail to load” and “broken” to describe the scenario where apparently Javascript issues prevent any tiddlers from loading and subsequently being unable to make any changes from within the wiki as usual. This combined with the JSON storage method essentially leaves me with a broken file that needs to be recreated from scratch.

@clsturgeon, I don’t think editing by hand was the cause of this particular issue. This particular wiki wasn’t edited in that way that I can recall.

Jeremy, Thank you, I didn’t know about that option. I will probably utilize that moving forward.

The purpose of upgrading is to allow my wiki to be uploaded to Tiddlyhost without running into a (Tiddlyhost) bug that disallows fields starting with numbers in 5.1.XX wikis.

I may decide upgrading is not worth the headache and look into alternative solutions for my problems. I have already asked Simon Baird to fix the bug so I can continue using 5.1.XX. I appreciate the responses in this thread and the helpful community members. Thanks again!

#8

A suggestion in case it helps narrow the problem.

I’ve got a copy of the TW 5.2.3 upgrade wizard hosted here.

If you can successfully upgrade your TW 5.1.17 to TW 5.2.3, then we know something introduced after 5.2.3 is causing a problem.

If your TiddlyWiki instance does not work when upgraded to TW 5.2.3, then it might be worth trying with older upgrade wizards until you find success.

That aside …

If there is a way for you to share your TiddlyWiki instances, somebody in the community might be able to figure out the issue preventing your instance from successfully upgrading.

#9

Thank you for that.

Here’s a copy of the wiki in question:

https://matador5117.tiddlyhost.com/

#10

This is very nice to have available. It makes me think, though, that we should have one of these posted alongside each archived version. While we’d always encourage people to upgrade to the latest version when possible, this would let us debug problems, let plug-in developers easily find the first compatible version, and in this case, let the OP—with a little binary search—be easily able to determine the the earliest version that causes an upgrade issue.

What do you think, @jeremyruston?

#11

you need to disable (uncheck) the import of these tiddlers when upgrading
$:/core/images/chevron-down
$:/core/macros/colour-picker
$:/core/modules/startup/rootwidget.js
$:/core/modules/tiddler.js
$:/core/modules/utils/dom.js
$:/core/modules/utils/dom/scroller.js
$:/core/modules/utils/utils.js
$:/core/modules/widgets/checkbox.js
$:/core/modules/widgets/reveal.js
$:/core/ui/Buttons/more-tiddler-actions
$:/core/ui/Buttons/save
$:/core/ui/Buttons/save-wiki
$:/core/ui/PageTemplate/sidebar
$:/core/ui/SideBar/More
$:/core/ui/SideBar/Tools
$:/core/ui/SideBarLists
$:/core/ui/ViewTemplate/subtitle
$:/core/ui/ViewTemplate/tags
$:/core/ui/ViewTemplate/title
$:/plugins/anstosa/tw5-material

#12

EDIT: This reply of mine, as I first wrote it, made no sense at all because I did not notice Maladroit shared the TiddlyWiki that is not upgrading. Now that I’ve seen that TiddlyWiki …

Yeah, core tiddlers in TiddlyWiki 5.1.17 that have been changed seem to be breaking subsequent versions of TiddlyWiki (well, I’ve only tried TW 5.2.3.) When core tiddlers are changed, that’s pretty much guaranteeing upgrade issues at some point, I think.

#13

That kind of thing has always been my standard practice in my career. I never thought about it outside of work (yet I do that with BAM? Hmmm…), and I think that is a wonderful suggestion.

That’s a great idea. It can immensely help pinpoint the source of a problem, whether a bug in TW or an unanticipated kind of customization or a plugin that needs to be updated/replaced or whatever-else-under-the-sun.

#14

It’s a good idea. The only complication might be that the upgrader is primarily designed to upgrade wikis from earlier versions, rather than downgrading to earlier versions. It offers to replace plugins with newer versions but doesn’t by default offer to replace plugins with older versions. So it might take a little more care than a usual upgrade

#15

IF (and only if) you’ve made changes to core tiddlers, and you are trying to import those modified TW 5.1.17 tiddlers into TW 5.3.3, it should come as no surprise that TW 5.3.3 is going to completely break.

Every core tiddler in TW 5.3.3 is expecting every other core tiddler to have what TW 5.3.3 needs by design.

So if you’ve indeed modified core tiddlers, you are going to need to exclude those tiddlers from the upgrade process (which I’ve tried, and your upgraded TiddlyWiki instance no longer complains with a JavaScript error). That said, your TiddlyWiki doesn’t look/work right after the upgrade. (Whether because your upgraded TiddlyWiki is missing things because of changes needed in the core tiddlers, or because your tiddlers depend on something in the core tiddlers that are no longer there in 5.3.3 ???)

It is then a matter of going through each core tiddler (identified during upgrade process and excluded from the process) to see what has changed from 5.1.17 to 5.3.3: are the diffs because of a change you made to the core tiddler, or changes made during enhancements to TW, or both ?

You’ll then have to figure out a new way to implement what you had in 5.1.17 in 5.3.3, but in a way that does not alter core tiddlers (at least as much as possible.)

Unless somebody can recommend something much less painful, here’s what I think is involved:

A browser window with the TW upgrade wizards, in which you drag your TW 5.1.17 instance for upgrade. This is to identify the core tiddlers that do not match.

A browser window with the TW 5.1.17 instance, to find the core tiddlers that do not match.

A browser window with TW 5.3.3, to drag into those TW 5.1.17 core tiddlers that do not match. That so you can see the diffs between the tiddlers, and figure out what you have to do to your upgraded TiddlyWiki (I.e. 5.1.17 to 5.3.3, without bringing in the core tiddlers). So there’s going to be some tedious work that needs to be done, I think.

image
image1900×963 311 KB

#16

Yes, that would make it harder to find the earliest supported core version for plugins written against more modern ones, That’s too bad, and I don’t think it’s really fixable; we don’t want to try to support a downgrade path. But this would still help finding upgrade problems:

Hmm it works in 5.1.17, but an upgrade to 5.3.2 fails. What happens if I try 5.2.0? Oh it still works. How about 5.3.0? Oops, no. Let’s try 5.2.5,

And in a few steps, the OP can find, say, that it works in 5.2.5 but not 5.2.6 Then a check of the release notes, or a more careful manual port of a 5.2.5 version can identify the problem. Once fixed, the user can again try to update this now 5.2.6 version to the latest. I’d be surprised if it took more than two of these cycles unless the wiki or its plugins have overridden great numbers or core tiddlers.