Break backwards compatibility - when/what?

#1

In May 2021, @jeremyruston wrote the following in PR#4920:

If we do have a release that makes non-backwards compatible wikitext changes then I’d want to together all such changes that we’ve considered over the years. Off the top of my head:

Fix the paragraph generation (newline behaviour)
Turn off camel case by default
Extending the transclusion syntax to pass parameters

Jeremy, what would be the tipping point for a make over that definitely breaks backward compatibility?

(Yes, specifically the “transclusion syntax to pass parameters” is evidently coming already now and it is, if I understand right, achieved without breaking backwards comp by adding functionality rather than replacing it.)

Other than the above mentioned changes, could there - without any commitment - be some list to aggregate the things considered for such a rework? As someone who more or less follows the github discussions, I see “can’t be done because of backwards issues” every now and then, so I’m thinking there are really a substantial number of potential improvements that would be a pity to miss out for evaluation when the time does arrive.

Thank you!

1 Like
#2

Hi @twMat good question!

That is still my view.

It is amusing that parameterised transclusion was on that list from 2021; I think I’ve said elsewhere that it is an enhancement that I’ve been thinking about for a long while.

As it has turned out, the implementation does not break backwards compatibility. The rationale for the proposed version bump to v5.3.0 for parameterised transclusion is actually the other way around: once we’re accustomed to having parameterised transclusions in the core it will be useful to be able to easily tell if a plugin/wiki etc. is from the era before they were included.

But I do think we should consider taking this opportunity for some targeted backwards compatibility breaks.

Turning off camelcase by default is an easy win, I think.

I’m not sure that we’ll have time to implement a fix for paragraph generation. There are actually two approaches:

  • Modifying the core templates to avoid extraneous paragraph tags in TW’s own UI
  • Modifying TW’s parsing/rendering architecture to do improved paragraph generation that observes HTML’s nesting rules

I’m really interested in the latter, which is a big job. The former I think is the focus of @pmario’s current investigations, and that may well be ready in time.

A big change that we might consider is to stop automatically removing duplicates from filter lists. We investigated it a while ago in Add switch for enabling duplicates within filters by Jermolene · Pull Request #3790 · Jermolene/TiddlyWiki5 · GitHub and discovered that the impact on existing code is generally very limited.

Another change we might want to consider is changing the default tv-action-refresh-policy to always, (see https://tiddlywiki.com/#ActionWidget%20Execution%20Modes for details).

2 Likes
#3

Is it possible to have a plugin to include backward compatibility and let the $:/Core itself deliberately breaks it.

While I am in favor of using modern web features in TW, but having backward compatibility is one of the best and greatest features of TiddlyWiki.

#4

Oh! So it’s basically now!? But… but… won’t this mean that it’ll be years and years until the next backward break opportunity? Assuming this is indeed the case, then shouldn’t we at least collect a list of ideas that be taken into consideration even if they all end up being dismissed by you?

Thank you again.

#5

Well, up to a point. It obviously prevents some things and it bloats the code when things are worked around.

#6

In some cases, it will be possible to move the deprecated functionality into a plugin. But in many cases all that would change is the default (eg. tv-action-refresh-policy).

It’s a tricky balance, and we need to collectively decide how far we want to go.

Changing the version number to v5.3.0 gives us an opportunity to consider making some changes to backwards compatibility, but we have a free hand, and it’s open to discussion.

3 Likes
#7

That’s right, but from my point of view, it’s also one of the biggest pain-points if you want to add new functions to the core and core UI.

BTC implemented the keyboard handling for eg: the search-dropdown and other dropdowns that are part of the UI.

This new code is so incredibly convoluted, that it is mind-boggling. Not because the new functionality is so complex or BTC doesn’t know better. In the contrary … It’s because it’s not allowed to break existing stuff, that may expect a certain structure of the UI tiddlers.

There are new TW functions that would allow us to make the UI even more flexible. Like “tagged lists” and “filter cascades” … But we can’t use them in a simple way, since 80% of the existing UI is many years old and it has to be compatible.

Working around compatibility issues probably consumes 80% of the time to implement a new function, which needs 20%.

So instead of creating 5 new features we are only able to create 1 in the same amount of time.

Just some thoughts.

2 Likes
#8

Not all of these may break backwards compatibility, this is just my wish list for the core.

  1. Make it easier to adjust the position and CSS of the tiddler title and viewtoolbar independently of each other and independently of the CSS for header 2. If I want to change the font size of one, I end up affecting some or all of the others. And moving the title or the viewtoolbar, independently of the other, to other places in the viewtemplate, should be as easy as changing the order of the list field of $:/tags/viewtemplate. I’m not saying the two can’t be in a div together side by side. But they should be less tied to each other.

  2. Include a parameter to adjust the height of the edit textarea and input windows.

  3. Create a core search widget so that one can place the search window anywhere, with a way to adjust parameters (which fields to search, what list item template, etc) similar to the list-search macro but more flexibility.

  4. The resulting html after exporting to static html should look identical to the originating tiddler as it appears in the TW.

  5. The fields editing section: Display the field name above the field value rather than to the left of it, so that the input window can go all the way across the width of the tiddler. This will allow for easier editing of list fields and other fields with many entries. The fontsize for the field name could be smaller so the section doesn’t seem as tall after adding a few fields.

  6. Add an optional style or class parameters to every widget that users could conceivably stylize.

[edit: 7. Tweak “Missing” to include all tags that have no corresponding tiddler, not just missing links]

There, that’s enough to keep you guys busy! :slight_smile:

#9

What are your thoughts on a list collecting these back breakers - including things that can be considered for revision (e.g the keyboard handling you bring up). Again, it would be a list for consideration not anyones “to-do list”. I think it would be useful for people to see what is hypothetically possible for the eventualities that we accepted some backwards comp breaking, and it is much easier to make good decisions and priorities when ideas are

  • tangibly formulated and noted down
  • grouped
  • and can be compared against each other

BTW, such a list is not the same thing as the Issues or PR lists on gh but there is for sure some overlap. (Not least some matters that have been closed or rejected with the exact argument “no can do, it would break backward compatibility…”).

Is it sensible with such a list?

#10

I think for the beginning we need 1 fundamental decision.
When do we start to completely ignore Internet Explorer in a way that we explicitly break it. So we just don’t care if it works with IE.

#11

Ironically, its user share is so small that it is surprisingly difficult to find recent statistics on exactly how small it is! It seems to be 0.64% but the desktop version is a bit higher and I’d guess TW use is also somewhat skewed towards desktop but on the other hand I’m convinced TW users are by their nature skewed to avoid IE. So my guess is that IE usage share among people reading TW’s is significantly less than 0.64%.

One approach could be to say:
“OK, MS stopped developing IE and even supporting it (June 2022). So we do serve a legacy TW version that works with it (e.g TW5.2.3), and it will always be findable but this TW version is not actively developed any more”

@jeremyruston - what are your thoughts on this?

#12

Anecadata related to IE, the only reason why I see someone using it at my work is because the system is imaged in such a way where it is still the default browser. This is despite the fact multiple internal sites we use do not function correctly. So I need to remind people when they complain something is not working they need to use Firefox or Edge. This is at large airline with 80,000+ employees.

#13

@markkerrigan, a similar issue happened at work. Four years ago we changed to Chrome as the default browser. Edge was installed a couple of years ago. However, we still have the IE icon in the taskbar after all this time (no, deleting it won’t do anything, it’ll reappear at the next reboot).

If making TW backwards compatible entitles a significant amount of work, probably is not a cost effective use of our time, IMO.

#14

I think the brilliance of tiddlywikis design should allow us introduce many changes and features while avoiding Breaking backwards compatibility. For example,

  • As far as I can see, the main place this is a problem is in the composition of tiddlers. We already permit other markups and @pmario’s Custom Markup, and “space space” paragraph etc… all tackle this with backwards compatible additional features.

It seems to me, with notable exceptions, that changes “that may break backwards compatibility”, just need a little more imagination to make them backwards compatible.

TiddlyWiki has a nice structured, layered and content driven architecture, perhaps we should leverage this more to introduce changes without causing compatibility issues.

#15

The simple answer to “when to break backwards compatibility” is “when the costs of BC outweigh the benefits”. Usually it’s a great ideal to strive for, but products which stick to it too hard can sometimes end up stacking workaround or dealing with illogical logic and alienating new users.

I think a good question to ask is: “What exactly will happen if we drop this specific backwards compatibility?” I’d say there are broadly three categories of problems to consider:

  1. Using old plugins in a new wiki
  2. Upgrading an old wiki to a new version
  3. Mental tax on existing users having to learn new things / context switching between old/new wiki.

Using old plugins in a new wiki:

  • If we know what plugins are the most popular we can test a new release against them to see what breaks and either provide guide on how to deal with it or help getting the plugin updated

Upgrading an old wiki to a new version

  • This can have the same problems as the above
  • But also break user’s customizations, macros, scripts, workflow, etc
  • I don’t think much can be done here except good release notes

Mental tax on existing users

  • This is something every user has to deal when a new functionality is added anyway, I don’t consider this a big issue (but I speak from privilege here)
  • Cost of context switching can be removed if the users see the value in updating all of their tiddly wiki instances.
#16

Back in 2011-13 when the foundations of TW5 were being laid down, Internet Explorer was still very popular and so we had no choice but to accommodate it. Having gone to the trouble of attaining that compatibility, our policy for the last couple of years on Internet Explorer has been to avoid unknowingly breaking compatibility with it.

As others have noted, the death of Internet Explorer is somewhat exaggerated. Large, underfunded institutions across the globe still run it in sizeable numbers to keep backwards compatibility with their vintage internal intranet applications.

One of the goals of TiddlyWiki is to try to be universal; to be useful to everybody, with minimal barriers to usage. The best example is the fact that TiddlyWiki doesn’t require a server, making it accessible to a whole class of users who are never going to feel comfortable running a server.

I have found that an unexpected consequence of the path that TiddlyWiki has taken is that sometimes we have to put the needs of users before the needs of developers. By that I mean that most developers would prefer to be able to use the latest and greatest features of the platform, but our users are actually better served by being able to run TiddlyWiki on the broadest range of platforms.

1 Like
#17

Clearly a noble stance, but what happens when other obscure browsers have a higher user share than the down-trending IE user share? (QQ or Sogou Explorer seem like candidates.) Should TW accommodate for them? Not to mention new types of devices. Some kind of objective “cut off” point seems sensible, even if very low. Anyway, I’m only raising this concern because Mario claims IE is one of the most limiting factors holding back TW dev. Would you agree that it is?

Is there really a dichotomy here? Of course it wouldn’t make sense to allow just any “dev fad” but on the whole, surely, developers develop stuff for the benefit of usage and users. But, more critically, it would still be you that decide what actually goes into the core so it’s not like things could go haywire. If anything, it should just make it easier to implement plugins and other solutions that can be tested to see if they serve end users in a good way, like any plugin.

Regardless, I think the perfect compromise could be made, which really should make everyone happy:

We have to assume that an IE user in, say, 2015 who used the latest and greatest TW of that time (TW5.1.1) was happy with it, just like everyone was. So if he/she can still use that very same setup today and could even download a new copy of that same TW version if is was somehow messed up… well, then he/she should still be happy.

So, instead of an inhibitingly(!) strict backward compatibility policy, we could introduce a generous backward serving policy, serving fully functional versions for legacy systems. Legacy users happy, modern devs and users happy.

Wouldn’t this make sense? Do I misunderstand something?

#18

It is not a noble stance, it is a practical philosophy that drives the development of TiddlyWiki.

I don’t quite understand why that’s a problem.

The policy I quoted above is literally the opposite of a commitment to retain support for Internet Explorer. The policy just says that we should not unknowingly break backwards compatibility with it. That means that we consider the trade-offs of breaking compatibility with Internet Explorer on a case-by-case basis.

We don’t really need to “accommodate” specific browsers any more. There’s now good enough compatibility between them that the kind of browser-specific hacks we needed 10 years ago are very seldom necessary.

I do not agree with that.

Yes. The dichotomy is that users benefit from TW running on a broad range of platforms, but developers generally prefer working with modern platforms.

Plugin authors are not constrained by the same compatibility constraints as the core. There’s nothing to stop them using the latest web platform features.

They are not really equivalent situations. Such a user would be cut adrift from everything happening in the community.

Of course old versions continue to be available for those that need them. But making them available is not a substitute for maintaining broad platform compatibility because again it fractures the community.

Stepping back, it feels as though my position might not be coming across clearly. I am not saying that we should maintain Internet Explorer compatibility indefinitely. I am acknowledging that that compatibility is still important for some users. But my main point is that we are already compatible with Internet Explorer, and so we shouldn’t break that compatibility without explicit consideration.

It’s basic tactical thinking; if you’ve attained an advantage then you shouldn’t let it be lost without getting something more valuable in return.

3 Likes
#19

From my perspective this applies across the board to any break in backwards compatibility. The gains from any particular break have to significantly outweigh the loss for end users.

2 Likes
#20

Thank you for your clarifications Jeremy. The consequential thoughts I have are better suited for a separate thread.