[TW2036] Planning for v5.4.0

#1

TL;DR

This post picks up from the initial Project 2036 post to go into more detail about the proposed v5.4.0 release.

  1. The proposed v5.4.0 release is focussed on improvements that require small breaks in backwards compatibility
  2. The primary goal is to clear the existing backlog of new features that have been held up due to backwards compatibility concerns
  3. A secondary goal is to add the groundwork for tentpole new features that have become table stakes amongst our competitors
  4. We will trim the functionality that is included in this release so that we can achieve a mutually agreed target release date
  5. We would welcome feedback on these plans, including the specific features to be included

Most importantly, this is not a call to compile a general wish-list of new features. To get anything done we need to focus on establishing and achieving much tighter and more specific goals for this release.

Introduction

A unique thing about TiddlyWiki is the value that we put on making it future proof. We want users to be confident that their TiddlyWiki data will be under their control and freely accessible for as long as it is needed. We also want 3rd party plugins and templates to continue to work without requiring constant maintenance. We achieve all of this by being rigorous in maintaining backwards compatibility.

At the same time, we want to keep improving TiddlyWiki so that it becomes more usable, flexible and powerful over time, and hence more useful. That process of continuous improvement is arguably what drives a lot of community activity: discussion with users influences the direction of TiddlyWiki, and users can choose to contribute directly to improving TiddlyWiki.

However, there is a clear tension between these two goals. There are many perfectly reasonable improvements that we cannot make without minor breaks in backwards compatibility.

Accordingly, if you follow the development of TiddlyWiki on GitHub you will have noticed that there are quite a few worthwhile improvements that have been held back on the grounds that they break backwards compatibility. Examples include:

  • Allowing comments within filters and between widget attributes
  • Improvements needed to TiddlyWiki’s support for bidirectional scripts

Clearing this backlog of potential improvements is the motivation for a new release of TiddlyWiki that judiciously and surgically makes some modest breaks in backwards compatibility. From the perspective of end users, this will provide a wealth of welcome improvements and fixes. For developers the result will be a cleaner TiddlyWiki that is easier to work with.

Criteria for inclusion in v5.4.0

This list almost entirely comprises new features and updates that have been held back due to backwards compatibility concerns. To maintain momentum, to be considered for v5.4.0 these PRs need to be essentially ready to be merged.

For example, 5 years ago we did some experiments that showed that there would be only a very minor practical impact from making the default behaviour be to allow duplicates within filter expressions. This would clear up a significant pain point, particularly for those heavily using the maths operators but it would undoubtedly not be fully backwards compatible.

A criterion for breaking backwards compatibility is that the break must be minor, with an accompanying solid argument as to why it should not affect users in the field. If the backwards compatibility break is likely affect existing usage of TiddlyWiki then it belongs in TWX.

I have also proposed a couple of bigger tentpole features that we have never got around to but that are now table stakes for note taking software. In some cases I don’t expect to have the features fully fleshed out in v5.4.0, but I hope we can at least include the groundwork and necessary backwards compatibility breaks. The best example is WYSIWYG editing functionality.

There is also room for improvements that are either trivial to implement, or would be inconvenient to maintain as a long term PR. For example, we don’t have PRs for the improvements listed under streamlining, but they are all relatively straightforward to implement. The key requirement is that the improvements require a break in backwards compatibility.

We must avoid letting this turn into a general wish list for the future of TiddlyWiki; most of that discussion belongs in a separate TWX thread. Here we need to ruthlessly focus on the feature backlog, and these few important new features that require active development.

Schedule

It is too early to be able to figure out a precise date for v5.4.0 because we need more discussion about what should be included. I think the best we can do is agree a rough target based on our collective sense of what feels right. We would then trim and refine the work to fit the timescale.

We also need to discuss whether we should make an interim v5.3.7 release with the updates since v5.3.6 was released in November 2024. The argument would be that v5.3.6 is now roughly the same age as v5.3.5 was when we released v5.3.6. If we defer the outstanding changes until the v5.4.0 release then we’ll have features that were merged back in November/December last year sitting on the shelf for more than 6 months.

For the moment, my proposal would be that we imminently make a v5.3.7 release, and then we shift our entire development effort to v5.4.0. We would still be able to make a v5.3.8 bug fix release if it was needed, but for example the prerelease on the website would be v5.4.0.

Table stakes

These are features that are generally available in other products that are needed in order for TiddlyWiki to remain competitive and up to date. Some of them will require significant development work; the goal for v5.4.0 is to lay the groundwork and make any necessary backwards compatibility breaks.

Existing PRs held back due backwards compatibility concerns

Proposals that breaks backwards compatibility but have no PR

  • Fix the core to not require camel case linking so that it can be optionally disabled by users
  • Ability to use the location hash to customise startup behaviour
  • Use bundled plugins to establish way for plugins to embed their own language plugins
  • Extend import to also import let widgets
  • Default advanced search tab to be filter
  • Default to fluid-fixed
  • Max width for story river, centring
  • Independent version numbering for plugins
  • Comments in between attributes
  • Line breaks within filtered attributes
  • Deprecate all old palettes in favour of those from https://yatagarasu.tiddlyhost.com

Deprecating dangerous, confusing, unnecessary features

All require breaking backwards compatibility

  • Replace safe mode with something more useful
  • Remove old event catcher syntax
  • Deprecate view and reveal widgets
  • Redefine standalone {{{filter}}} syntax to be more useful
  • Deprecate keyboard driven input macro, replace with lower level primitives based on custom widgets

Streamlining

These proposals are concerned with making TiddlyWIki more efficient, with less redundancy.

Internal improvements

  • Adopting a more modern subset of JavaScript

Documentation

  • We will need a new section on tw-com for backwards compatibility, with individual structured tiddlers about each backwards compatibility break
  • A new symbol for the release notes and the docs to indicate backwards compatibility breaks
11 Likes
Reveal Widget Replacement
Project 2036: the future of TiddlyWiki
Project 2036: the future of TiddlyWiki
#3

I would love to see my Simplify permalinks PR or something like it included. I don’t think it’s ready for merging, but the thread died off when some of the ideas presented were outside my current TW knowledge. What might it take to get this ready for inclusion?

Edit: Perhaps this would fall under

  • Ability to use the location hash to customise startup behaviour
#4

If you do intend to remove deduplication by default, would it be difficult to include a per-wiki setting to reinstate the legacy behavior? I understand that the proposed changes aren’t intended to be entirely backwards-compatible, but I make very heavy use of filters in my wikis, and they’re all written under the assumption that deduplication will occur. I’m not sure whether this would truly “break” anything, but it would certainly introduce a lot of redundancy, and it will be a lot of work to add +[unique[]] to hundreds of filters… not to mention any affected filters built into plugins.

#5

I have a question. I have a lot of custom content, and a lot of content that I want to get to. This content is generally speaking more complex, and while it doesn’t use js to get going, it makes heavy use of many of TiddlyWiki’s powerful widget features. So I’m a bit confused now, not sure if I want to keep doing this. If all the stuff I’ve done will become incompatible after version 5.4.0, that might mean that all the work I’ve done now is somewhat wasted. Some of the features that have been removed, is there a corresponding alternative, and what would be required to shift to the alternative. I often use the reveal widget to operate on display content.

The current colour palettes could be saved, only added to and not removed, or converted into an officially maintained plugin. Because some may be used to these colour schemes already.

#6

Really warming to see this post and I like the restrained ambition it represents, which I think is in the general spirit of TiddlyWiki.

I would say it’s worth separating out changes solely held back due to the need to support older devices, versus those that could impact a wiki’s functioning.
That is, where a TiddlyWiki would continue running as expected with no change in user-visible behaviour, as long as it is run on a reasonably modern web browser.

I will focus the rest of my post on making a case for monitoring and acting on known areas where TiddlyWiki is held back from its full potential, due to support for legacy platforms.

At least one notice here is challenging that historic requirement of TW’s development:

Just adding a link to the relevant GitHub discussion.

There is significant performance improvement Jeremy highlights in that discussion, which was withdrawn because of a user running TW on an iPad 3, a 2012 device running an OS version from 2016. The user could have updated to iOS 12 (released 2018 and patched until 2023) according to iOS Ref, but chose to stay at iOS 9. This all happened in 2023.

I want to take a moment to appreciate changes in typical computing/web-browsing environments that have happened in recent years:

  • Microsoft released Windows 10 in 2015 and are offering a free update to the next version for the first time, assuming the hardware supports it.
    Browser support will likely continue well after Windows 10 reaches End-Of-Life status this October.
  • Apple provides software support for iPad and iPhone for 5-7 years cf. endoflife.date, while permitting any device post-2008 to be updated to today’s macOS (examples: iMac, Macbook Air)
  • Android devices are highly variable in terms of the OS version, but actual browsers like Chrome for Android support versions as early as 8, released in 2017. The Play Store continues to work on old devices.
    Android’s WebView component can be updated independently of the OS and Android typically just uses Chrome if that’s available.
  • Chrome OS devices are probably the worst for this, at least before 2021, where I understand it was dependent largely on the manufacturer. Since then, Google has committed to 10 years of updates. Google is still experimenting with having the Chrome browser decoupled from the system
  • Linux users (like yours truly) can solve their own problems :wink:

I will grant this does not cover embedded uses of TiddlyWiki e.g. in an Electron application. This will be context-dependent, and the expectation would be that the vendor should be updating their overall environment alongside TiddlyWiki in such instances. I appreciate this can be difficult.

What I am trying to illustrate is that - in the majority of cases, users will be on the latest version of a browser whether they like it or not, with little stopping them from updating. Not even financial factors are really at play here, in terms of preventing users from updating.

This is not a call to put TiddlyWiki on the bleeding edge, which would be counter to its approach so far. Rather, I am highlighting that users being stuck on old versions of web browsers is becoming rare, due to a mix of longer hardware & OS support cycles, as well as automatic updates in all major browsers.

The most conservative project I know of for browser support is another wiki ironically - MediaWiki (which powers Wikipedia), that ceased support for IE this year. To give you an idea, they supported IE8 (non-JS) until 2020, finally dropping IE support in late 2024.

In TiddlyWiki, Jeremy upheld continued support for IE at least in early 2023, but I’m pleased to see the constraint of IE support likely coming to an end in 5.4.0.

In practice, I think these considerations mainly affect the core, as plugin developers are free to use whatever version of web platform technologies they like, deploying all manner of modern features.

Without committing to anything, I think it is worth identifying browser/platform support-related issues, and assessing benefit against harm it might cause to adopt these new features.
These issues should be flagged and regularly reviewed in case there are few enough users to justify breaking compatibility.

1 Like
#7

Adding to the discussion, another question is will the extra p-tags be dealt with? If it’s an incompatible update, then this issue might be able to be resolved.

Will some modern UI components be provided in this update that don’t require end-users to manipulate JS, just use a micro-widget like edit-text, but are more modern, aesthetically pleasing and implemented. Well, this might just be my personal wish list, but I really like TiddlyWiki and I always think of TiddlyWiki whenever I need to use an interactive tool to save my content.

P tags where I do not want them

#8

The idea from the list above is that there are only very minor breaks to backward compatibility. You will have to test your wiki to be sure it doesn’t fall into one of those cases, but that should be quite rare.

And of course, you can stay on 5.3.x as long as you choose.

#9

I hope so, I’ll continue to do the work I want to do in TiddlyWiki, and if I update to version 5.4.0 after that, I’ll also use the latest version and try to fix any problems or crashes that may exist in there.

#10

IMO dropping support for Internet Explorer is a matter of importance for v5.4.0. Now I think uglifying is also useful for tiddlywiki (perhaps we may introduce @Flibbles 's uglifier to it)

This one is important, too. If implememted, it will allow us to use indirect parameters in filters for JS macros.

#11

I find this thread interested reading wrt to wide variety to interests.

My two-cents worth.

I would like to see better debugging. I know you can use the $log widget and a browser’s inspect element facility but some way of tracing code execution coupled with a method of inspecting data elements, hopefully in a seperate window or frame, would be great.

bobj

#12

Thanks Jeremy for providing a very sound and exciting vision for TW ! I’m keen to look at ways to contribute to this.

One thought I’ve had about backwards compatibility and new features is that the plugin mechanism is an enornamous asset in TW’s favour - potentially “soon to be” or fully depreciated features like RevealWidget could be shelved into a seperate plugin that users could put back in if they need it. I also think plugins should be used more for feature requests to allow the core to stay stable and streamlined.

That said I think a better mechanism for the plugin library(s) is needed (along line of the CPL) so that every user can easily find useful plugins beyond the official ones and yet they are less centralised with the core. Also needed then is some processes for verifying/endorsing plugins (for security) and publishing a stable plugin as part of the endorsed library that developers an follow.

:smiley:

1 Like
#13

Idea: Modular Approach for TiddlyWiki 5.4.0 Enhancements

One idea that has been discussed before is keeping the core of TiddlyWiki as small as possible, while allowing features to be added as modules or packages (e.g., plugins). This would enable users to customize their experience efficiently without unnecessary bloat.

A possible improvement is having a download page with a default set of options when installing TiddlyWiki, while still allowing users to add more features as needed. This would offer a smooth onboarding experience while preserving flexibility.

Additionally, a few predefined starting editions—such as a personal notebook, digital garden, research journal, or project planner—could make it easier for users to get started without needing to configure everything from scratch.

#14

Another idea worth exploring is better integration with modern web technologies, including support for real-time collaboration, offline capabilities, and seamless synchronization across devices. Enhancing the way plugins interact could also improve extensibility, making it easier for developers to create and share new features.

#15

A TiddlyWiki Store: Elevating Plugin Discovery & Integration

My latest idea is to introduce a TiddlyWiki Store—a centralized, user-friendly hub for discovering, installing, and managing plugins, themes, and extensions. While the Community Plugin Library (CPL) has done an outstanding job, it feels like the right moment for TiddlyWiki to take the next step and establish its own integrated store.

Why a TiddlyWiki Store?

A built-in store could streamline plugin management, making it easier for users to browse, search, and install features without diving into manual configurations. It could also:

  • Improve discoverability: Users would have a well-organized marketplace to explore plugins, themes, and custom editions.
  • Enhance security: Verified plugins could be categorized, reducing the risks of untested or outdated extensions.
  • Encourage community contribution: Developers could showcase their work in an intuitive platform, fostering collaboration and innovation.
  • Simplify updates: A store could enable automatic updates and version tracking for installed plugins, reducing maintenance efforts.

Potential Features

  • A categorized plugin and theme repository with user ratings and reviews.
  • Seamless installation and updates directly within TiddlyWiki.
  • Support for premium plugins to encourage developers to create advanced features.
  • A customization wizard allowing users to assemble tailored editions (e.g., academic journal, personal productivity tool, knowledge garden).
1 Like