Project 2036: the future of TiddlyWiki

#1

For a long time during its early years, TiddlyWiki 5 had the tag line “a reboot of TiddlyWiki for the next 25 years”. I wanted to communicate my commitment to keeping TiddlyWiki going as long as I could, and my desire to architect it in such a way as to make that possible. (At that point, 25 years in the other direction was 1986; not much software from that era was still in common usage apart from Unix, which naturally ended up being an inspiration).

We’re now 11 years until we get to that 25 year mark. What with the passage of time and other developments, I think this is a good time to start talking about where we want TiddlyWiki to be by that point, and to consider the longer term future of TiddlyWiki.

I think there’s a few areas that we might need to look at:

  • I love working on TiddlyWiki with this community so much that I have perhaps ended up working on too many fronts: the core, TiddlyDesktop, the planned TiddlyWiki app for iOS, MWS, and many more minor experiments. I have a relatively large number of open pull requests for core improvements that are under development. I think that to make more progress I need to focus on the core. That means organising ourselves a little more formally so that others can start to form teams and partnerships to take on all the other things that need doing. I am delighted that this has already happened with MWS: @Arlen22 has stepped up to take the lead, and is surging ahead with an improved implementation, working with others in the community to do so
  • Planning a new release of TiddlyWiki in the near future that judiciously and surgically makes some modest breaks in backwards compatibility in order to allow us to bring in some of the improvements that are currently held back by concerns about backwards compatibility. For example, allowing comments within filters and between widget attributes would clearly be very useful, but can only be implemented at the cost of a rather technical and narrow break in backwards compatibility. I’ll write a separate post with some further ideas for what I’m tentatively calling v5.4.0
  • We could do a better job of capturing the underlying motivations, philosophy and vision of TiddlyWiki (or mission, vision and values, if you prefer). My work is driven by beliefs such as “the highest calling of developers is to pass our magical powers onto others”, or “the only way to meet the diverse needs of everybody is to allow deep customisation of the user experience”. Another one is “the most important characteristic of software is the business model; we need business models that are aligned with user needs”. Insofar as ideas like that are drivers for TiddlyWiki I’d like to get them down. However, I believe the perspective of end users matters more than mine: I would like to capture what people value about TiddlyWiki and what makes them care. Making the case for TiddlyWiki is about storytelling, and there are gaps at the moment that make it quite hard to follow
  • I think we also need to start thinking about what comes next: a completely new version of TiddlyWiki that uncompromisingly eschews backwards compatibility in order to be the essence of everything that we’ve learned: the smallest, simplest possible implementation of TiddlyWiki. We’ve called this TWX in the past. For me, a big motivation comes from the desire to remove the cruft and irregularities that obscure the central simplicity of TiddlyWiki. Back in 2011 I still didn’t know what I was doing, and it’s taken all this time since then to fully understand what TiddlyWiki is, and what it could be. I want us to take everything that we’ve learned and apply it to make the best thing we can come up with – and get a prototype standing up before we get to 2036

I am sure there are more things along similar lines that it would be helpful to think about, and would welcome suggestions.

22 Likes
#2

Next 10 years we will observe AI’s exponential growth, PR with AI, review with AI, users expect we should have AI integration, and even AI makes manually taking note not longer necessary.

I think in the coming years, predefined UI will gradually become unnecessary, except for very complex UI like calendar, WYSIWYG editor, whiteboard. AI will help assemble user interface out of void dynamatically . So a simple core is necessary.

For example, Do we even need a $:/ControlPanel UI predefined? Maybe even not necessary, let AI modify the config tiddlers, or write a UI out of nothing when we need it.

From my perspect of view, TiddlyWiki is a graph database. And plugins provides some predefined UI to do CRUD with it. I’m still writing UI with AI’s help, so normal users won’t need to, but in the next few years, normal users will be able to DIY, and discard the UI when it finish the job, because AI token will be so cheap (like what DeepSeek does), discard an AI result will feel like Exhale after Breathe. (That is why I don’t think AI result should be store in the wiki persistently.)

3 Likes
#3

@jeremyruston , I think you are right in going through this exercise of future planning. One person can not do it all. You’ve obviously had the vision and technical/programming skills and knowledge to pull off an amazing environment. I think TW is without doubt, unique in many ways.

Every time I use TW I am reminded of Hypercard. It also was a simple yet powerful metaphor for development. I seem to remember an Apple project that considered a movie file (a set of images played at a determined rate) to be like a Hypercard stack (one card per movie frame). Not sure whatever came of it but it promised programmable movies. Like TW, Hypercard offered a simple yet powerful environment for developing a UI to a data set.

The strength of TW in my mind is cross linking at run time. It is so simple to use the wiki elements to allow the user to cross reference anything with anything. This is both a pro and a con, cause it is easy to ‘get lost in hyperspace’ (see the discussion about the documentation, Using Tiddler Field Values). The hypermedia/wiki community may be able to shed some insight into addressing this.

In backwards compatibility I would suggest TW is without peer and this is important to maintain if possible. I understand aspects of this may need to be sacrificed but lets make sure it is done for the right reasons and not just technical simplification.

I would be happy to assist in this endeavour if and where I can.

@linonetwo , I am unsure what you are proposing. Maybe you can elucidate?

bobj

4 Likes
#4

My 2¢ is that the tiddly wiki “language” needs to become more consistent. Right now there are tags (where attributes are variables), macros/procedures etc. (where variables are passed using a c-line function call, but using ‘:’ instead of ‘=’ as most languages), wiki rules (sometimes variables appear after, like ```) and filters (with a very unique way of work). I think to achieve this one needs to have a proper parser, one that is not based on regexs. This should encompass everything. One obvious way of avoiding a huge parser framework is to use the one already available: javascript (it even has a way of templating, using ``, so the body of a procedure can be wrapped in backtics and evaluated).

While the statement is that the above mechanisms were created so non-developers can enhance TW, I have to say that with 28 years of professional experience in software development, TW is the hardest to develop in. And not just because of the plethora of unique syntax and caveats, but also because there’s no tooling available. E.g. I can’t really debug a filter other than to start placing breakpoints in the javascript. I find myself stepping through core code all the time to try and understand why something doesn’t work.

This can go beyond the above. For example, in Notion, adding a “database” is a UX exercise. While in TW it requires “development” with careful writing of things like filters, data tiddlers and list widgets. I like semantic mediawiki’s approach of treating each link as an edge in an ontology.

I think keeping the single page approach is a must (once you use node.js, you’re competing with many other alternatives), but maybe improve on it by easily allowing saving some things (images) as separate files, or compressing the content and uncompressing only when a tiddler is actually used.

I think a real wysiwyg editor, like in Obsidian (and even MW) is warranted in 2025. This means seeing an actual bullet, and not ‘*’, an actual number, and not ‘#’. Etc. Yes, there’s a preview, but that takes screen space, needs to align scrolling in long texts, and in general is a “developer” flow (write, run) and not a user flow (just write).

2 Likes
#5

Я не программист, по образованию математик, бывший менеджер и архитектор разных проектов автоматизации интеллектуальной деятельности, пользователь интернет со стажем и имею некоторый опыт стратегического развития. А сейчас просто безработный пенсионер, но продолжаю активно учиться в сети и решил программировать, сам писать код. Это решение возникло после того, как я несколько лет с интересом исследую разные компьютерные проекты. Tiddlywiki среди них номер один. Это то, чем я пользуюсь чаще всего, в ежедневном режиме, когда знакомлюсь с новой темой и конспектирую тексты, акцентируя их основные идеи. Не хватает плагинов для интеграции почты и лент из соцсетей и блогов. Создаю свой интеллектуальный профиль, к которому постоянно возвращаюсь для оптимизации и перманентного структурирования. Так вот, для кодирования нужна конкретная задача, которую я себе поставил, исходя из своих текущих представлений. Так возникла идея создания архитектуры и интерфейса, полностью клонирующих Tiddlywiki, но вместо JS, используя другую систему. Выбор пал на проект PicoLisp. Сейчас, естественно, изучаю WАSМ-WASI и в планах переписать интерпретатор PicoLisp, который сейчас компилируется в LLVM. Ну а по пути оптимизировать синтаксис, лексику, которая мне там не нравится. В этой части, считаю лучшим форматом ASON. Параллельно смотрю на развитие проекта STRVCT, как прототип объектной системы будущего. Вот такая цель. Скорее все, что мне не удасться её достичь, тем более одному. Но главное идти … Конечно, я знаком с разными проектами и выделенные, это, исключительно, мои личные представления, но вот так. Можно было бы в связи с этим написать целую книгу, но лучше коротко, а дальнейшие интерпретации могут быть никому и не интересны.

edited by: @Mario
I did use ChatGPT to translate the Russian text.

I’m not a programmer; by education, I’m a mathematician, a former manager and architect of various projects in the automation of intellectual activity, an experienced internet user, and I have some strategic development experience. Now, I’m just an unemployed retiree, but I actively continue to learn online and have decided to start programming and writing code myself.

This decision arose after several years of interest in exploring various computer projects. Among them, TiddlyWiki is number one. It’s what I use most frequently in daily routines when I explore new topics and take notes, highlighting their main ideas. However, I lack plugins for integrating email and feeds from social networks and blogs. I’m creating my intellectual profile, which I constantly revisit for optimization and permanent structuring.

For coding, I needed a specific task that I set for myself based on my current understanding. This led to the idea of creating an architecture and interface fully replicating TiddlyWiki but using a different system instead of JavaScript. The choice fell on the PicoLisp project. Currently, I’m, of course, studying WASM-WASI, and I plan to rewrite the PicoLisp interpreter, which is now compiled in LLVM. Along the way, I intend to optimize the syntax and lexicon, which I don’t like. In this area, I consider ASON the best format.

At the same time, I’m keeping an eye on the development of the STRVCT project as a prototype for the object system of the future. That’s the goal. It’s unlikely I’ll achieve it, especially on my own. But the main thing is to keep going.

Of course, I’m familiar with various projects, and the ones highlighted are exclusively my personal preferences, just like this. One could write a whole book about this, but it’s better to keep it short, and further interpretations might not even interest anyone.

2 Likes
#6

Maybe this can help: Introducing the debug-log[] filter

1 Like
#7

I just think with a consistent wikitext language and a small core (as database), every one can programming with TiddlyWiki. And user only need to validate, AI will do the programming part. Just need to make sure user can debug what AI write.

Database in TiddlyWiki is already there, don’t need to “create database”, but you are right about creating interface for that database. But I believe that will be automated soon.

3 Likes
#8

I am using TW for business and private purpose in so many different use cases.
Now I am at a point that the inconsistency is blocking me more than helping.
The next days I will try to move to MWS and hope that the usage of SQLITE will help me to solve the projects I have in my pipeline.
The future? I really hope that there will be a clear break, may be in version 6, to leave everything behind of old and backwards combability. Yes, my expectation is to have a tool what will highlight all the incombability to help me to solve them but not more.
More and more I realize in all my applications that I need a server based solution more than a local single file solution. So I really hope that in the future TW will also offer an out of the box edition for a raspberry pi. And if I write our of the box than really an edition what is doing nothing else the TW. Download it to the sd card, plug it in the raspberry pi connect it to my router, start it and I can use it. We all have at home enough bandwidth.
And finally, I fully agree with @linonetwo . The future will have more AI interfaces. The searching line will be more and more also the line to order TW to write code. Something like: "Get the first chapter of tiddler A, make a table of all fields of tiddlers with tag abc, sort them descending by field name and filter only … and so on.
Unfortunately I cannot support here in writing code but if you need test results from the field I can support.

1 Like
#9

I think there is a great discrepancy between what TW per se can do in the hands of intermediately-to-highly technical users and how accessible and practical it is to a non-technical user.

I wish we had a hosting service on-par with competition, which would: be easy to set up, accessible over web without installing software/extensions, handle many/large attachments, be encrypted/private by default, have granular sharing options, reliable backups, ability to move between online/offline use. None of the currently available solutions for TW has it all (to be honest, neither does its competition, afaik).
This would go well with a compatible reasonable local way of hosting/saving. TiddlyDesktop seems a big burden to maintain, producing drawbacks of using TW not-though-a-browser, while not really offering much benefit.

Another point is UI/UX. On desktop, one would wish a lot more were possible with keyboard shortcuts. On small and touch screens, TW looses a lot of its usability – workarounds like Notebook theme make it a bit better, but e.g. drag-and-drop reordering of tags is out of reach.

There are surely many other similar things, e.g. WYSIWYG or syntax highlighting/suggesting editor, as already mentioned by @Ittayd.

My main point is, making TW great in itself, and improving it, in “lab conditions” is just as important as “delivering it”: making sure it can be used by non-tech users, on any device, to its whole potential, without worrying about the technical side of saving/backups/attachment sync.

3 Likes
#10

On macOS TiddlyDesktop is a superb tool for managing files and projects, doing what TW in a browser cannot. There are tools to improve the macOS Finder, like PathFinder which I used for several years; but the TW/TiddlyDesktop combination is far superior.

3 Likes
#11

I stand corrected about usefulness of TD then, I didn’t realize it could be useful this way. I hope my main point is understandable nevertheless.

#12

Also, if you have a problem with JS, there are tons of resources. But almost none for TW. Also, parts of TW still lag behind JS. In particular, the filter operators for dates are almost non-existent. The further you live away from Greenwich time, the more you notice this.

#13

I was prompted to take a look back at TiddlyWikiClassic to remember your motivations to launch TW5. Surprisingly I find its still well maintained site with tiiddlers being added even today. So a few points come to my mind:

  • Break compatibility if you feel you must (and looking at some commenst so far I think you may need to). Others will bridge the gap by maintaining TW5 as they have TWC.
  • That said - maybe add some focus at the conversion or plugins needed to consolidate content into the new TWX. Your community of supporters may be split into these different dialects and hence not giving you the scale of a single community needed to keep driving forward as easily.
  • Some analytics for knowing what features are being used and how big your community is may be helpful in driving conversion funcion priorities and keeping the core lightweight (i.e what might be better moved to plugin libraries)
  • As with TWC - I think others will step into maintaining each project you mentioned if you want to step back from ‘so many fronts’ (which is understandable) … However, what is needed most from you is a vision for what those projects are and what the goals of TWX is.
  • Just to be controversial - would you start to commercialise an application while still providing a free/open source tier… Looking briefly at Obsidian - it seems like far less flexible in the markup (e.g. no widgets) and plugins (only via javascript). Is this maybe a MVP for a free version ?
  • I remember at TW5 launch - JQuery was a inspring new tech that helped shape the direction… Now we have more tech like React, JSX ( which could help inspire how to simplify syntax IMHO) and aybe help with building desktop and mobile apps.

:slight_smile:

3 Likes
#14

I think it worthwhile to update the javascript dialect used by the core, so that we can replace some obsolete code and implement features like regex sticky flag. For backward capability, we had better drop support IE in v5.4.0.

#15

… yes! to all that @Bob_Jansen says, especially about Apple HyperCard (and before that, Xerox PARC NoteCards) … and yes! to all that @jeremyruston and everyone else says!

… and from a user-not-developer TiddlyWiki viewpoint (my personal perspective), it might be valuable to gather and share simple “User Stories” to help inspire bits of the future TWX – e.g.:

  • “As a writer, I want to create a new deck of cards with links between them, so that i can organize and share a network of ideas with my readers.”
  • “As a reader, I want to search for cards that mention specific items, so that I can find answers to my questions.”
  • “As a collaborator, I want to add comments and pictures and links to pre-existing decks of cards, so that I can enhance the value of those decks.”
  • “As a novice-level developer, I want to add simple automated features to pre-existing decks of cards, so that I can perform well-defined boring mechanical repetitive actions (such as jumping to a random card with certain characteristics, or counting how many cards have certain characteristics).”

etc, etc – all super-obvious stuff to developers, and all “magical powers” to 99% of the future TiddlyWiki user base! :innocent:

3 Likes
#16

As an ordinary user, I hope that the editing functions can be enhanced. Currently, most note-taking software is evolving towards being more like an editor, allowing users to start editing notes immediately upon opening the software. However, when we open TiddlyWiki, we still need to click the edit button to enter the editing mode. Don’t think that just one extra step doesn’t matter; it can actually be quite annoying. In other words, TiddlyWiki might need to consider this aspect. Because this could potentially be the real issue that breaks TiddlyWiki’s backward compatibility.

Another feature I’m looking forward to is that when we can directly start editing as soon as we open TiddlyWiki, we should be able to write our wikitext, especially macro code, in the editing area, and then have the corresponding functions generated automatically. Just like what Obsidian does now, when you use # followed by some content, you can create a tag, and then a tag button will be generated. When you click on this content, a search interface will pop up immediately, showing other entries that have this tag. I don’t mean that we have to replicate this exact feature. In fact, it’s not difficult to implement such a function with a macro. The challenge lies in achieving this effect within the editing area. In other words, we need a more hybrid editing approach, a way that incorporates the presentation of interface content into the editing process. I think this is why outline note-taking software like logseq or others are becoming popular.

There is also another aspect: we can further reduce the user’s learning cost. No one doubts the power of TiddlyWiki, but its learning cost is really too high, which makes it difficult for other users to stay with it. Using TiddlyWiki is like entering a high-level school for these users. They go through a lot of hardship, learn some skills, and finally leave for other products to shine. I think this is rather strange.

An important aspect of improving the editing method is to optimize the long-form content editing experience. I understand that the product concept of TiddlyWiki is to write content as a combination of different notes. But there is still a wide demand for writing long articles. Currently, I don’t use TiddlyWiki to write long articles. The article I posted on the forum last time was written using Siyuan Notes, not TiddlyWiki.

3 Likes
#17

@dongrentianyu’s comments on writing chunks of text took me back to my AI research days . I got a wonderful book,

Mapping Hypertext: analysis, linkage and display of knowledge for the next generation of on-line text and graphics by Robert Horn from the Lexington Institute, 1989.

The book is written as a hypertext, pages and pages of chunks of text and graphics, all hyperlinked by textual links to other chunks of stuff.

Completely unreadable!

Random access a chunk and follow links, OK, but ‘reading’ as we know it, a waste of time. You get lost in hyperspace very quickly. The undoubted knowledge embedded in the content is scrambled and gets lost very quickly.

TW suffers from this as well as you can see by trying to find stuff in the documentation except the authors have tried very hard to organise the content to varying degrees of success.

Got the book out of my bookshelf today and openened it again to a wonderful world of reminiscing.

If we are anticipating a new version of TW, it might pay the designers to have a look at this book. IT does contain lost of useful snippets related to writing and structuring content, ways that might for a basis for content organisation facilities in the new version.

Still available, seemingly, from Amazon, $461

https://www.amazon.com.au/Mapping-Hypertext-Organization-Generation-Line/dp/0962556505/ref=sr_1_1?crid=3B3LOWBBP9IFQ&dib=eyJ2IjoiMSJ9.rvAG1N2MuHBOA3yk4GZ0x8amuVuap1wQ3IGk_J8Xi2RvJ5DaguuLRfmGORd5l3ljP1HsFaq-GSzoz-5jn6nyyn_-dqpn5b6lUa_R1Z2fwIjVEtbQ-7j7jfvzT29-zRX1.m4hc9_4D8JV2RwRL0T6N3KbWTljdwTA8AnZWxmJ0hEA&dib_tag=se&keywords=mapping+hypertext&qid=1742944053&sprefix=mapping+hypertext%2Caps%2C292&sr=8-1

bobj

#18

Internet Archive has a version available for browsing here: https://archive.org/details/mappinghypertext0000horn

Looks interesting, I’ll give it a look.

TWX in 2036

I rediscovered TiddlyWiki back in December while looking for a platform or framework to make simple data-driven applications. I am in love with it. It meets almost all my needs. If we are thinking about TWX in only 11 years, here’s what I would want to see:

  • Better handling of structured data: I am planning to spend some time with MWS and its SQLite backend. I would love to have SQLite power accessible if possible, either a core function of the framework or via a plugin. That could remove the need for data tiddlers, which @jeremyruston is on record saying were probably a bad idea.
  • Simplify metaphors: mentioned above, there is some cruft around procedures and macros that can be a little confusing. Simplify the fundamental data types and components of the framework as much as possible.
  • Better support for collaboration and external media: MWS is promising and I hope the ideas being tested there are part of TWX. I also saw a thread about embedding external media playback with annotation. I think that’s extremely powerful and would love to see that as a first-class feature

All in all, if TWX never comes to be I’m very happy with what TW5 provides. Here’s to another 25 years!

#19

If we’re talking about breaking compatibility, then the first suggestion from me is either shifting to Markdown as the default markup language, or adding support for Markdown as an fully functional alternative to WikiText.

1 Like
#20

I think this is a topic where TW’s flexibility is both a boon and a bane; which user-base should the future be geared towards?

Developers, from what I can glean as a non-programmer, want something that is more similar to other software solutions.

Product Users who want a mostly ready to use product would benefit from a more modern UI and a little more “linearity” but I think the problem with both is that they can mean different things to different users.

Tinkerers are the most likely to roll with TW’s quirks regardless. My biased view as a tinkerer is that this is the only tool that exists for those without full on developer skills but are still interested in building and not just using, so keeping things open to non-developers is important.

I was alive in the 90’s when the internet came about and remember the chatter about what it could be used for, and then seeing what it turned out to be. I hear the chatter about AI now and I see how the early usage by the general public (non-tech people) is already veering from what the tech people think AI is going to be used like. Your average person isn’t going to be integrating AI with TW for their needs, they’re going to grab a premade fully finished product, possibly one made with TW, but at that point the tinkering and flexibility of TW isn’t what they were looking for in the first place.

My day job is in metal fabrication. I work with everything from modern machining to processes that are fundamentally the same as they were when discovered in The Bronze Age. Even as a non-programmer, I can see that AI in programming is yet another tool and not a means to an end.

Your average person can’t be bothered to run a home 3D printer despite technology being as cheap and easy as it is nowadays, but they will use Amazon Alexa to do shopping. They’re not going to bother with AI beyond finished consumer applications. I think hypothesizing about that potential user-base to attract misses the point of what makes TW unique.

7 Likes