Introducing TW5-Graph, Tiddlymap's spiritual successor!

[Flibbles]

TiddlyMap is an amazing and useful piece of software written by Felix Hayashi which has provided graphing capability to TiddlyWiki for over a decade now.

Unfortunately, it is no longer maintained, so it is time for something new.

Introducing TW5-Graph!. The new graphing plugin inspired by TiddlyMap!

It’s designed to be lighter weight and far more flexible and extensible. Currently, it’s still in it’s adolescence, but it should already meet the needs of many people who use TiddlyMap. Why should you consider migrating? Let’s get into it.

TW5-Graph is hackable, customizable, and extensible

Like Tiddlymap, TW5-Graph allows you to use wikitext to create graphs.

TW5-Graph plugin does so by introducing four javascript widgets: <$graph>, <$node>, <$edge>, and <$properties>. Each serves its own purpose, and can work together with the other so that you won’t have to write more than a few lines of wikitext to create what you want.

Meanwhile, TiddlyMap has <$tiddlymap> widget.

And what does that do?

The majority of TiddlyMap is coded in monolithic javascript hidden behind that widget, which provides little opportunity for users to work with the plugin outside it’s preprogrammed lines. As a result, its feature-request list has become long and burdensome, with many people not being able to do exactly what they wanted.

TW5-Graph is far more responsive

It’s faster. It has the same kind of SideBar view and simple graph editing that TiddlyMap has, but has the advantage that any editing you make to your graph takes effect in real time. It not longer has to save and refresh, which allows for faster, more responsive, and frankly more fun, graph configuration.

TW5-Graph is independent of its underlying graph engine

Tiddlymap uses vis-network, which is a fine engine. TW5-Graph uses it as well. However, it’s very old, and hasn’t seen any updates in years. Tiddlymap is stuck with it.

TW5-Graph, however, is designed separately from its underlying engine, so it’s quite capable of using any other graphing engine out there, so long as its wrapped in a plugin which implements a simple graphengine module.

Currently, it also only supports vis-network, mostly so it will provide some familiarity for existing Tiddlymap users, but in time, it can have support for other engines, allowing it to be far more versatile and timeless.

TW5-Graph is lighter weight (No more tmap.id!)

TiddlyMap had a terrible habit of touching every single tiddler in your wiki when it was installed to add a guid called tmap.id. This allowed TiddlyMap to track node titles even if they were renamed, but this caused problems for a lot of people. TW5-Graph does not do this! Instead, TW5-Graph integrates with Relink to manage renaming. As a result, TW5-Graph installs and uninstalls quite cleanly.

Why should you NOT use TW5-Graph??

Because it’s young. I’ve been working hard on this for several months now, but it doesn’t have feature parity with TiddlyMap… yet.

I am actively developing TW5-Graph (and I’m officially stopping my lingering support for TiddlyMap today), so if there are features you need carried over, feel free to open up a feature request. TW5-Graph should be able to grow much more easily than TiddlyMap, not only because of it’s more flexible design, but also because it has a fully-fledged testing framework (unlike TiddlyMap).

Also, currently, I’m releasing it as experimental. Will have most of the outward features done, some internal workings are still subject to change pending user feedback.

Like I said at the start, TiddlyMap was, and still is, an amazing piece of software, but I for one am ready to pass the torch. Here is the link if you missed them up above.

https://flibbles.github.io/tw5-graph/

[TW_Tones]

Thanks @Flibbles I have a deep need to start graphing and see no reason not to start this Journey with your new solution. Thanks for your effort and design approach.

• Perhaps we could include a graph with the plugin that satisfies the model discussed here (or something similar) as this is all about native tiddlywiki relationships and would help people a lot “out of the box” and provide a powerful example for @Sergey_Shishkin
• I have a habit of learning things deeply when I do and always try and push the boundaries by testing all the extreems or deep intergrations. I hope I can do this with TW5-Graph to assist.

[dongrentianyu]

How is the plugin’s performance? I’m not complaining, but when I used a filter that returned around a thousand entries, it took a few seconds, maybe even longer, to display the results. I didn’t actually use this filter expression; I just wanted to test the limits of its performance. In actual use, I feel that the plugin’s performance should be better than I imagined.

[Flibbles]

@dongrentianyu, The performance will be based on whatever underlying graphing engine you’re using. Vis-network starts getting pretty choppy in the high hundreds, and unusable in the thousands.

You might try that same filter with the Memgraph Orb engine provided on the demo site and see if that’s better. Orb is less feature rich, and I haven’t fully fleshed out its adapter, but it’s hyper-efficient compared to Vis-Network. It can deal with several thousand nodes easily. Is this something you would need? Or was this just a curiosity?

@TW_Tones, I’ll hop into that discussion. It seems like a particularly specific use-case, (and I don’t fully understand it yet) but that makes a good case with which to field test this plugin.

As for deep learning, please go right ahead! I would love if someone could see the faults or the bad design choices I’m too close to see.

[Christian_Byron]

This is a great improvement @Flibbles - thank you … looking forward to exploring it more…

How does the interface with the graph engine work ? could you implement it with a translation to svg ?

Cheers
CB

[vuk]

Small bug report:

@Flibbles

Introducing TW5-Graph!

s/sophistocated/sophisticated/ straight on the main page. Or just replace the scary Greek word with the simpler “complex” (pardon the pun! )

[jeremyruston]

Hi @flibbles congratulations, a terrific piece of work that I am sure will be very popular. The demos and the presentation of the site are great and I would encourage everyone to click through and try it out. The functionality is a great advance on TiddlyMap, particularly how you’ve broken the required functionality down into components that can be recombined.

When you’re ready, I would like to promote TW5-Graph on tiddlywiki.com. I have long believed that line-and-blob graphing is a critical capability for TiddlyWiki, and so for a long time TiddlyMap was featured as a thumbnail in the HelloThere tiddler (see this commit).

Is TW5-Graph at a point where you would be comfortable for it to be featured in HelloThere? It could help get a bit more early feedback. We might put a ribbon on the badge saying “ALPHA” or “WIP”. (I would also like to feature it in a future newsletter).

Looking further forward, I would be open to including TW5-Graph in tiddlywiki.com to showcase it. We would want to apply it in a way that was useful to visitors, but perhaps that could be as simple as a graph of the top couple of levels of the existing TOC.

[Xrizzy]

@Flibbles

TW5-Graph is hackable, customizable, and extensible

TW5-Graph is far more responsive

TW5-Graph is independent of its underlying graph engine

TW5-Graph is lighter weight (No more tmap.id!)

I love all of this.

Thank you so much for alll the work you put into this and of course for sharing.

[Peter]

Awesome work!

Can TW5-Graph be used to show a “links to” / “linked from” for the “current” tiddler? I use the CPL echarts plugin for this today.

[dongrentianyu]

I’m just curious, not really in need of it. But generally speaking, I might need to display around a few dozen tiddlers. Because my TiddlyWiki has more and more tiddlers. So performance is still my concern.

[fastfreddy]

fantastic work @Flibbles … This …will be most useful for so many of us!

I am getting an Internal JavaScript Error in Vivaldi (latest) but not in Chrome (latest)… yet they are both Chromium/v8-based and have the same javascript engine version ?!
Any ideas?

Vivaldi’s v8 version:

vs Chrome’s v8 version:

[Mohammad]

This is amazing. I started exploring and learning it, and I really appreciate the design approach and modularity.

@Flibbles

TiddlyMap had a terrible habit of touching every single tiddler in your wiki when it was installed to add a guid called tmap.id. This allowed TiddlyMap to track node titles even if they were renamed, but this caused problems for a lot of people.

This was the main reason I chose not to use the powerful TiddlyMap and stuck with the older, less feature-rich Tid Graph (TW5 plugins! — for TW5 plugin development and installation).

[dongrentianyu]

A somewhat daunting feature request that may not be feasible. Have you considered providing some quick syntax to achieve a generation effect similar to Mermaid diagrams or Graphviz? I think this may be a common requirement. Additionally, the ability to export to SVG or PNG would be useful. I briefly skimmed through the documentation but haven’t reviewed it in detail yet, so I’m not sure if a similar feature has already been proposed.

[twMat]

I’m also getting this. I contacted @Flibbles via dm about it.

[Lamnatos]

@dongrentianyu

Have you considered providing some quick syntax to achieve a generation effect similar to Mermaid diagrams or Graphviz?

Maybe the Bottom-Up Tutorial is close to what you’re describing? Not necessarily “quick syntax” but it looks like you can explicitly define a graph via pure WikiText.

[Lamnatos]

Echoing everyone else, @Flibbles congrats on the release! What a marvelous piece of code this looks to be, I honestly believe it will elevate TW for almost everyone. I’m already thinking of use cases for my current notebooks and having ideas for new ones just from what I saw in your examples.

Also a massive, massive bravo on the beautiful documentation. The depth, breadth and most of all clarity is phenomenal.

[dongrentianyu]

I have read this part of the documentation. However, I am a bit greedy and would prefer a quick syntax that can be written directly. I don’t need so many widget symbols. This is because nodes and edges are not actual tiddlers and field values. Of course, I can use other plugins to achieve this, but I would rather see the potential of TW5-Graph.

However, I am not making any demands, and I do not think that plugin author must implement this or any other function for I wish. I fully appreciate the hard work of the author.

[etardiff]

@Lamnatos

Also a massive, massive bravo on the beautiful documentation. The depth, breadth and most of all clarity is phenomenal.

I can’t second this enough! The bottom-up tutorial is particularly impressive; it makes a powerful and complex tool feel very accessible.

I also really appreciate that you’ve made it all configurable with pure wikitext. I’ll be very glad to say goodbye to the fiddly TiddlyMap GUI.

I haven’t had the time to dig into it as thoroughly as I’d like, but so far, the only TiddlyMap feature I do find myself missing is the search bar that lets you zoom in on a selected node. Is this something you’d consider adding in the future?

[TW_Tones]

@dongrentianyu

ut when I used a filter that returned around a thousand entries, it took a few seconds

Not withstanding a need for performance.

If this is mearly a list of titles there is less value in visualising it beyond a title list, if there is a complex set of relationships between a thousand items a thousand nodes can easily become 10,000 relationships and whilst you may be able to represent this in a graph, the value for communicating with the human visual system and brain is questionable. It seem better to build a solution where we focus on a subset of the total set and navigate from there.

• With large numbers there are other representations such as column graphs, expandable lists, statistical summaries and click to expand that would serve better.

@Peter

Can TW5-Graph be used to show a “links to” / “linked from” for the “current” tiddler? I use the CPL echarts plugin for this today.

See my last reply that references here which includes this idea and more, and @Flibbles is reviewing.

• Somtimes a simple list of titles is best for this kind of relationship.

@dongrentianyu

This is because nodes and edges are not actual tiddlers and field values.

There is an argument that could be put forward that this is under using tiddlywikis capabilities, the tiddler can be like a record in a database and to suggest there is no record suggests you are bypassing a key structure. Sure there will be exceptions and these can be handled like we often do using virtual tiddlers, or titles drawn from a list of some kind that are not actual titles. Including;

• Using any lis as input titles need not have underlying tiddlers
• There are powerful features and tricks available for handling missing tiddlers

Mapping nodes to titles, if they exist or not, perhaps arguably also fields (or their contents) map to relationships, helps keep a tiddlywiki flavor and ease of comprehension.

[Flibbles]

@Christian_Byron, The engines are small plugins that wrap existing public libraries and exposes them using a graphengine module (whose API I’m still finalizing). TW5-Graph is its own plugin independent of these wrappers, and it’s where 90% of my work went into this. One could create an SVG engine which creates and SVG image out of an input of nodes and edges. It could even be interactable, and even animated to some extent, though I expect there won’t be any physics.

@vuk I’ll fix that up typo with the next release. Thanks!

@jeremyruston I would be honored to have this plugin featured in the HelloThere tiddler. It’s already got stability set to “experimental” on the plugin, so that should be clear, but as long as people aren’t hacking my plugin or making their own engines, it’s pretty solid and good-to-go already.

A built-in TOC graph is definitely do-able. And I can have a built in widget for that soon. I haven’t done it yet simply because it’ll just take some work to make it look nice.

@Peter It can show relationships for a current tiddler. The feature already exists as its “live view”, which you might find to your liking. I should probably update the documentation to make this more readily discoverable.

@dongrentianyu A few dozen is perfectly fine for performance. Vis-Network can handle up to a few hundred before it starts getting choppy. Once I get around to finishing up Memgraph Orb as an alternative engine, it will be able to handle several thousand with no problem. But I’m going to polish the Vis-Network plugin to a shine first.

As for emulating Mermaid or GraphVis, it’s very likely I’ll be making a suite of <$graph.*> style widgets which do that stuff very simply. I’m just hoping to get some feedback first to get a sense of what people would like to see.

I recommend opening some feature requests on the TW5-Graph github page for specific requests. We’ll all be able to track them better there.

@fastfreddy Yeah… A lot of people are getting that, but I’m having trouble reproducing it. I’ve got some ideas (It looks like Felix Hayashi, the TiddlyMap creator, may have run into this too). I’ll push out a possible fix in a day or two.

@Mohammad tmap.id was such a problem. Some people discovered that when you mixed TiddlyWiki with lazy loading, it could delete your entire wiki. Those were some very angry bug reports.

@Lamnatos Thank you very much. I’ve always believed that documentation is equally important to the software itself.

@etardiff I’m sure I can do a “zoom to node” feature somehow. I just didn’t flag it as “first release” important. I haven’t looked into how Felix did it for TiddlyMap, and I’m not sure whether it was his own brew, or if it was vis-network specific. But either way, if he did it, I can do it too. If you’d like, you can open a github feature request for it. That’ll help me keep it in mind.

@TW_Tones I agree. In all my projects, when I started to get over 200-300 nodes in a graph, it became cumbersome, even if it was still rendering quickly. Either way, for those who want to have huge visualization, perhaps using more number-crunchy procedurally-generated nodes, Memgraph orb will actually be good for that once I get to polish it. It was meant for super large graphs.