External TW content for many wikis

#6

I recall that in TiddlyWiki classic, it was possible to do a dynamic live import of content from one site to another.

So, I had one tiddlywiki file (hosted on tiddlyspot) that was packed with definitions. I set up all of my other projects to automatically load ALL the “remote” definition tiddlers along with all the local tiddlers.

Benefit: if I modified any definition on the central site – or added new definitions – they would virtually propagate to all the “child” sites. (This would happen only with a re-load; it wasn’t fully dynamic, which would presumably have cobbled performance, if such “push” updates were even possible in theory.)

I don’t recall the mechanics of how that worked. I don’t think anything like that is part of how TW5 functions; many powers were added, and a couple neat features of TWC dropped away…

Anyone remember this – how it worked, and whether it’s really gone forever?

Update: I found a backup of one such TWC site. The function was called “include” and it could invoke a wholesale “import” of tiddlers through a proxy of the other file. Here’s the macro through which (with the right plugin) the ethics site pulled in all (non-system) tiddlers from the reasoningwell site:

<<include "http://ethics.tiddlyspot.com/proxy/reasoningwell.tiddlyspot.com">>

AND! Here’s the still-live webpage explaining the plugin, which was by Udo Borkowski:
https://tiddlywiki.abego-software.de/#[[IncludePlugin%20Documentation]]

-Springer

1 Like
#7

I remember using it too, this may have in part resulted in TW5 on top of Node, because the tiddlers are seperate files. It does say;

included tiddlers are not copied into your main TiddlyWiki. This keeps your individual TiddlyWikis small.

Which in many ways is similar to using an iframe to another wiki.

  • This way you can at least see and interact with content of another wiki
  • I recall if the wiki that is in the iframe is a node/bob wiki it was fine to drag and drop from and to it, even if its open in multiple iframes/wikis.
    • I am just retesting it now.

You can use html versions of include to display content found elsewhere too.

For plugins the best approach is using plugin libraries but we could design better.

File uploads plugin and more may be able to get shared files, I am not sure.

#8

In practice, the IncludePlugin system was much better than an iframe: the “included” tiddlers were available for all linking (including freelinks), transclusion, search, etc. The only interface difference was that the remote tiddlers allowed only a “view” button where internal tiddlers allowed “edit”.

In other words, for the web visitor (who couldn’t edit anyway), there was literally no difference. The presence of “included” tiddlers was seamless – as if they were in the loaded tiddlywiki.

Yet from the authoring vantage-point, there was a significant benefit of having a “write once, read many” way of handling a central store of useful tiddlers. It was ideal for someone who had distinct tiddlywiki projects served up for different audiences (or oneself in different project-contexts), but still some need for overlapping resources.

In my current workflow, by contrast, if I tinker with important tiddlers in one file, I have to keep track of a workflow for propagating that change to others, and it’s easy to lose time in diff comparisons if I can’t immediately jump on opening the other files and dragging newly-improved tiddlers across.

Tiddlyspot hosted all of my TiddlyWiki projects at that time. If there was anything important role played by node.js in making “IncludePlugin” work, it was nothing that affected the end-user.

I’d love to hear from @simon or @saqimtiaz or anyone else familiar with how the TWC nitty-gritty differs from TW5 (or of course @jeremyruston — but only if nobody else can answer readily), as to whether this “include” possibility is definitively foreclosed now. (Freelinks did eventually make it to TW5, and so this difference is the only bit of nostalgia I still carry for TWC!)

1 Like
#9

Depending on the specific use case this model might still be viable.

@Springer the include plugin is only one manifestation of the ability in TWC to pull in content from other wikis and sources that we do not have parity with in TW5. The entire student notebook project from around 2008-2011 relied on a wiki being able to automatically pull updates from a central authoritative source based on the sync/adaptor framework in TWC.

The major stumbling block is actually browser restrictions for cross domain communication rather than the TW5 architecture. The options open to us are to either rely on iframes for cross domain communication as the plugin library does, or rely on hosting the wikis on the same host, or on CORS enabled services (such as Github pages). Using the latter I have wikis setup for a colleague that do pull the latest content from a central wiki, though the content is then also saved in each of the client wikis in that particular use case as read-only tiddlers.

The very first iteration of that concept was posted as a demo a while back: https://groups.google.com/g/tiddlywiki/c/SJkmnGVQVjI/m/6YymkCqOAAAJ

1 Like
#10

This is exciting! Given that all my public-facing wikis are either on tiddlyhost or github, it sounds like perhaps the browser restrictions would not be a deal-breaker for the solution you started sketching out.

I suspect this project would motivate many people to step up with support, so that you can take the idea from proof-of-concept to something sturdy.

And, @saqimtiaz, discovering your gg thread about this only now proves that I never looked up for air in the spring of 2021.

#11

The last couple of years have been chaotic for many of us.

Unfortunately this area of work isn’t something that I am directly focused on at present. Had there been more interest and concrete use cases back when I posted the demo, I may have been able to fold them into the work I was doing then.

I am however working on interacting with remote data sources via HTTP requests from wikitext, along with manipulating the resultant data. This underpins the recent WebDAV work as well. All the primitives needed to enable the workflow you describe already exist but need further testing and improvement. Once this work is mature enough, it should provide the basic wikitext affordances to be able to fetch tiddlers from a remote TiddlyWiki and load them into the current wiki, all from wikitext.

Worth noting here also is that TiddlyHost provides access to just the JSON of the actual wiki content, which would make the loading of remote tiddlers a lot less data intensive than having to load complete TiddlyWiki files in the background, see for example Simon's Tiddlyhost site #1

1 Like
New Features in TiddlyWiki 5.3.0: HTTP request and deserialize operator
#12

@Springer Here is an extremely quick take at a version of that prototype to load tiddlers from a remote wiki utilizing the wikitext based approach I mentioned: Load Remote Tiddlers — prototype

I suspect there might be enough there as a starting point for someone to polish it into usable form.

1 Like
#13

Looking forward to checking this out! Thanks so much, @saqimtiaz!

1 Like
#14

I was able to get a working version, pulling tiddlers from another one of my public hosted tiddlyhost files!

But I got it to work only by cloning your file and uploading it as a starter. (Somehow trying to find and add all the necessary pieces to an empty 5.2.6 starter kept coming up short.)

This would be a fantastic plugin, if someone with the know-how can make sure to pull all the necessary bits from that site together.

-Springer

2 Likes
#15

The requirements are the following tiddlers:

  • StartupActions
  • LoadRemoteTiddlersActions
  • $:/config/SaverFilter
  • $:/publishFilter
  • $:/plugins/sq/webdav-utils/utils/startup-http.js
  • $:/plugins/sq/webdav-utils/utils/http.js
  • $:/plugins/sq/deserialize

I forgot to leave that last one open in the story, otherwise the idea was that the story contains the necessary tiddlers.

A good tweak might be to modify the actions in LoadRemoteTiddlers to tag the imported tiddlers with a system tag like $:/tags/autoimported and then use tag based css rules to hide the edit button on those tiddlers.

I would roll it into a plugin for you now, but I would need to first separate out the HTTP stuff from the webdav-utils plugin and make it a standalone plugin that both the webdav stuff and this code could both use… I might get there in time. :slight_smile:

2 Likes
#16

Note that the startup actions filter out system tiddlers so that they are not loaded from the remote wiki. You could take out that restriction but you will run into timing issues with some startup related tiddlers where they loaded after the wiki was initialized and therefore had no effect.

#17

I’m still figuring out how this works with system tiddlers.

But I’ve updated my site with some linkstyle stuff (thanks to @mklauber) so that the remote tiddlers are clearly flagged as such (though I haven’t yet tried to do anything to suppress the edit button for them).

Also, there’s a tag, load-external, on each of the essential tiddlers you named, plus the ReadMe and LinkStyle tiddler, so that it’s easy to drag all the basics over to another site… though there’s no neat “settings” interface yet for specifying the source url as it appears in the StartupActions, which is the first step anyone would want to do in order to make this useful!

#18

In the tiddler StartupActions, modify the list widget to:

<$list filter="[tag[$:/tags/load-external-wiki]!is[draft]get[text]addsuffix[/tiddlers.json]]" variable="wikiURL">
	<<loadWikiActions>>
</$list>

Then for each wiki you want to load, add a tiddler tagged $:/tags/load-external-wiki with the URL for the TiddlyHost wiki (not the tiddlers.json) in the text field, e.g. https://quick-demo.tiddlyhost.com

Edit: corrected typo in above code.

1 Like
#19

Found a stray angle-bracket there in the filter (in case anyone’s coming to this thread for the steps). You want:

<$list filter="[tag[$:/tags/load-external-wiki]!is[draft]get[text]addsuffix[/tiddlers.json]]" variable="wikiURL">
	<<loadWikiActions>>
</$list>
2 Likes
#20

Hm, please, how I can include tiddler in iframe? It renders whole another wiki interface, not? I not tested it really, and at this time I dont test include TID file in canonical URL field links or IFRAME. How can be iframe content of tiddler from another wiki (not whole wiki UI with sidepanel etc.) inserted to iframe?

#21

Just in case this can give you some ideas: [Idea] Export minimal wiki for iframes - #9 by Charlie_Veniot

And details: [Idea] Export minimal wiki for iframes - #12 by Charlie_Veniot

#22

This reply relates specifically to including content with an iframe. Other answers in this thread are using other approaches. And I raise quite a few above. This is not automatic, but requires interaction, but is one way to improve distribution of content between wikis. Personally my key methods are;

  • Saved bookmarklets containing a package of tiddlers, click to apply
  • Import JSON packages of tiddlers (I have a powerful supporting workflow)

The iframe approach
The iframe will display what ever you ask of course. If the whole wiki is given as the source, or one tiddler within it (with a permalink) the whole wiki is loaded. However like any wiki you view in a regular tab, you can alter what view you get of the wiki, such as a particular tiddler(s) Permalink/Permaview, different default tiddlers, sidebar showing or hidden etc…

  • There are a range of different ways to change what you see when following a link to a tiddlywiki. Starting with a permalink https://tiddlywiki.com/#EditTextWidget that opens the named tiddler.
  • How the wiki is last saved such as hidden sidebar
  • If a “search” parameter is given on the URI, using ?
    (I will try and find example and more details).
  • If there are start up actions that alter the way the wiki starts up

Keep in mind with this load whole wiki method that if you externalise the javascript you will only load your content tiddlers from file (if the javascript is already in the cache)

  • The loading of the whole wiki has a cost, but it also has the benefit of a fully interactive wiki. This enables a lot of smarts to be designed in, in this case inside the iFrame.

If you want only to Load a specific tiddler and not load the whole wiki then that tiddler needs to be “stand alone” there are two obvious ways;

  • Export the tiddler as static HTML and iframe the link to that static tiddler
    • You could write a custom export template to export anything, not just a static tiddler, but that would be more involved.
  • There is a way using a Node Server to have all tiddlers also appear as a “static” tiddler automatically, you just have to address them correctly.
1 Like
#23

Note that with TW 5.30 prerelease, this is now entirely doable with just wikitext:

3 Likes
Uglify is even uglier. New features. Sourcemapping extravaganza
#24

I have packaged the code for loading tiddlers from one wiki into another:
https://saqimtiaz.github.io/tw5-plugins-sandbox/#ExternalContent

1 Like
#25

Impressive as always! I can see all sorts of possible uses.

Note two minor typos in the “Loading content on demand” sample code:

- <$button actions=<<loadMyWiki>> load content </button>
+ <$button actions=<<loadMyWiki>> > load content </$button>
<!--                              ^                ^    -->
1 Like