[Discussion] How to reduce the size of your Tiddlywiki

#40

I’m surprised. It’s loading the file via a relative path. So it won’t reduce TH’s overall storage load unless behind the scenes the library is a symbolic link to a central library.

You can fix this in your own TW by (while still on TH) editing tiddler

$:/core/save/offline-external-js

Change the macro definition coreURL to:

\define coreURL() https://<your tiddlywiki name>.tiddlyhost.com/tiddlywikicore-5.2.3.js

Where <your tiddlywiki name> is the name of the file that you have externalized (is that a word?). Save. Then download from your TH dashboard. Now you should be able to use that downloaded file as you would a stand-alone file, as long as you have an internet connection.

#41

@Mark_S Seems like the js file is also accessible at this url: https://tiddlyhost.com/tiddlywikicore-5.2.3.js

So no need to specify the wiki name

#42

The question is whether that is an accident or on purpose.

#43

I’ve just pushed an update that is suposed to automatically give everyone the shared https://tiddlyhost.com/tiddlywikicore-5.2.3.js .

(I might switch to https://tiddlyhost.com/tiddlywikicore-5.2.3.min.js soon, which is a little smaller.)

1 Like
#44

@simon With uglify it’s possible to reduce the size of the core to 1,197MB (vs 2MB for tiddlywikicore-5.2.3.min.js)

https://github.com/Telumire/TW/raw/main/tiddlywikicore-5.2.3-uglified.js

#45

Maybe an iframe could be a solution, see Easy way to fix mastodon verification issue (for wiki over 1MB)

#46

Just a follow-up to confirm that it will indeed work on tiddloid with a local library. You need to write the full path to the library. e.g.

/storage/emulated/0/share/tiddlywikicore-5.2.3.js

I can see where it would be handy to have a switch mechanism so you could reset the library location depending where you were using it.

2 Likes
#47

Interesting idea @Mark_S

An Offline switch?

  • But since this file will not change much as soon as you switch to a local copy would you just not use that indefinitely.
  • What I am suggesting is setting something like a cookie in each “location you are using it”.
#48

I find this whole thread fascinating!

As an idiot I could benefit from a simpler run-through of …

  1. The advantages of splitting-out the core / + plugins to … (where?)

  2. How easy would it be, in practice, for an average bear?

IF it improves performance on start-up I’d be likely very happy.
BUT, I slightly have some qualms that core on its lonesome could be a problem, sometimes?

Just queries from a naive end-user
Best, TT

#49

Separating the core and plugins from a wiki is useful in these scenarios :

  • You have a lot of single file wiki with the same plugins / core
  • You want to use the wiki online.

The browser is able to cache external javascript, so the loading time is improved, and a smaller file is also faster to save.

2 Likes
#50

The external JavaScript configuration is useful in certain circumstances, hence its inclusion in the core. But I strongly believe that it is far from a universal solution, and that TiddlyWiki should not present it as a mainstream option.

The problem is that the concept of dependencies between files is difficult for many end users, despite it being a commonplace idea for developers. In fact, it’s not uncommon for web users to only have a hazy notion of files and directories.

For most users, moving from a single file configuration to a pair of dependent files opens the door to a range of fairly horrific bad outcomes when things go wrong with version mismatches and dangling dependencies. That’s bearable for sophisticated users, but it’s not something that we should expect of TiddlyWiki’s mainstream users.

The usability will always be problematic – for example, see the instructions for upgrading a single file wiki with external JS. Even if we smooth out those processes, there’s still a fundamental complexity introduced by shifting so much of a burden to users having to ensure that they keep track of the right tiddlywiki.js files.

As it happens, if we were going to promote a dual file solution for TiddlyWiki I’d be more interested in structuring it differently: a generic “tiddlywiki.html” viewer application that loads/saves from external “tiddlers.json” files. I think that’s closer to a conventional mental model of how applications work, and is more conducive to being packaged as an installable Chrome app etc.

Just to be clear, I am not saying that people shouldn’t use the externalised JavaScript configuration. Far from it, I think we learn a lot from these experiments. And of course it is entirely appropriate for online services like Tiddlyhost or Xememex to use it, where it should be entirely invisible to end users.

6 Likes
#51

It would make sharing / doing backups of specific tiddlers much easier, but is that possible to do without additional software? If so then wont tiddlydesktop become obsolete ?

#52

With the File System Access API you wouldn’t need additional software, though browser support isn’t universal. I have playing around with something similar on WebDAV.

I work a fair bit with users in bandwidth scarce regions where the savings in size would be very beneficial. However, in practice I have found that the usability drop is considerable when moving from a single file to two files that both need to be present, saved and shared etc and easily outweigh the bandwidth savings. Granted, these are also users with limited digital literacy.

1 Like
#53

There are no average bears. But, if you don’t understand the instructions or implications at the top of the wiki here, then Externalising your core probably isn’t for you. Which is alright. Or bearable.

Splitting out the core means rapid fire saves. Especially on hosted services, I always wait for a save to complete before continuing to work. It should also mean faster loads after the first load. At least on desktop chromium, the library file is cached in memory. Which means that reloads are faster. But the cache will be emptied when the browser is shut down. How long it is kept in the cache is entirely up to the browser. If the cache works the same way on a phone, then you should be pulling far less data.

And if you have it set up on your phone to use a local library file, then that’s 2mb of data that doesn’t have to be transferred. Local data is going to be much faster than pulling data over a phone/internet connection.

A service like TiddlyHost could save oodles of space using an external core. And there would be somewhat less bandwidth used since presumably most people are using browsers that will cache the core at least for awhile.

The main downside is that it is probably too complicated for people that don’t understand file dependencies and/or file paths. So, for instance, if you download your externalised TW from TiddlyHost and then turn off your internet connection, your TW file won’t work (unless of course you have a properly downloaded and pathed local library file.)

:snowman: :christmas_tree: :snowman: :christmas_tree: :snowman: :christmas_tree:

1 Like
#54

While I’m cheering with @TiddlyTweeter from the bear hut at a distance, I’m curious if this technique would allow for a “TW administrator” to present a certain “central setup” of a TW, including content, that end users can then locally modify? A use case would be an organization with a central information wiki, e.g policy docs, but where end users can also locally add and manipulate their own content? Maybe admin can even continuously add new stuff (e.g “Todays lunch menu!”) to be accessed upon reload?

Could I even filter what such, um, “pure content wikis” should fetch… like "The core and all tiddlers tagged “Public”?

…and could I use Tiddlyhost as the CDN for my admin wiki and then give empty “content wikis” to my friends or colleagues that, upon load, fetch the core and my stuff, and then they can add their own stuff locally?

Not sure who I’m asking here… except for the TH matter which of course is for @simon :slight_smile:

2 Likes
#55

I think there is massive value in what this offers and I agree with the risks raised, there may be good ways to minimise the potential issues. A “half way house”.

  • Yes and especially for those that host multiple wikis or use multiple online wikis because they could all share the same “cached external javascript”.
  • The more often a cached resource is used, the more likely it will be in the cache.
  • I need more clarification here on plugins. Can plugins be installed in the non-core part and still work?
  • It seems the default should deliver the core and no more, if someone includes plugins, they do so in the part they “control”.
  • External core + Plugins could be published publicly (with version in the name) almost like editions of externalised cores
    • I would like one with some key core plugins (tiddlyWiki internals) and relink, and perhaps some well known plugin Libraries.
    • One with local storage may also be useful.
  • I do not yet know if we can have plugins in both the externalised and user parts, and what if different version plugins are in both?
    • What If I override something in the externalised core?
  • The end user for sure, even for many users, but someone hosting multiple tiddlywikis this is clearly a good option.
  • That is people are likely to be given wikis set up this way even if they do not write them.

I agree, For both @jeremyruston and @Mark_S above, moving from a single file configuration to a pair of dependent files can just be left to advanced user or host.

  • can I suggest there will be cases where dividing the core makes sense and naïve will just be “handed it on a plate” and may never know its in use, and yes, as a result they may face a failure to load they just don’t know how to handle.

We could discourage its use for regular users, to avoid these unnecessary complications, however perhaps the best approach would be to build in the control panel some easy to read information and tools along with some trouble shooting tools for when the host delivers wikis this way.

  • That is make it a little more informative and robust when a user is given such a wiki.
  • Perhaps the content part of the wiki could include a very simple boot and error message when the external JS is missing? and link to trouble shooting.
  • Expose the setting for the external JS file here so people can see what is used, and even select a different CDN (Eg Chinese users inside the great firewall of China) or for when the U.S. disallow Iranians access to GitHub.
  • Allow the wiki to be saved as a standalone, where the external content is pulled in, this may work out of the box.
  • Would there be a way to save the external core(s) locally outside the cache, eg in the local file directory?
    • Then someone with very low device resource could have one externalised core and multiple smaller wikis on their device and/or place less demand loading the wiki from the internet (poor internet connections)?
    • Perhaps select users could Host the externalised core(s) locally, eg an additional option in node or bob @inmysocks
      • Perhaps node tiddlywiki could just include an externalised core by default in its server functions, inspired by @inmysocks including the “Bob saver server running on 127.0.0.1:61192”.
      • However a much simpler server is all that is needed for this since its a single read-only file.
#56

Yes, by using the plugin by cdruan. Demo : Libro — A custom tiddlywiki edition

You can also add plugins internally and they should work.

EDIT: after testing, the internal plugins can’t be from an external plugin library, so you will either need to prepare a js files with all the plugins you need, or install them from another library (CPL library for example).

2 Likes
#57

I love the work by @telumire and @cdruan here! This solution allows me to be able to create a light and rather speedy website including several single file Tiddlywiki (comparing to my old solution of having a 4Mb html per wiki).

So as @jeremyruston explained above, I think average users and developer can benefit from this solution while it may not a good option for other.

Some real examples (suggestion):

  1. In a website every member (like every faculty member) uses a TW as his/her website
  2. In a website of a research institute each project has its own TW file as project page (with its all details, …)
  3. …
#58

As common with TW – User content always wins. So plugins added by the user will be initialized later and therefore will win.

#59

As I wrote … The whole thing is a standard TiddlyWiki. NO extra plugin is needed to install any 3rd party plugin. They still work. You only loose the size advantage. 3rd-party plugins will add size to the html file.

If you want to add 3rd party plugins to the externalized-plugin you’ll need a plugin.

You can import any 3rd party plugin from any 3rd party library and add it to externalized-plugins with the plugin from cdruan. I did just test it.

You can also add 3rd party plugins to a setup with an externalized-core, externalized-plugins and have internal-plugins. … As usual those internal plugins add size to the html-file.

BUT

As this post points out and as Jeremy mentioned. The increased flexibility comes with a cost. Extra management complexity.

2 Likes