Plugin Compression

#1

I know compression has been discussed to make TiddlyWikis smaller in a more general sense. However, I’m considering a more focussed application.

I have developed plugin integrations for several JavaScript charting libraries. Of the feedback I got, the most common related to how big they were.

The bulk of each plugin was the respective chunk of minimised JavaScript for each library. The only way to make them smaller was to apply some sort of compression.

I had recently been working with a small unzip library for my work, and thought the inflate code would be applicable. I’ve uploaded the result to:

https://www.scss.com.au/family/andrew/tiddlywiki/CompressedPlugins.html

If you download that wiki and search the source for “TinyInflater” you’ll find the minimised inflater code, followed by a wrapper function that is between the inflater and TW code. After that are the two core TW functions I had to modify: “defineTiddlerModules” and “defineShadowModules”.

(Aside: “defineTiddlerModules” processes differently based on “tiddlers.field.type”, while “defineShadowModules” does not - that seems wrong to me? I would have said the switch in the former should be factored out and then called by both?)

The end result is roughly halving the size of every plugin, just by compressing already minimised code.

This would also be applicable to other plugins that incorporate large JavaScript libraries, such as the KaTeX plugin. Compressing just the main katex.min.js saves about 170KB.

Naturally, I think supporting compression in this way would be a worthwhile addition to the core.

King James Bible Wiki
#2

Have you tried Uglify? flibbles/tw5-uglify: Compress TiddlyWiki5 javascript and plugins (github.com)

Uglify is the simplest minimizer in TiddlyWiki. It has a Wizard! Uglifier — compress your tiddlywiki (flibbles.github.io)
just drag your wiki and get the minified one!

You can also use the Node.JS version.

#3

If you like higher compression, you may have a loot at: What about Tiddler compression as a feature like encryption? (LZMA.js) · Jermolene/TiddlyWiki5 · Discussion #7920 (github.com)

Using some rough calculations based on the main page of TiddlyWiki, the numbers came out as follows:

  • TiddlyWiki 5 (5.3.3) = 7.30MB
  • TiddlyWiki 5 Uglified (5.3.3) = 6.20MB
  • TiddlyWiki 5 Uglified + LZMA.js Level 9 (5.3.3) = 3.66MB
#4

Very interesting.

Is there a repository anywhere?
Where does the “deflate” tool come from?
Where can we see the changes to the core?

#5

That’s fantastic. Great work!

The tradeoff in compression is always speed of de/compression versus byte reduction. Do you have any numbers on the time to decompress these plugins?

It’s right there in the module posted, inside boot.js:

  • $tw.utils.TinyInflaterFactory on line 1504 is the actual inflation
  • $tw.utils.uncompressText on 1506 is the TW function that invokes it.
  • tw.Wiki.prototype.defineTiddlerModules (1527) and defineShadowModules (1502) call this function.

Or maybe you were asking only about the deflator? It’s not clear to me whether this is something that would be applied at plugin-creation time or by tools like the Node server upon request. Andrew, how would that work?

#6

The immediate source is my repo: GitHub - andrewgoz-minimize - jsunzip
Which is a fork of: GitHub - operasoftware - jsunzip
Which is a port of: GitHub - jibsen - tinf

So far I’ve been using it with a large number of ZIP files (generated by a Windows app using some third-party library). I have also used it with data generated by Python zlib.compress().

For my demo TW I made use of a JS deflate library: RawDeflate

I just copied the minimised plugin library JS into the edit box here: Deflate Demo then replaced it with the resulting compressed data and added the field “compression=deflate” so my changes to the TW core code would decompress as necessary.

In terms of startup time, a quick comparison with my demo TW versus my original TW showed startup times of about 3100ms versus 2600ms. I could see the “uncompressText” function shown as taking 576ms. Note that with my demo wiki it is decompressing all the plugins with compressed code - about 25 plugins in total. It might be more appropriate to consider just one plugin, such as Chart.js. Its size is about 12% of the total, so 12% of 576ms is 70ms.

I had read the discussion about using LZMA to compress the entire wiki. My first comment to that is that the LZMA is a custom implementation. The inflate code I’m using is implementing the standard ZIP Deflate - RFC1951. That make it more inter-operable with a variety of data producers. My second comment is that compression algorithms work best with larger chunks of data. That counts out compressing individual tiddlers and instead requires either compressing TW as a whole (which make me feel very uneasy), or compressing selected large tiddlers, like I’m proposing. In particular, I was only considering compressing already minimised JS libraries that are 10s or 100s of KB, which already may as well be a large chunk of opaque text.

And just to make it clear, what I was thinking was compression at plugin creation time, so the only baggage carried by TW was the inflater.

#7

Thank you @andrewg_oz. I am not against the idea of supporting compressed plugins, and the deflate library you’ve found seems a good, lean candidate for further exploration.

To be really useful both the inflate and deflate functions need to be part of the core so that those shipping compressed plugins can rely on its presence. We can perhaps cancel out the size of the library by compressing the core plugin itself.

#8

As TiddlyWIki is an open source project, if you had a copy on a thumb drive and was marooned with a rechargeable laptop you could learn all about it by looking at itself. I do see value in compression etc… but I would urge such solutions take into account the possible loss of openness that occurs.

  • This includes searchability of the content.

This could be used to provide visibility of compressed content from a readability perspective, and its openness. For example perhaps we could have a preview that allows us to browse the decompressed content in wiki, and possibly even search within compressed content.

Many decisions were involved leading to what TiddlyWiki is today, perhaps we take this readability for granted, but I urge us not to accidently start hiding things. This does not preclude anything, it just asks that the tools for openness, accompany the tools for hiding/compressing…

#9

When you serve your HTML wiki via webservices, they auto gzip for you. Further gzip maybe useless.

#10

I think for larger single-file wikis, the total size also matters. If large libraries and the tw-core are zipped, the data that needs to be sent over the wire in general is smaller.

I personally prefer an external-core and external-plugins, which completely removes them from the data to be saved to eg: GH-pages or GitLab-pages.

1 Like
#11

I agree.

What I have noticed, is that the compressed plugins on my demo site are still navigable, viewable and debuggable in the browser DevTools in their uncompressed state. The only thing that would make TW code less readable and understandable would be to minimise it.

In light of that, it would be interesting to apply compression to core un-minimised TW JavaScript code only. However, I would like to emphasise that my primary interest is the application of compression to already minimised third-party JavaScript libraries, such as my charting plugins, but also plugins like those for KaTeX and Railroad diagrams. i.e. where the code is already minimised and practically unreadable, and not a fundamental part of TW.

Also, if anyone is wondering where I found the deflater (compression) code I used in my demo, I checked my browser history: GitHub - dankogai/js-deflate: RFC 1951 raw deflate/inflate for JavaScript

The same repo also has an inflater, but I prefer the “tiny” one I’m already using.

#12

I am pretty sure GPL2 license is not compatible with the TW core.

#13

Is the MIT license compatible? If so, the fflate library looks nice. At 32KB its a lot bigger, but is feature-packed with inflate, deflate, zip, gzip, async/sync calls. I’ve updated my test wiki with it.

I have to say, though, that for my original idea of compressed plugins, only inflation is needed, and I’d be quite happy compressing JS manually as required.

#14

IMO we need to have both, so we can dynamically compress pluins if needed. I did have a look at pako, which is part of jsZip, which we have as a plugin already.
In the dis directory it seems to be 40k minified for inflate and deflate.

MIT license is compatible with the core, which is BSD 3 cause.

#15

GitHub - 101arrowz/fflate: High performance (de)compression in an 8kB package is even smaller