Creating Libraries

Folks,

Is there a definitive reference or notes anywhere giving the details of a library and the file formats needed?

I would like to build an interactive in single file and node wiki library builder.

I understand;

  • we can use the make library command and node.js to build plugin libraries.
  • Libraries are multiple files hosted at an address plus a library tiddler
  • with the ZIP plugin we can assemble multiple files from templates in tiddlywiki, as I did recently for this Export to Evernote - #10 by TW_Tones

Any guidance, or even reverse engineering of $:/core/modules/commands/savelibrarytiddlers.js would be appreciated.

Bump: Your help please!

Hi @TW_Tones a fairly simple way to see what makes up a TiddlyWiki plugin library is to review the files in GitHub:

If you’re more comfortable browsing on your local file system you can just download a zip of the repo here:

https://github.com/Jermolene/jermolene.github.io/archive/refs/heads/master.zip

The plugin library is currently built under Node.js using some special commands:

"--makelibrary","$:/UpgradeLibrary",
"--savelibrarytiddlers","$:/UpgradeLibrary","[prefix[$:/]] -[[$:/plugins/tiddlywiki/upgrade]] -[[$:/plugins/tiddlywiki/translators]] -[[$:/plugins/tiddlywiki/pluginlibrary]] -[[$:/plugins/tiddlywiki/jasmine]]","recipes/library/tiddlers/","$:/UpgradeLibrary/List",
"--savetiddler","$:/UpgradeLibrary/List","recipes/library/tiddlers.json",
"--rendertiddler","$:/plugins/tiddlywiki/pluginlibrary/library.template.html","index.html","text/plain"

There’s certainly no reason in principle that the process couldn’t operate entirely in the browser; one would just need a very large wiki that contained all of the source plugins.

1 Like

What do you mean by library? Plugin library that user can drag into their wiki to install plugin in a click? Or something like d3.js plugin?

For the plugin library, you can see https://github.com/linonetwo/Tiddlywiki-NodeJS-Github-Template 's config file, it automatically publish plugin library based on plugins folder, and auto release in each push via Github Actions.

For d3.js plugin, I have no idea too.

@jeremyruston thanks for that reference to the core plugin library.

I was wondering if the library build process itself was documented somewhere, I can try and reverse engineer it if not. I understand if it was used by a select few and part of publishing core plugins, as well as part of the build process etc… it may never have had its details documented.

I want to use the library method for distributing my own macros and plugins to myself or to a broader audience, and I am not so well versed in the node commands but I am also keen to make an interactive method for building libraries, including from single file wikis.

@linonetwo I was speaking of the TiddlyWiki library. The one you use to get plugins.

The build process is the sequence of commands that I quoted above. That doesn’t really help because two of those commands are specially made for making plugin libraries, and we don’t have equivalent primitives that work in the browser.

It’s a great idea, and it’ll be interesting to see how far you’re able to get. It’s entirely possible that all the pieces are now in place.

1 Like

I am keen to continue with the idea of an interactive way to build a Plugin Library, primarily because I think it would provide a better way to share plugins and tiddlers, not just to the world but between managed wikis on one desktop or the LAN. Below are some ideas that may help someone with the background knowledge to help send me in the right direction.

  • perhaps you can read through this and contribute?
  • Do you have a published/public library you can zip all the resulting files and send to me? @pmario or @Mohammad perhaps?
    • With that I can possibly reverse engineer this.

I believe I am 90% of the way to a solution.

But first;
Perhaps we could make a wiki, not unlike the upgrade wiki on which you drop a set of plugins and/or tiddlers. It would then construct a library tiddler, the library and the payload files by pushing them into a Zip tiddler and file for download to the libraries URL.

  • It seems that tiddlyWiki now has fundamentally different features since the original library generation function was developed, and it was primarily to build the core plugin library.

I discovered on an install of Bob.exe the plugin $:/plugins/tiddlywiki/pluginlibrary containing the following tiddlers;

  • $:/plugins/tiddlywiki/pluginlibrary/asset-list-json

    • Is only used to set a variable containing a list of asset tiddlers, according to items in a non-existent tiddler $:/UpgradeLibrary/List, I placed two tiddler titles in this tiddler and it seems to recognise them.
  • $:/plugins/tiddlywiki/pluginlibrary/readme

    • Mostly just points to source
  • $:/plugins/tiddlywiki/pluginlibrary/library.template.html

    • transcludes the other two tiddlers (not readme) effectively inside a HTML page.
    • This seems to populate the above HTML page for saving to make the library root html files.
    • If you view as HTML (preview) it seems to be valid, but I don’t know if it needs a specific filename Full HTML copy below
  • $:/plugins/tiddlywiki/pluginlibrary/libraryserver.js

    • The Guts of this appears to be $:/plugins/tiddlywiki/pluginlibrary/libraryserver.js and is presented below.
  • The titles listed in $:/UpgradeLibrary/List appear inside the "rendered HTML as follows;

<code>var assetList = </code>$:/plugins/tiddlywiki/menubar
$:/plugins/tiddlywiki/internals<code>;
</code>
<pre><code>/*\
  • The assetList is used to name the tiddlers.
  • If you trawl through the included JavaScript you can see how it tries to get these “assets”.

What do I need now?

  • I basically need to understand what is normally retrieved from the plugin files, or tiddlers and the necessary files and layout that are included so I can replicate this from my single file wiki.

Reference material for you convivence follows.

  • This is the resulting Library tiddler rendered in HTML
<p>&lt;!doctype html&gt;
&lt;html&gt;
&lt;head&gt;
&lt;meta http-equiv="Content-Type" content="text/html;charset=utf-8" /&gt;
&lt;meta name="application-name" content="TiddlyWiki Plugin Library" /&gt;
&lt;meta name="application-version" content="v0.0.0" /&gt;
&lt;meta name="copyright" content="Copyright 2015 Jeremy Ruston" /&gt;
&lt;link id="faviconLink" rel="shortcut icon" href="favicon.ico"&gt;
&lt;title&gt;Plugin Library&lt;/title&gt;
&lt;script&gt;
<code>var assetList = </code>$:/plugins/tiddlywiki/menubar
$:/plugins/tiddlywiki/internals<code>;
</code>
<pre><code>/*\
title: $:/plugins/tiddlywiki/pluginlibrary/libraryserver.js
type: application/javascript
module-type: library

A simple HTTP-over-window.postMessage implementation of a standard TiddlyWeb-compatible server. It uses real HTTP to load the individual tiddler JSON files.

\*/
(function(){

/*jslint node: true, browser: true */
/*global $tw: false */
"use strict";

// Listen for window messages
window.addEventListener("message",function listener(event){
	console.log("plugin library: Received message from",event.origin);
	console.log("plugin library: Message content",event.data);
	switch(event.data.verb) {
		case "GET":
			if(event.data.url === "recipes/library/tiddlers.json") {
				// Route for recipes/library/tiddlers.json
				event.source.postMessage({
					verb: "GET-RESPONSE",
					status: "200",
					cookies: event.data.cookies,
					url: event.data.url,
					type: "application/json",
					body: JSON.stringify(assetList,null,4)
				},"*");
			} else if(event.data.url.indexOf("recipes/library/tiddlers/") === 0) {
				var url = "recipes/library/tiddlers/" + encodeURIComponent(removePrefix(event.data.url,"recipes/library/tiddlers/"));
				// Route for recipes/library/tiddlers/&lt;uri-encoded-tiddler-title&gt;.json
				httpGet(url,function(err,responseText) {
					if(err) {
						event.source.postMessage({
							verb: "GET-RESPONSE",
							status: "404",
							cookies: event.data.cookies,
							url: event.data.url,
							type: "text/plain",
							body: "Not found"
						},"*");
					} else {
						event.source.postMessage({
							verb: "GET-RESPONSE",
							status: "200",
							cookies: event.data.cookies,
							url: event.data.url,
							type: "application/json",
							body: responseText
						},"*");
					}
				});
			} else {
				event.source.postMessage({
					verb: "GET-RESPONSE",
					status: "404",
					cookies: event.data.cookies,
					url: event.data.url,
					type: "text/plain",
					body: "Not found"
				},"*");
			}
			break;
	}
},false);

// Helper to remove string prefixes
function removePrefix(string,prefix) {
	if(string.indexOf(prefix) === 0) {
		return string.substr(prefix.length);
	} else {
		return string;
	}
}

// Helper for HTTP GET
function httpGet(url,callback) {
	var http = new XMLHttpRequest();
	http.open("GET",url,true);
	http.onreadystatechange = function() {
		if(http.readyState == 4 &amp;&amp; http.status == 200) {
			callback(null,http.responseText);
		}
	};
	http.send();
}

})();
</code></pre>
&lt;/script&gt;
&lt;/head&gt;
&lt;body&gt;</p><p>&lt;h1&gt;HelloThere&lt;/h1&gt;</p><p>&lt;p&gt;This is the TiddlyWiki plugin library. It is not intended to be opened directly in the browser.&lt;/p&gt;</p><p>&lt;p&gt;See &lt;a href="https://tiddlywiki.com/" target="_blank"&gt;https://tiddlywiki.com/&lt;/a&gt; for details of how to install plugins.&lt;/p&gt;</p><p>&lt;/body&gt;
&lt;/html&gt;</p>

Hi @TW_Tones
I use the same procedure as explained by Jeremy!
So, on Node.JS the build library bundles all plugins in your local TiddlyWiki plugin folder into a library!
I then upload it using git to GitHub.

I understand but I have not done that, so I was looking for someone who has, to share the resultant files you upload, so I can reverse engineer it, to build a single file wiki library generator.

Jeremy already shared the files for a plugin library earlier in this thread:

Also I recall that a while back in Google Groups, Jed shared a browser only mechanism for creating plugin libraries. Might be a useful thread to revisit.
Edit: link to the thread in question: https://groups.google.com/g/tiddlywiki/c/Tlje6l8UZiU

If you need a TW based version to create plugins you should use Mohammads Gatha project


As I did rewrite the https://tiddlywiki.com/#Plugins docs tiddlers I also did create a video-series

Talk: New HowTo Playlist: Use Plugins with TiddlyWiki and Build Single-File Wikis using the CLI

Youtube: Use Plugins with TiddlyWiki

It’s a step by step explanation about all things plugins starting from single file wikis up to node-js. How to install Node.js and use it with TW and TW plugins.

As with all of my video series it starts from simple to complex. If you do not view them in the right order you may miss information that may be important. So starting at video 1 is recommended.

**I think this should give all the info required to create your own plugins with node.js – In combination with the TW docs. **

All those files are “auto created” by the node.js library build process. !!! Do not mess with them !!!

Hello pmario,

Unless i have misunderstood TW_Tones need, the issue here is to create a plugin-library (not a single plugin) from the single file version (without using node.js). This subject does not seem to be addressed in Gatha project or your videos-series,

This is something, that I would be really happy to also have.

I have tried Jed solution referenced by saqimtiaz.
It seems to be able to create a tiddlers.json and a index.html text.
Therefore a good start. However it does not seem to generate a zip version of the plugin library including individual tiddlers.
I also do not understand what we need to input in the list item of Jed’s solution.

BR,

Eskha

Hi Tony,

If you have installed TiddlyWiki + Node.JS then do as follow

  1. open command window
  2. issue the below commands
tiddlywiki mytestlib --init pluginlibrary
tiddlywiki --build library

That’s all.

The library from all plugins on your machine (the one you run Tiddlywiki) is created in the output folder of above mytestlib.

Nothing special or to be done from your side!

No, Just about libraries.

Thanks, good stuff, but Not OT here.

I have, and it looks like I will have to publish my own library this way, just to see what results.

I am however trying to see if it is plausible to generate libraries from inside a single file wiki, this seems to be the case, but since there is no documentation that specifies what the “library building process is”, that is, what it does to generate libraries, I have to reverse engineer it.

If I am not wrong my prior post details 95% of the processes, it seems all I need to do is;

  • Create a synthetic tiddlers.json
  • Ensure the plugin files are correctly formatted.

I may be off line most of this weekend.

If you dont want to use node.js you can turn your wiki itself into a plugin library (which probably is not a great idea if it’s a big wiki, but should be fine if you use the external core setup), or use the file upload plugin to generate the required files. This is what I’ve done on PLM — Plugins Lifecycle Management (see library).

If you too are creating plugins on tiddlyhost, then maybe you could avoid the use of the fileupload plugin, as tiddlyhost provide a json link for each tiddler. I did not test this, but maybe this could be a way to remove the need for github and be able to distribute individual plugins, once you set up the correct route.

Note that my library is not perfect: plugin that have dependencies do not show up, and the plugin description in the library plugin list do not display for some reason, but it’s a start.

I have several projects that I procrastinated on I need to finish first (notably a file explorer and a tiddlywiki app), when I’m done with these I’ll try to fix these issues.

2 Likes