Let's make it possible to make plugins installable through npm!

Developers
#1

Edit: I’ve changed this post into a proposal for npm support! See my reply right below this post.

Hey all, I’m trying to set up my plugins so they can be installed through npm. For plugins like Uglify, this could be big, since it would allow it to be integrated into the build process of other plugins, and thus replace things like gulp’s minify.

But I can’t figure out how.

The closest I’ve come is to have Uglify prepare its package so it’s installed node_module (in ./node_modules/tw5-uglify/) looks like:

LICENSE
package.json
README.md
flibbles
    uglify
        plugin.info
        ...

Then I’d have the npm scripts of other projects run tiddlywiki like this: TIDDLYWIKI_PLUGIN_PATH=node_modules/tw5-uglify tiddlywiki --build test or whatever, and the other projects’ tiddlywiki.info would include flibbles/uglify in their plugins object.

But this isn’t great. It requires users to modify their TIDDLYWIKI_PLUGIN_PATH to run stuff, which might be different depending on local or global installations of my Uglify package. All the searching I do and LLMs I ask indicate that tiddlywiki will automatically make node modules detectable for plugins, but this doesn’t appear to be the case. Or at least I’m doing it wrong.

Does anyone know what I’m doing wrong?

Flibbles

Edit: I’ve finished looking through the TW source code, particularly in boot.js. There does not seem to be any consideration for the node_modules directory. LLMs are dirty liars.

2 Likes
#2

Cricket, cricket, huh?

I’ve looked into this a ton, and I’ve concluded that TiddlyWiki absolutely could support the installation of plugins through npm, which would be a great boon to plugin developers working in a Node.js environment.

For instance, consider tw5-uglify. It’s fully capable of integrating into a build environment to optionally produce minified plugin files, but currently it requires developers to checkout the repository and connect it to their TIDDLYWIKI_PLUGIN_PATH environment variable.

Barf.

Instead, if it were installable via npm, one would only need add the following to their package.json file:

"devDependencies":
  "tiddlywiki": "^5.3.0",
  "tw5-uglify": "github:flibbles/tw5-uglify"
}

(or maybe a version like "^v1.10.0" instead of a github reference if I ever put uglify into the npm repository.)

Then, they include the following in their tiddlywiki.info file:

"plugins": [
   ...
   "flibbles/uglify"
]

And that’s it. When they execute npm install, npm will install not only a local tiddlywiki instance, but the required uglify plugin (and whatever other plugins they want).

…We could make this happen very easily.

The way a node_modules directory lays itself out is simple. Here’s an example with the dependencies tiddlywiki and gulp installed:

node_modules/
   .bin/
      tiddlywiki   (symlink to node_modules/tiddlywiki/tiddlywiki.js)
      gulp         (symlink to node_modules/gulp/bin/gulp.js)
      (Other symlinks of other packages. Always a flat directory)
   tiddlywik/
      LICENSE
      package.json
      tiddlywiki.js
      (All other tiddlywiki files)
   gulp/
      LICENSE
      package.json
      bin/
         gulp.js
      (All other gulp files)
   ...

Pretty straight forward, right? It neatly sandboxes every dependency in its own directory, and the package.json files describe symlinks to add to .bin. This makes it all easy to remove later too.

What we can do: Those symlinks don’t have to point to script files. They can point to directories. So I experimented by configuring uglify to install like this:

node_modules/
   .bin/
      flibbles   (symlink to node_modules/tw5-uglify/plugins/)
   tw5-uglify/
      LICENSE
      package.json
      plugins/
         uglify/
            plugin.info
            readme.tid
            (All other plugin files)
         uglify-wizard/
            plugin.info
            (All other files for this supplemental plugin)
   ...

If Tiddlywiki were configured to look at the node_modules/.bin directory, then it will automatically discover ALL TW plugins installed in this manner. For instance, it will find uglify by following flibbles/uglify/plugin.info, which is the directory structure Tiddlywiki expects.

There are only a couple caveats to this.

  1. Tiddlywiki isn’t configured to look in node_modules/.bin automatically, and I think this is a crying shame. When you run anything through npm, it configures the PATH to look in all node_modules/.bin directorys all the way up the directory chain, including in /usr/local/lib/node_modules/.bin where global installaitions go. Why can’t TiddlyWiki do the same?
  2. The current way TiddlyWiki looks for plugins in directories (i.e. (author handle)/(plugin name)/plugin.info), means we can’t install two plugins from the same author. My flibbles symlink above needs to point into Uglify/plugins. It can’t also point into Relink/plugins. But if Tiddlywiki were tweaked such that it could also look for (author handle)_(plugin name)/plugin.info. Basically, use an underscore to merge the two nested directories into one. Then we could have the following layout.
node_modules/
   .bin/
      flibbles_uglify          (symlink to node_modules/tw5-uglify/plugins/uglify)
      flibbles_uglify-wizard   (symlink to node_modules/tw5-uglify/plugins/uglify-wizard)
      flibbles_relink          (symlink to node_modules/tw5-relink/plugins/relink)
      flibbles_relink-titles   (symlink to node_modules/tw5-relink/plugins/relink-titles)
   tw5-uglify/
      LICENSE
      package.json
      plugins/
         uglify/
            plugin.info
            ...
         uglify-wizard/
            plugin.info
            ...
   tw5-relink/
      LICENSE
      package.json
      plugins/
         relink/
            plugin.info
            ...
         relinik-titles/
            plugin.info
            ...

And everything works perfectly.

If anyone knows about my plugins, I have incredibly simple repository layouts that don’t use fancy things like gulp or sass, and yet this npm system is so simple that my plugins (and anyone’s plugins) can fully support this npm structure without introducing any of that. In fact, I can have npm dependencies point to my plugins directly in github, and it works fine. I’ve actually started doing this for TW5-Graphs’s dependency on relink for tests. But this is currently imperfect. You’ll see those package scripts are having to manually add relink for tests with ++ notation.

So how about it, guys? Is this a direction TiddlyWiki would benefit from? Obviously it’d be great for uglify’s adoption, but it could be a huge boon for many other Node.js oriented plugins.

@linonetwo, I’d love your input especially, since you’re the one who encouraged me down this path. And you’re far more familiar with npm than I am.

#3

First of all, I think that is too geek, only a few of TW user can go though it, and we don’t have too many users even compare to Obsidian, not saying Notion. Due to low user count, geek count is even smaller. So I still recommend using CPL or drag&drop on our chinese tutorial , based on discussion.

Secondly, that requires an unpacked plugin folder, which may usually inside /dist folder. But yes, we can use files field | npm Docs to only include plugin folder from dist folder.

Anyway, I don’t think put too much time on it will worth it, because we have CPL or any successor of it. But you could still have a try, there is an long opened issue for this, you could just add additional logic to boot.js on the core, to make it much simpler, It will take less time than explementing things on user side. On an opensource code you could have be adminstrator.

#4

I’m not terribly familiar with CPL, but it’s my understanding it’s more geared toward standalone wikis, not Node.js setups. It functions through the plugin browser inside tiddlywiki itself, right? How would CPL have handled the constraint that Relink is a required plugin for TW5-Graph to run its tests on Node?

I can understand it we choose not to implement this, but I would point out it would not be a matter of wasting time. I looked into it, and I estimate it would be about 6 lines of code, and about an hour of work on my end. The hard part was the research, and I’ve already come out of the rabbit hole.

#5

signing in just to say

wrt : npm ( and package management generally an interesting topic of discussion (no “automated” recommendations to existing threads on this topic as yet) … any non synthetic suggestions?? welcome! … )

_

is the sentiment i wish to convey

.><

#6

CPL is also for nodejs wiki, also for TidGi-desktop which is a multi-nodejs-wiki launcher.

Usually I will use pnpm run dev:wiki (with Modern.tiddlywiki-dev) to start an editable wiki from project’s wiki folder, and open it on browser, install necessary dependency from CPL, and git commit, that is done.

Or simply build a JSON plugin of new version, and move it to project’s wiki/tiddlers/ folder.

That is just how I work with tw plugins. Your npm support is welcome.

#7

Hi @Flibbles that’s a great idea, I’d strongly support it. The core changes seem straightforward.

It is certainly worth exploring whether we should use npm as the officially recommended way to install plugins.

#8

I should clarify that I meant “… the officially recommended way to install plugins on Node.js”.

#9

I love the overall idea!

I’m concerned about Windows, though, where instead of symlinks, there are (Command and Powershell) scripts that call Node with the relevant JS files.

I’m wondering if we might create a new TW Node command like

tiddlywiki path/to/wiki --install-plugin github:flibbles/tw5-relink

that simply downloads and installs the plugin (from github/npm/where else?) in the proper plugin directory.

#10

I prototyped this a while back - supporting installation from github repos as well as URIs for TiddlyWiki files - and mean to pick that work up again post 5.4 release

2 Likes
#11

All right, after considering all the facts, I’ve produced a new solution.

Instead of relying on the “.bin” directory, which doesn’t work because of Windows. We have a dedicated “tiddlywiki” section in package.json which maps plugins to directories (and possibly to json plugin files if we want).

This is a pretty clean implementation. Our plugin packages would have package.json files that look like this. And then end users only need to list that package as a dependency. Then they can specify plugins as normal in their tiddlywiki.info file like usual. No special handling. No TIDDLYWIKI_PLUGIN_PATH manipulation. No digging into node_modules.

It’s clean. It works. I’ve tested it. Check out the PR to see the implementation.

1 Like