Use $:/AdvancedSearch>Filters to create plugins and "importers"

Tips & Tricks
#1

I’m pleased to announce TiddlyTools/Search/SaveAs, that adds “save as plugin” and “save as importer” buttons to the $:/AdvancedSearch>Filters tab.

Start by entering or selecting a filter in the $:/AdvancedSearch>Filters tab. When matching tiddlers are found, “save as plugin” and “save as importer” buttons will appear. Pressing either button displays a popup. In the popup:

  • Enter a new plugin title or press the “list” icon to select an existing plugin title
  • Enter optional plugin fields such as author, name, description, source, etc.
  • You can enable the “add matching tiddlers to existing plugin” checkbox to update the contents of an existing plugin; otherwise, its contents will be replaced
  • Press the “done” (checkmark) button to create the plugin tiddler. If it already exists, you will be asked to confirm before it is overwritten
  • If a plugin tiddler does not contain javascript tiddlers its shadow tiddlers will be immediately available; otherwise, you must save-and-reload for the plugin to be “unpacked” at startup

Alternatively, you can create an “importer” that is similar to a plugin in that it contains a set of tiddlers stored as a JSON object, but works like a snapshot of a TWCore standard $:/Import tiddler.

Instead of installing shadow tiddlers at startup, the set of tiddlers remains “frozen” in the importer tiddler until you view that tiddler and press the “import” button to complete the process and unpack its contents as REAL tiddlers. Note that when an importer tiddler is unpacked, it’s contents are automatically replaced with the same kind of “status” output as the TWCore’s $:/Import tiddler.

Also note that if your filter results include any existing plugins or importers, the contents of those tiddlers will be merged into the new plugin or importer. You can use this feature to combine existing plugins and importers, or convert plugins to importers, or vice versa

enjoy,
-e

8 Likes
JSON Filters of Data in an Array
#2

Comprehensive indeed. Kudos @EricShulman

#3

@EricShulman This is a fantastic utility that will save me a LOT of time. With the recent discussions of using JSON data structures and limitations around filter operators there, I think packaging content as plugins as you’ve made easy here is going to often be the right alternative.

I’m trying to figure out what the checkbox for “add tiddlers to existing plugin” actually does - as it seems to work both with and without that checkbox being used and it’s bugging me not knowing what difference it makes.

#4

Suppose you do a filter search and get one result, “TiddlerX”. Then, you use “save as plugin” to create a new plugin named “MyPlugin”. Since you are creating a new plugin tiddler, it will be empty to start with, so enabling “add tiddlers to existing plugin” has no effect, and the resulting “MyPlugin” tiddler will contain just “TiddlerX”.

Then suppose you do another filter search and get one result, “TiddlerY”, and use “save as plugin” and enter (or select) “MyPlugin” as the target tiddler.

Without enabling the “add tiddlers to existing plugin” checkbox, the current contents of “MyPlugin” will be replaced, and the resulting “MyPlugin” tiddler will contain just “TiddlerY”.

However, if you enable the “add tiddlers to existing plugin” checkbox, the “TiddlerY” search result will be merged with the current contents of “MyPlugin”, and the resulting “MyPlugin” tiddler will contain both “TiddlerX” and “TiddlerY”.

Thus, the effect of the “add tiddlers to existing plugin” checkbox is that it lets you perform a series of separate filter searches and incrementally collect the results into a single plugin tiddler.

-e

#5

Ah - I feel silly now, for some reason it hadn’t even occurred to me to try it without the plugin itself or it’s shadow tiddlers as part of the filter. Makes perfect sense.

#6

As you’ve noted, you can also add to an existing plugin tiddler simply by including the plugin tiddler title as part of the filter itself. However, for some complex filters, that might not be possible, or at least not practical.

Also, if you are using a saved filter definition, you might not remember to add the name of the plugin tiddler to the end of the filter, so writing to the existing plugin will wind up discarding all the previously packed tiddlers.

-e

#7

Two small updates to TiddlyTools/Serch/SaveAs:

  • when generating the target tiddler, use format:json<tab> to “pretty-print” the JSON output
  • if “[x] add to existing…” is checked and the target tiddler does not yet exist, prevent it from adding an “empty” item in the target tiddler’s JSON output

-e

1 Like
#8

For all Something to investigate “Tiddler accumulation”

I have had discussions with @EricShulman on the “saveas tool”, I thank him for devloping a tool to do something that I feel has long being neglected in tiddlywiki. As @stobot points out we can now at least add tiddlywikis JSON format for tiddlers as a useful resource with ease.

  • Now I hope to simplfy the operators to access these

I just though I would mention a method that interests me that is now possible. Tiddler accumulation.

Drop a plugin on a tiddler then using “saveas solution” to copy in a set of tiddlers into a plugin/JSON you want to accumulate, export the resulting plugin and (optionaly) delete from the current wiki.

  • Depending on the circumstances you may want to move or delete the tiddlers you accumulated into the plugin (or JSON), this is easy given the delete option is available in Advanced search filter tab.

Now import to another wiki and do the same. You could do this indefinatly to maintain a collection that can be deployed anywhere based on an accumulation of tiddlers.

  • You could tag or flag tiddlers to earmark them for moving or copying to a “collection” as they arrise, an example may be functions and helpers you develop over time, I have hundreds of those.
  • You could use this to progressivly develop a plugin of your own, progressivly, accross multiple wikis, the key is maintaining a standard where any changes are exported to a single filename which acts as the accumulated source of truth. Also consider the plugin version number, and before you make or apply any changes you load this first from disk (to capture changes made elsewhere). This is easy because importing the plugin only brings in shadows of any modifications you have already done in the current wiki, and you can even see the differences (see preview).

There is still a gap:
There is currently no way to programaticaly delete a tiddler from within a plugin (if already in the target wiki), other than no longer include it, such that a new version of a plugin can’t see or delete a tiddler it previously delivered and has being edited into a “tiddler”. This can be delt with using Erics tool but will need awarness of the issue.

Tip:
If you are going to start moving filters and other tiddlywiki script into a plugin, keep in mind the tiddlers may then exist as shadows and may need to also apply to shadows+tiddlers.

#9

You can use the following filter to remove “SomeTiddler” from a plugin named “MyPlugin”

[[MyPlugin]plugintiddlers[]] -[[SomeTiddler]]

The first filter run gets the list of tiddlers currently contained in the plugin
The second filter run removes the desired title from the list
Then, make sure the “[x] add tiddlers to existing plugin” checkbox is NOT checked so the current plugin contents are overwritten.

-e

#10
  • I understand that but you cant then delete the tiddler you excluded if you import the plugin in another wiki and that tiddler is already there (from an earlier version)

@EricShulman could you add the following next time you revisit this solution?

Add a button to save to a tiddler as JSON but not an import tiddler (perhaps config option to hide)

A way on finding a plugin, to return its tiddlers eg $:/plugins/PSaT/Global we would then filter as [[$:/plugins/PSaT/Global]plugintiddlers[]] the trick is to get a title from the search results then create and use this filter.

  • For now I will look at making my own list item to offer this option in plugins
  • I have in the past provided a custom item template for advanced search to provide more information.
  • Perhaps a little core tweak would make the item template used here hackable.

Love your work

Post Script. Actualy I will just add a new Avanced Search tab with a custom list item

#11

All “plugin” and “import” tiddlers are already JSON tiddlers (i.e., they have type=application/json) and regardless of any plugin-type field definition, their contents can always be accessed and updated using the TWCore standard json filter operators so there’s really no need to have a separate “save as JSON” feature.

Instead, you can simply create an “import” tiddler (and progressively add more tiddlers to it if you want). Then, when you are done, just edit the import tiddler and delete the plugin-type=import field.

-e

#12

Still, if I do this all the time, I would prefer a button. Perhaps just document where I hack this then. Please :pray:

#13

Since “import” tiddlers are already JSON tiddlers, what is the purpose in removing the plugin-type=import field?

Manually editing its text content would be exactly the same regardless of the plugin-type field value.

The only thing it changes is that when viewed, the JSON tiddler would be displayed as “raw” JSON rather than an import interface.

#14

First just consider answering this question has an open ended answer. What is the purpose of having JSON tiddlers in a tiddlywiki (that are not plugins or importable)?

  • We can retrieve and create a tiddler from the one stored in the JSON
  • We can use operators to retrive lists of the content titles (could be a logic/existance test)
  • We can use operators to retrive fields and field values in such JSON packaged tiddler.

To me its “can we have JSON tiddlers” quickly generated from the advanced search.

  • The answer is yes but a little fiddly.

So what could tiddlers stored in a JSON be used for?

  • A trash bin to store deleted tiddlers
  • Add tiddlers and later export as JSON for distribution to other wikis (import will still trigger)
  • A backup of a tiddler you may wish to restore (plugin may work well here too).
  • Store an Example test data that is deleted as part of testing, restore that data set.
  • A file of tiddlers from which you can do later make into a plugin, make into an import, export as a file and also copy into an import so it is not deleted after.

I have more but dont want to overwhelm

My current project

I am looking at building a robust tool for relationships between tiddlers. This may or may not benifit from hiding tiddlers with details about that relationship in a JSON. More likely as a plugin and use the shadows.

Actually I would like

Access to the internal macros in SaveAs (or build my own), to use in my own tiddlywiki script eg move/add/extract tiddler to JSON

#15

Tiddlers that have a plugin-type=import field are JSON tiddlers, so all of the above actions and use-cases can be performed without needing to remove the plugin-type=import field. The ONLY difference is that when you view an import tiddler in the StoryRiver, it will display the TWCore $:/Import interface rather than showing raw JSON text.

OK… I’ve just re-factored the TiddlyTools/Search/SaveAs code so that the “internal” macro definitions can now be easily accessed by using “\import TiddlyTools/Search/SaveAs”. Of course, if your use-case needs are even just a little bit different from these macro definitions, you’d be better off just copying the code and making whatever changes you need in your own tiddlywiki scripts.

-e

1 Like
#16

The main issue here is on import it is deleted. I want persistent and reinstallable tiddler. Its fine you dont need to incoporate my needs, but I think this option, although trivial may prove to be a common use for SaveAs. I can fork it for myself and those who ask.

Thanks for you help.

#17

I think this is worth raising separately in its own topic, or possibly on GitHub.

#18

I had a question for you @EricShulman:

I understand that you don’t distribute TiddlyTools as plugins mostly as you don’t run Node and find the browser-local publication process cumbersome.

Would you be more open to putting your tools up through the plugin infrastructure, now that you have made it easier for yourself and others?

#19

Perhaps, I was just calling it out here. I have some ideas and I will return when I have a suggested approach.

  • My First being a process before you remove a plugin a tab in the plugin view that lists overwitten tiddlers and allows you to review and delete as needed.
  • Similarly the plugin itself could have a list of tiddlers (including those no longer in the plugin), to test for and recomend review/delete. But only if it is importiant to its operation.
#20

After some more thought, I decided to extend TiddlyTools/Search/SaveAs to permit creation of plain JSON tiddlers (i.e., where the plugin-type field is blank). This new code also handles creation of “theme” tiddlers as well (plugin-type="theme").

Note that the buttons in $:/AdvancedSearch>Filter still only show “save as plugin” and “save as import”. However, when you display the “save as” popup, it now includes a dropdown list with choices for plugin, theme, import, and json so you can pick the type of target tiddler you want to create.

If you want $:/AdvancedSearch>Filter to show a “save as json” button, you can create a new tiddler (e.g., “SaveAsJSON”), tagged with $:/tags/AdvancedSearch/FilterButton, containing:

\import TiddlyTools/Search/SaveAs
<$list filter="[subfilter{$:/temp/advancedsearch}limit[1]]">
<div style="padding:0.5em;"><<saveAs json>>

You can also remove the $:/tags/AdvancedSearch/FilterButton tag from TiddlyTools/Search/SaveAs, so that only the “save as json” button will be displayed.

enjoy,
-e

1 Like