Using tiddlywiki documentation when contained in a plugin

[TW_Tones]

Folks including the big guns including @jeremyruston, @pmario and others.

I am building a way to privately annotate tiddlywiki’s documentation tiddlers in a wiki of my own. I am setting it up so I can accrue my annotations over multiple releases and documentation updates.

• I started this by exporting all non-core tiddlers from tiddlywiki.com to a custom plugin.
• I then imported this into an empty.html
• I updated the standard search to also search non-system shadow tiddlers, so I can find the documentation tiddlers.
• I am building a set of tools, plugins and view templates to make annotating the documentation and looking up code and copying code snippets to the clipboard amongst other things.
  • Some of which I may later propose adding to tiddlywiki.com as documentation updates.

Its all looking really good.

The Problem

• Since the all the documentation tiddlers were written as “tiddlers” the ones that list “tiddlers” do not include shadows, and we get missing information.
• Also if you have a Plugin and edit every tiddler to become a tiddler, you now have twice the storage requirement.

Short term solution and feature?

• Is there a way I could make all tiddlers within my documentation plugin look like tiddlers (not shadow tiddlers) without editing them?
  • A function or filter would be sufficent to get the list of titles
  • In someways it would be a little like skinny or external tiddlers, only they are stored in a plugin, that can be added/removed, enabled or disabled.
• I need to keep them as shadows, so If I do make any edits, I can tell them apart from the supplied ones.
• In a way I need to “rule in as tiddlers” a set of shadow tiddlers, found within a nominated plugin (without editing them).
  • I wonder if it would be sufficient to write a custom widget that overrides the list widget to somehow also include the nominated shadows.
  • I would hope if one of these shadows that are fake tiddlers, has being edited the content of the edited one would be returned.

Long term solution!?

• There would be no impact on the documentation tiddlers if they were modified to list [all[shadows+tiddlers]] because on tiddlywiki.com they are all tiddlers, however if extracted and used elsewhere in a plugin, as I have done, they would also still work correctly.

Bigger picture

• This work stands to allow users to maintain their own notes against a copy of tiddlywiki.com, whilst being able to keep the documentation tiddlers and core at the latest version. This needs the long term solution.
• However if there were a workaround as in the short term solution this feature could be leveraged for additional special features;
  • Archiving tiddlers into a plugin that can be removed and reimported when needed yet made available as “tiddlers” when needed.
  • Provide some new ways to manage wiki editions as a plugin
  • Provide other opportunities for sharing code and data via plugin rather than as functional code solutions.
  • Perhaps this could be used against other JSON tiddlers containing tiddlers within them to allow switchable tiddler data sets.
    • For example I could import the documentation inside a JSON file and the solution would “publish” the tiddlers as if they are separate tiddlers.

[Scott_Sauyet]

@TW_Tones

This work stands to allow users to maintain their own notes against a copy of tiddlywiki.com, whilst being able to keep the documentation tiddlers and core at the latest version.

That would be a great boon. I have no knowledge that will help with this, but I’ll gladly cheer you on.

[jeremyruston]

Hi @TW_Tones @Scott_Sauyet some time ago I experimented with putting the core documentation in a plugin, which you can find in the core plugin library:

I discovered the same issues that you have encountered with the filters in the documentation not picking up shadow tiddlers. I think the best fix for that is just to edit the relevant documentation tiddlers.

It is also very inconvenient that the documentation tiddlers clutter up the user namespace. I would like to investigate making the docs tiddlers be system tiddlers (eg $:/docs/tw/Filters), with a caption field that would be used for display purposes.

A step like that would probably require that the documentation be presented in its own custom story river (like the current “Help” panel).

An excerpt from my notes on this topic:

• Titles of documentation tiddlers and tags have a prefix $:/docs/ or maybe $:/docs/core/, and hierarchical titles that are tightly defined (eg $:/docs/core/widgets/action-setfield or $:/docs/core/concepts/filters/filter-run-prefixes)
  • Need rules about the use of upper vs. lowercase, when to use dashes and when to use spaces (never in titles)
• The root tiddler for a docset (such as $:/docs/core) has information like the name of the documentation source
• A custom search tab that shows results from loaded documentation sources
• A custom view template for documentation tiddlers that presents the caption (and hides the title), the documentation source, adds breadcrumbs based on title structure, and rebases links to work within the $:/docs/<docsource>/ namespace
• How do we deal with:
  • URL anchors being the full ugly URL
  • titles in the “open” sidebar tab being ugly
    • We could introduce a cascade for rendering “pretty links” to tiddlers. Links to docs tiddlers could get a custom icon and style.
• Need to clarify the distinction between tw5-com content that goes into the docs plugin, and that which doesn’t
• Maybe it would be helpful to have a wikitext syntax that introduces additional parameters for scoping links eg [link docs[something]]

[TW_Tones]

@jeremyruston thanks that plugin $:/plugins/tiddlywiki/tw5.com-docs is a useful step to what I need.

• However there are a few tiddlers I don’t want within that package such as $:/StoryList $:/DefaultTiddlers $:/favicon.ico $:/SiteSubtitle $:/SiteTitle $:/SplashScreen etc…

Re your notes:

• By the way I have already investigated many of the ideas in your notes of my own accord, and have working solutions. I can provide more information if you are proceeding with some of this, or interested.
• However it does not introduce the mechanism I am looking for which would;
  • Allow us to avoid editing the existing documentation tiddlers.
  • Let us leverage the mechanism in other use cases, content sharing and plugins.

@jeremyruston

I think the best fix for that is just to edit the relevant documentation tiddlers.

• This would solve the documents issue, but then changes may be quite comprehensive given we need to add [all[shadows+tiddlers]tag[... on all lists.
• My called for mechanism, would eliminate this requirement.

I can see from your notes, you are thinking about placing documentation “to one side as a reference”. For me I want to keep them visible in the user namespace, in fact I want to treat all the subtiddlers in a plugin as if they were tiddlers, without overriding them.

Can you envisage a mechanism that could allow us to nominate plugin(s) by filter and have its subtiddlers appear as tiddlers?

• perhaps a method by which a filter can include shadows in the group we would call tiddlers?

A way to make such a mechanism?

On reflection it may be possible to achieve what I am after by changing the is, all and default filter operators, to include these shadows as tiddlers.

• With the help of others, I recently successfully hacked tiddlywiki, to classify other title prefixes as is[system] which is similar. Effectively to create new system namespaces.
• The mechanism I am now looking for, is possibly it’s complement, to bring selected shadow tiddlers into the non-shadow “tiddlers” set.

Actually I would love to discuss what I have found in this exploration. We could for example use a separate namespace for documentation tiddlers and flick a switch to toggle them between being system, or non-system tiddlers, thus which lists there appear in etc…

• If we had a way to also switch a set of titles from belonging “in the shadows” to being “in the tiddlers set”, it could be very useful, for example;
  • to make plugins and their subtiddlers instantly searchable in the standard search.
  • To nominate system tiddlers such as $:/ControlPanel and $:/AdvancedSearch to come forward into the group of “tiddlers”.
• If we do this we can use the project name “Out of the shadows”

[joshuafontany]

Is there a way I could make all tiddlers within my documentation plugin look like tiddlers (not shadow tiddlers) without editing them?

“Unpack” the shadow tiddlers by DELETING a field that DOES NOT EXIST. (this “touches” all shadow tiddlers in the plugin and then triggers an internal “save tiddler to store” with their existing values.

A simple filter plus a button that targets a field with a gibberish name like ARG5Q (making sure none of your shadows have this field name, lol) would work.

---

OTOH, I really like the idea of packing documentation into the system namespace. the new multi-wiki server’s “bag/recipe” model would work perfectly for this. Doc tiddlers could be “preloaded” by including the docs bag into the recipe, but if you ever edit a docs tiddler, it saves to the “top level bag” (i.e. the one driving the current wiki/definition that the recipe comes from).

[TW_Tones]

@joshuafontany

Unpack” the shadow tiddlers by DELETING a field that DOES NOT EXIST. (this “touches” all shadow tiddlers in the plugin and then triggers an internal “save tiddler to store” with their existing values.

Very interesting approach, but I want to retain the shadows and not cause them to turn into tiddlers.

• If I do then edit them, later I can compare the changes etc… with the shadow version.
• If I change the plugin of tiddlers to a new one it simply replaces the ones I did not edit.

I am deep in the weeds in this and specifically would like to;

@TW_Tones

If we had a way to also switch a set of titles from belonging “in the shadows” to being “in the tiddlers set”

• Whilst keeping them as shadows would be exactly what I need.

The best approach I can see at present is to modify $:/core/modules/filters/is/tiddler.js to access a filter defined in a config tiddler and include the result in the titles considered “tiddlers”.

• Where TiddlyWiki-Documentation is a plugin.

[[TiddlyWiki-Documentation]plugintiddlers[]]

Then any tiddler matching this filter will be listed as a tiddler (not only as a shadow)

• A cascade could be created for this.

[etardiff]

@TW_Tones

If we had a way to also switch a set of titles from belonging “in the shadows” to being “in the tiddlers set”, it could be very useful, for example;

• to make plugins and their subtiddlers instantly searchable in the standard search.

I just wanted to add a +1 to this idea. It does sound a bit counterintuitive at first glance, but I think it could be instrumental to making plugins a viable way to share packages of content, not merely behind-the-scenes tools that many users may never need to look at.

I’d also previously considered packaging content/data “libraries” as plugins for easy update/transfer between wikis, but gave up when I realized a) exactly how many all[tiddlers]... filters I’d have to rewrite, and b) that it would make all packaged tiddlers inaccessible through standard search. In the end, it just wouldn’t have been as convenient as I hoped. Being able to promote a plugin’s contents to a more tiddler-like category would be very useful!

[TW_Tones]

@TW_Tones

$:/core/modules/filters/is/tiddler.js to access a filter defined in a config tiddler and include the result in the titles considered “tiddlers”.

If there are any javascript wizards who can review this rather simple tiddler along with its perhaps more important companion $:/core/modules/filters/all/tiddlers.js to see if we could rule in by filters additional “tiddlers”, ideally using a config tiddler to store this filter.

[joshuafontany]

Functionally, “modifying/editing” a Shadow Tiddler is the same “unpacking” I mentioned. The Shaddow Tiddler in the plugin still exists and can be diffed against, etc. The “current live Tidder” simply is an exact copy of the shadow.

[TW_Tones]

Yes, @joshuafontany I don’t want it unpacked, I want to reserve that unpacking, if and only if, an intentional edit occurs.

• In the mean time I want selected (by filter) shadow tiddlers to be listed along with standard tiddlers using [is[tiddler] and [all[tiddlers].
  • So I do not need to change m/any filters.
• This allows me to still compare an edited tiddler with its shadow tiddler if its edited, otherwise if not edited get it to appear as a tiddler.
• However in this case the vast majority of tiddlers found within the plugin will not be changed. Then when a new version of the plugin is provided all its tiddlers will be updated.
• I will be annotating tiddlers without touching them.

Notes;

I may make the resulting wiki available to the community with all my annotations such as click to copy to keyboard code snippets.

• This mechanism would make it easy to keep it up to date with tiddlywiki.com

I am confident just as existing tiddlers titles can be found and searched etc… and are listed in browser memory, there would be a way to insert others into this list.

[CodaCoder]

I love annotations therefore I love this. Bravo Tones.

[TW_Tones]

I am glad others are interested here.

All that remains for tw document annotation is either finding the the trick to make selected shadow tiddler to apear to be tiddlers (technical) or modify the original documentation tiddlers to include shadows in every filter (fiddly).

Unless we come up with another approach.

@joshuafontany perhaps a variation on your trick to turn them into tiddlers, where we create an empty tiddler with the same title as the shadow, then build a view template or cascade to display the shadow.

• we may still need to include tags and other fields
• not replicating the text field will save bytes and make it clear its not an overwritten shadow.
• [Edited] my working title for this is “false tiddlers”

Any further ideas?

[linonetwo]

@TW_Tones

twice the storage requirement.

I think it won’t be more than 100MB. So don’t worry. Think about my wiki, which is 1GB and 30k tiddlers.

Also, what about store the original uri in the uri field of your note, means you are annotating the resource in that uri? And use a plugin to allow preview the original uri. So you don’t need to include the original source, but still be able to view it.

[TW_Tones]

@linonetwo

I think it won’t be more than 100MB.

This is a single file wiki I am preparing, and the documentation tiddlers are around 7.2MB

• They are full searchable which is important in this application
• However I like your idea, and think it has its place. I will keep it in mind.

I will explore this idea of “false tiddlers”, it is similar to some other tools I made and could have a broader application.

The following is my planned approach only read if you want to follow, or contribute to this project in detail

Basically, I clone the shadow tiddlers and delete the text field. Placing a macro within it that transcludes the shadow tiddler text field.

• Searches can find content in the shadow tiddler but will open its false tiddler. Arguably it is only a “stub tiddler”, but it will look and behave as a full tiddler, only without the text field in it.
• I will always be able to identify “false tiddlers” so if instead I edit a shadow tiddler with a special button, it may first delete the false tiddler, then turn the shadow into a tiddler.
  • This will allow me to preview differences between the edited and shadow tiddlers.
• In the case of the underlying documents plugin changing shadow tiddlers, in a new release, nothing changes for the “false tiddlers”, you just find and see the new version.
  • I will be able to find new tiddlers in the new document plugin, because they will not yet have false tiddlers. A batch process will add them, capture and report the fact they are new (with the created date).
  • My edited tiddlers can then be compared with the new shadows, if necessary.
• Some annotations could be added to “false tiddlers” but maintaining the annotation to not touch the shadow (or false) tiddler remains my objective.
  • I have another idea discussed elsewhere called meta tiddlers, that will allow me to add other tags and tricks without touching the shadow/false or edited tiddlers.

[TW_Tones]

@joshuafontany

A simple filter plus a button that targets a field with a gibberish name like ARG5Q (making sure none of your shadows have this field name, lol) would work.

Rather than a gibberish field I can set the text field to the macro <<false-tiddler>>, this edits the tiddler, adds it to the “tiddlers” and looks like it was the shadow tiddler when viewed.

• All fields except the text will be “cloned”.
• I discuss the idea of false tiddlers in my last reply.

[TW_Tones]

I am continuing research on this;

In my travels I stumbled on this Plugin Types which says;

By default, tiddlers within plugins with custom plugin types will not be registered as shadow tiddlers (the constituent tiddlers can still be addressed as subtiddlers by the TranscludeWidget, ViewWidget or the plugintiddlers Operator). Setting a configuration tiddler $:/config/RegisterPluginType/<plugin-type> to yes will cause the tiddlers within the plugin to be registered as shadow tiddlers.

I could see a version of this setting to allow “custom plugin types be registered as tiddlers”, not shadow tiddlers.

False tiddlers

I have successfully introduced a mechanism I call false tiddlers;

• Import a plugin containing all documentation tiddlers (no core tiddlers)
• Convert them all to tiddlers replacing their content (text field) with a call to a procedure to transclude the text field from the shadow text field.
• These “false tiddlers” are a clone of the shadow tiddler except for the text field.
• The wiki now works correctly because the documentation tiddlers are now all tiddlers.
  • No need to modify all filters within them to refer to [all[shadows+tiddlers]]

Here is my code if anyone is interested;

• Or wait until I share a full solution.

\procedure false-tiddler(plugin target)
<$transclude tiddler=<<plugin>> subtiddler=<<target>> mode=block/>
\end
\define new-text-field() <<false-tiddler """$(plugin)$""" """$(currentTiddler)$""">>
\function target() [[ActionWidget Execution Modes]]

<$let plugin="TiddlyWiki-Documentation">
<$link to=<<plugin>>/><br>

<$button>
<$list filter="[<plugin>plugintiddlers[]]">
  <$action-setfield $field=object-type $value=false-tiddler $timestamp=no/>
  <$action-setfield $field=text $value=<<new-text-field>> $timestamp=no/>
</$list>
Convert <<plugin>> tiddlers to false tiddlers
</$button>
<br>
<$button>
<$list filter="[<plugin>plugintiddlers[]]">
  <$action-deletetiddler $tiddler=<<currentTiddler>>/>
</$list>
Delete - overridden shadow tiddlers in <<plugin>> plugin 
</$button>

• Use shadow/tiddler default mechanisms to edit each documentation tiddler and replace the text field with a macro to transclude the shadows text field
  • The “tiddlers” will consume a fraction of the original plugins size since the bulk of content is held in text fields and this is the only part we do not copy.
• We include tiddlername literally in procedure call/transclusion for when current tiddler is not valid? such as in the sidebar.
• Timestamps=no means the false tiddlers have the same dates as the tiddlers they replace keeping this information available, such as in recent.
• With false tiddlers an independent object type we can distinguish them from regular tiddlers, their source and if they were standard edited shadows.

[Post Script] Consider the fact that more than one theme and language plugin can be present in a wiki with the same tiddler titles within them, and only one active at a time. What about different sets of external/imported tiddlers?

[Scott_Sauyet]

I just want to announce my encouragement. I have nothing to add to this work at this time, but I’m very impressed. I can’t wait to see where this goes!

[TW_Tones]

Thanks @Scott_Sauyet that helps. I will add some of my existing annotation tools to my annotated tiddlywiki.com before publishing on tiddlyhost.

As simple example see this widgetMessage tiddler which has an automatic annotation that provides “click to copy code” for the two forms messages are used in.

• Here you are looking at one of my false tiddlers.

[TW_Tones]

Just a further update, I have the false tiddlers concept working nicely and will soon move on to choosing “no touch tiddler annotations” and would invite those interested to raise the kind of content they would like to see as annotations, or any particular ways you would like to annotate.

• In the near future I will place this on a public tiddlyhost.