The problem with "custom solutions"

#1

People, including me, sometimes propose things that would require modifications to the TW core. A common reply is a recommendation to instead create a plugin or a macro. My post here is not to question this stance; as frustrating as that reply can be, it is fully understandable and TW could not be as incredible as it is without this restrictive policy. Rather, I wonder what can be done to mitigate the problem that follow from non-core solutions, a.k.a custom solutions? Namely, the problem that custom solutions become unreliable!

Presumably, the person requesting something that necessitates a change in the TW core (or the standard distro) is not trying to solve a one time hiccup. Rather, the proposal is estimated to be of repeated usefulness. But a “custom solution” is unreliable in that there’s a high chance that you AGAIN don’t have access to it the next time you need it! - “What was the name of that macro?” or, even worse, “There is no macro… how did I solve it in the code? Where was it?”  Even “Is it in this wiki at all?”

This is actually a big deal. “Custom solutions” must be carried over if you are to access them, but this grows more difficult over time: You have your favourite plugins, your tweaks, customizations and clever solutions . It is a hassle to carry over these things when starting a new wiki and, over the years, chances are that some of your custom tidbits are just lost.

The reply that “Not suitable for core, make a <custom solution> instead.” would also be less frustrating, and there would perhaps be fewer such requests, if we have a good solution to tackle this problem.

Considering how “customization” is TWs super power, maybe we should address this problem in a fundamental way? TW is extraordinarily customizable, perhaps like no other software, so maybe this calls for extraordinary tools to handle customizations, not seen or called for in other software?

5 Likes
Filtering tags (tagging operator)
Critique my implementation of creating a centered layout
#2

I think it’s an interesting discussion point.

There are editions, plug-ins, and more as concepts in TW. These have analogies in many other platforms.

What other platforms don’t have quite as much of is end user customization / programming that TW enables. So there are many more individuals using their own sets of customizations.

Are there existing tools within TW to export “everything that isn’t content” or “everything that is a core override”?

The “problem” of keeping custom solutions up to date is not solvable I don’t think. This is “just” person power of needing to see how your old customizations work with / take advantage of new core options.

If it is more than a single person using some customizations — then I would say how does the group of people wanting it work together to make a plugin or similar solution to share the load of customization over time. This is a pretty classic open source question — how do plugins get maintained?

Am I not seeing this correctly @twMat? Is “maintaining a plugin is too hard” capture what you’re trying to say?

#3

@boris

Is “maintaining a plugin is too hard” capture what you’re trying to say?

I guess it touches it but my point is more general: What I’m after is more what can be done to make it simpler to carry over all desired customized solutions to other wikis. Plugins is probably the easiest thing to carry over, even if there should be a way to e.g DnD a plugin including its overwritten tiddlers. That is one example of why it is tricky to carry over things. But what about, say, local macros, settings, overwritten shadows, etc.

Are there existing tools within TW to export “everything that isn’t content” or “everything that is a core override”?

There are no such tools that I’m aware of but those are good ideas! I did create HackSearch which, among other things, features a listing of “overwrites” and have been thinking that this should feature a single “all overwrites” button that is DnD’able.

Maintaining a plugin might be hard, but I’m pretty certain the absolute majority don’t even know how to create one to begin with. I, for one, only know how to do it by using Tinka. This is also relevant for this discussion; We need a core tool, or at least an official plugin, for making plugins (or bundles). At first it may seem like this is not “of general enough interest” for users but, as reasoned about in my OP, it does concern most users eventually to carry things over into other wikis.

2 Likes
#4

@twMat this is a serious question. Can we say it only arises when a solution needs to modify a core tiddlers in the empty.html?

One part of the “plugin first idea”, so the argument goes, is a plugin prior to the customisation being accepted into core. Even if it ultimately is, incorporated in the core, but we do not hear of that happening often.

In many ways the ideal solution to permit “custom solutions” is to ensure the core is hackable such that any custom solutions that arise can be installed without modifying a core tiddler, encouraging no core change and flexible ways for multiple custom solutions to be installed alongside each other. The recent changes to the view edit templates goes a long way towards this.

It seems to me the core is often, if not always, conflated with empty.html.

The other part is not so much what is core but what is included in empty.html as has being raised a number of times the idea of the empty or minimalist distribution which at the moment is Only core. Quite a few of us have argued in favor of “standard distribution”, which will be new users and everyday users first encounter (rather than the minimalist empty.html) it could have a contents tab, and arguably the relink plugin and a few more tweaks like the more page controls button to ease in new users. It could also include other “trusted” third party plugin libraries. This would be a lower barrier than the core.

We also have editions and package’s of plugins and macros to help “custom solutions”. Happy to explain his further if asked. However unless there is an easy way to publicly publish these they will not proliferate.

I have tacked the problems you voice, with a tiddlywiki containing most plugins I come across (many disabled), another documenting macros and tricks and a large searchable set of json packages with solutions or modules I have built. I now include components with a “just in time” approach. This is fine for me but it is a hidden repository, not public.

I would publish more except there is a barrier to making my own libraries from single file wikis which are my main “format”. I even have nice “draggable” packages of macros, solutions and components.

It seems to me there are a few hurdles to jump over to have custom solutions published to the community, in a central repository.

Finally, I have what I call a “visitor bookmarklet”, that installs and tweaks tiddlers, it also does not demand reload, so I can apply it to online read-only wikis such as tiddlywiki.com. I use it on empty.html files.

What do you think could be the first opportunity?

#5

Any ideas on how to work together to get new core or official plugin features built? Are you interested in leading this effort?

#6

Boris, there are already ways to do this natively, it can already be done without core modifications. Then there is also marios bundler plugin. There are ways to use the import function to easily make JSON bundles of tiddlers. Although it would be nice for this to be a published method with a user interface to support it.

#7

The danger with doing tweaks to core tiddlers is it can be hard to remember exactly what you changed so if you upgrade you have to backtrack what changes you made if things aren’t working / behaving correctly.

Instead if you understand that much of the interface is comprised of tagged tiddlers, you should duplicate the core tiddler you are tweaking, rename it and tag it accordingly.

#8

I think the way your (@twMat) original post read (to me anyways) is a very interesting point of discussion around the health of the plugin infrastructure and the process and rules around plugins vs. core functionality. After some back-and-forth in the thread though it sounds like we’re mainly talking about identifying customizations to be later re-applied.

I forced myself to use the plugin process a few times just to prove to myself I could learn it, but for plugins I build and use myself (which will be nearly all), I find that while this may make others’ life easier, it makes my own harder. So, I usually just now use the “json system”. This is not the proper name, but essentially I rely on the built-in ability of the Filter tab of $:/AdvancedSearch to keep track of what I want and make export/import easier (and because it uses filters, dynamic).

Here’s what I mean (note I don’t often do this, so I might get syntax a bit wrong):

image
image1344Ă—543 36.8 KB

Some common filters worth noting in this context:

  1. [is[shadow]] should get you most “core overwritten tiddlers” like font-family settings etc.
  2. [prefix[$:/stobot]] I use the common convention for my own “settings” to namespace my stuff
  3. [!prefix[$:/temp]!prefix[$:/state]] to remove noise not worth saving
  4. (Not shown) [tag[custom]] if you tagged all of your customization tiddlers with custom this would find them

Note that I’ve highlighted the export button and I use json. From there I can drag-and-drop this back on top of a new empty.html or whatever and then pick-and-choose what to import. In this way, and in the way that these can be easily deleted later, I find is actually more convenient than doing it as a plugin using Tinka or something. Those get loaded in as shadows (I think) which can make things a pain for future customization.

Also of note is that common filters like this already exist in the drop-down of the filter area too. I bring this up as maybe a wiki could be posted here to collect some of the best filters and the “winners” could be baked into future version to make this easier. That might make things easier?

image
image805Ă—710 37.6 KB

4 Likes
#9

As per @stobot I do the same, however I name tiddlers as follows;

$:/PSaT/projectname

  • PSaT is my “brand” short for “People, systems and Things”
  • Then I define tiddlers below that eg $:/PSaT/projectname/macros
  • the project tiddler includes a field “package-filter” of the form “[prefix[$:/PSaT/projectname]]” and any additional filter/tiddlers.
    • This also includes a filename used to export to, and I use a search to find the original to overwrite it in the save as dialogue.
  • I sometimes have a payload-filter as well eg tiddlers created by the projectname eg templates

We can make the package-filter visible in advanced search, however I have a macro that displays an export icon on “project tiddlers” and another that makes a draggable package which can be dragged directly to another wiki.

For smaller packages, I plan to make an additional icon which draggs the tiddlers as a bookmarklet (I have this already in a larger package), just click to install/restore from the bookmarks.

I share this to say along with stobot I have a large repository of folders containing various JSON packages I just import as needed with a simple search. This discussion makes me ask what can we do with a simple online JSON file repository?. We could have user folders in which people contribute there files and a JSON that documents each for inclusion in a community wiki, eg readme.html/index.html (can be a wiki) or readme.json. This would also help reserve names and prefixes eg PSaT would be my folder and prefix, and tiddlers typically named $:/PSaT…

I will test the idea on a cpanel host I have.

1 Like
#10

It is welcome to add your custom solution(plugins) to our experimental CPL GitHub - tiddly-gittly/TiddlyWiki-CPL: TiddlyWiki5 Plugin Library for TiddlyWiki Chinese Communities
It is a nodejs wiki, you can fork and use Tidgi to open it and rise a PR.

Then in your new wiki, your can open the plugin list to pick your plugins and click button to install them.

I myself is using GitHub - tiddly-gittly/Tiddlywiki-NodeJS-Github-Template: Template to generate a auto-sync-to-github, privatcy-in-mind, freely-deployed TiddlyWiki. which use CPL to preinstall many plugins I need in every my new wiki, and it is TidGi’s default wiki template so creating wiki with preinstalled plugin is very easy for me.

3 Likes
#11

Great post! Articulates the issue well. I have to think about it more to be able to write anything useful. So, later.

TT

#12

Me, too.

You’ve just written a belated specification for the existing bundler plugin by @pmario. Take your approach, save the filter to a new bundle and choose it next time you want to export the same bunch of tiddlers.

Lifesaver.

1 Like
#13

You are right. “Filtered Bundles” have the same structure, that the AdvancedSearch : Filter : dropdown uses. So every filtered bundle is automatically listed in AdvancedSearch …

The problem with the core UI at the moment is, that the dropdown can get a bit messy if you have many configurations. … So there is some room for improvements in the core UI too.

I personally use a little prefix in the description of the filters, to make them stand out. eg: -- description text

grafik
grafik1300Ă—1098 66.7 KB

3 Likes
#14

True.

Mine’s bigger than yours:

--- description

:stuck_out_tongue_winking_eye:

1 Like
#15

Dragging this back to the discussion at hand…

@twMat @pmario’s bundler plugin does go a long way toward addressing some of your concerns – but not all of them.

Nailed.

Nailed again.

This is where search needs improving PLUS adding searchable text that you can predict you might search for later.

In regular tiddlers, you can do the latter inside any div with display:none

<div style="display:none">
useful search terms here that aren't necessarily covered by the title or tags
</div>

I actually use a dedicated custom element purely for (what I call) search clouds:

<search-cloud>
useful search terms here that aren't necessarily covered by the title or tags
</search-cloud>

CSS:

search-cloud { display:none; }

As for the search itself, I use @Mark_S’s regex stuff that splits tiddler text on a search term in a ton of ad hoc macros I use all over the place. They’re fast and reliable (much faster than the dynannotate plugin which I switched to for a while then switched back due to the performance issue).

And when it comes to rendered output, I add invisible stuff to the HTML that appears in the dom inspector:

<debug noise="something indicative of what's happening here to remind me later" />

I can’t cover all your points, Mat, but there are ways and means to cover some of them without resorting to yet more plugins.

#16

I think the OP is slightly related to a discussion at gitHub from February this year: A new type of plugins for distributing contents like course notes, presentations, ordinary bundle of tiddlers · Discussion #5462 · Jermolene/TiddlyWiki5 · GitHub where @Mohammad suggested a possibility to publish “content as plugins” …

I think it tries to tackle the same “management problem” to make information more accessible for “end users”.

I think it’s not necessarily a core problem. … but needs help from the core

3 Likes
#17

I feel this thread so hard, lol. My JsonMangler plugin is stuck back in 5.1.23 (? I’d have to check), and while it didn’t get very widely adopted, those who have used it have felt the pain of it not being compatible with the current 5.2.* releases. This is because I chose to literally patch functions in the $:/core during startup, and this is a terribly brittle thing that doesn’t survive a core TW release unless I’m right there to update the patch-tiddlers and do bug testing.

2 Likes
#18

Yes. With so many projects, each with distinct customizations, and widely varying degrees of urgency for keeping up with upgrades to core, I often struggle to track what’s where and why.

One small point to add is that there’s an analogous challenge with edits to plugins. For example, I always hack telmiger’s otherwise-lovely details plugin, and also Shiraz, so that the colors come from palettes rather than being locked into hexademical values. I also always modify Shiraz (as well as the core’s language hint) to remove “Missing” and “empty” messages, so that ViewTemplate elements don’t waste space and attention when they’re not needed. Each time I upgrade a plugin (as well as the core), there’s the slow reconstruction of what needs fixing.

Something that automatically tracks not just WHICH shadow tiddlers have changed, but gives an instant overview of where and how, would be great. Of course, it would be a “best practice” for me to have some consistent flag, across wikis, for such edits. Perhaps I should add a custom “override-explanation” field – the contents of which can be neatly viewed with a Shiraz dynamic table. :wink:

-Springer

1 Like
#19

It will be interesting have a tool for custom solutions but I see one small and one big doubts/problems.

  • The first is having a source tiddlywiki for create the bundles.
  • The second is how avoid problems when several tiddlers do changes in the same core tiddler or other customizations that “overwrite” the anterior solutions (for example, in CSS)

A great power comes great responsability.

I think that something similar how the filter advancedsearch works and the button to export. Having list/s with checkbox to add tiddler to a temp tiddler. I don’t know if it is better a direct filter search (how @stobots does) or create tabs with custom list, basically they are the same but with different UI concept.

#20

I feel we need some extencions to help us all handle selection in the face of larger lists. Tiddlywiki.com demonstrates grouping well but we need to support interactive filtering.

Tiddlywiki has strengths in listing and selection which allows us to push it along way. As a result we do push it along way with large and sophisticated lists and relationships. As a result we hit new barriers to interactive selection and it is there we need to improve the options. Such improvements should be available to all expierence levels.