I’m a little confused by the question, since drag-and-drop is one chief way we maintain the explicit lists in a bespoke order.
I would prefer that we could use drag-and-drop or something similar to @vilc’s new offering to allow us to maintain list-before / list-after on the individual tags rather than list fields on the tag tiddler.
Actually, I was hoping you would do it for this tags issue, how do you logically combine drag and drop with set orders in list-before and list-after. In some ways they seem contradictory.
Start opinion!
I don’t actually see the problems you do with tiddlywiki. I feel the focus on backwards compatibility has informed the design of solutions that don’t compromise the past or the future as much as many people seem to think. Sure it may increase the complexity sometimes when trying to change the underlying behaviour, filling gaps we find as our use gets more sophisticated, or when learning how it works, but it actually is this “priority” that allows most users to “not know how it works, under the hood”, but to rely on the published practices.
If backwards compatibility was not assured you would always be having to know how it works under the hood, to avoid unexpected consequences.
I have always being interested in IT because it has sat at the forefront, bleeding edge, of human understanding, and I always push the limits of a system I get to know. Sure I am always pushing the limits, but that does not change how few limits are imposed on us by what we have today in tiddlywiki.
A new wiki, one that you don’t plan to distribute, can break every rule in the TiddlyVerse if it wants, so if it means enough to you, you need not design for the future, but fork it into a bespoke solution. It seems to me with tiddlywiki, the question is “are you going to generate technical debt for the future”, not how much technical debt tiddlywiki already has.
/end opinion
The list-before and list-after fields have been introduced, to allow users to add new tagged elements into ViewTemplates, without the need to modify shadow tiddlers. Eg: $:/tags/ViewTemplate is a shadow tiddler, that has a predefined list-field.
Let’s assume users use list-before, list-after in a tiddler that comes from a plugin. Those tiddlers are also shadow tiddlers.
This mechanism works well, except, when you start to drag and drop sort the order with the tag-pill dropdown. As soon as you do this all those shadows are converted to system tiddlers. $:/tags/ViewTemplate will be converted and the list field will be updated. The shadows, that contain the list-* fields will be overwritten and the list-before, -after fields will be removed.
So instead of not modifying shadows, the complete opposite happens.
If users actively change a shadow tiddler in the story river, we do display a dialogue and tell them what happens. So we (the core devs) can say: We told you, if there are problems.
To be consistent, we would need to show a dialogue if a tag-pill drag&drop happens. The problem with that dialogue is, that it would cause more problems, than silently overwriting shadows.
We (the core devs) have to make decisions, knowing that there will always be someone, which has a problem with that decision. We can only hope, that for 90% of our end users the decision is right and we will never hear from them 
I think you misunderstand. Perhaps I didn’t explain well enough.
I love the backward compatibility TW strives for!
But it involves a lot of cruft. Much of the technical debt can never be repaid. That’s the result of relentless backward compatibility.
So it’s a fun exercise to imagine what things might look like if we were to start over and build on the best ideas, ignoring the cruft. I don’t expect it to happen, not before TWX, anyway. But thinking about it can be quite enjoyable.
I do understand the problem of trying to design for many different users, and I also understand both how the current tag-pill drag-and-drop works, and the design constraints that it operates under. Nonetheless, it’s always possible there could be improvements.
I think the conversation @springer opened here suggests some real possibilities for a more useful UI. I’m far from sure that it could be done in a backward compatible manner, but I think it’s worth investigating. I don’t know if/when I’ll have time to play with it, but I believe it’s a useful conversation.
To be clear: My thought in this post (which @Scott_Sauyet rightly took from initial location) was NOT really a complaint about the way TW developers chose to handle ordering within tags. There’s a lot of considerations to juggle!
It’s a realization that I myself find the “bottom-up” way of tweaking order (using list-before and list-after fields) to be much more powerful — especially with the increasing role of cascades.
So, I’m imagining a plugin solution that might be helpful for myself and others.
The visual cues in my proof-of-concept are 100% backwards-compatible. This way, when I drop down a tagpill, I can quickly see WHY the things are where they are. This will help me make better decisions (even though opening the tag-child to add a field is considerably less convenient than drag-and-drop within the tagpill).
My stretch-vision involves making some kind of alternate “surgical drag” solution. Maybe with a modifier key? Or maybe using a marginal drag area that is otherwise easy to avoid? This would enable a more convenient access to modifying the children’s fields (rather than the list field of tag parent).
Even this would also be backwards-compatible and entirely optional, perhaps useful just for geeks who share a similar experience to mine (about the occasional downsides of the standard behavior for tagpill drag-and-drop).
I was trying to design something like this. (I am nowhere near trying to code it.) And I came up with the following:
Extended tag pill drag-and-drop.json (5.0 KB)
The content looks something like this:
We start with this, where only the lock icon and the title are actually shown. The others are just for this explanation. The right-hand column shows other fields on the tiddler in question:
(Note that the titles in the right-hand column are shortened, just for this display. title means $:/core/ui/ViewTemplate/title, etc.)
We drag row 8 atop of row 6, and new list-before and list-after drop zones are highlighted above and below row 6:
We now drop row 8 on the list-after row.
The important thing to note here is that the drag-and drop didn’t change the ViewTemplate's list field, only the list-before, list-after of dragged child.
And by default, the shadows are locked and cannot be dragged. You would be able to unlock them, but only while the tag-pill is open.
Of course this might also require updating the logic of sorting the items based on list fields, list-after, and list-before. And that’s already pretty complex logic. It is also possible that this wouldn’t be necessary; what’s there might just work.
I don’t know if there would be some additional trigger necessary to launch this, such as a modification key while dragging. It would be really nice, if we can get it to work, to make this the default, but it seems likely not to be backwardly compatible.
Currently, tag ordering is determined by applying a combination of the list field on the tag tiddler plus any list-before or list-after fields on the individual tagged tiddlers. In addition, if a tagged tiddler is not in the tag tiddler’s list and does not have a list-before or list-after field, it defaults to being added to the end of the list (alpha sorted).
This existing hybrid approach allows tiddlers to be explicitly positioned using list-before or list-after fields and is commonly used by 3rd-party creators to provide an initial placement for imported tiddlers (and especially for ViewTemplate or EditTemplate tiddlers).
Then, as soon as you drag-and-drop an individual tiddler to change the tag ordering, the list-before and list-after fields are removed from all the tagged tiddlers and the tag tiddler’s list field is updated to contain all those tagged tiddlers (including those that were previously just alpha appended to the end of the list).
Now, consider the following “minimal change” strategy that preserves any existing list-before or list-after fields:
list-before or list-after field, just update that field to contain the title of the tiddler after/before it, but do NOT update the tag tiddler’s list field.ctrl) is pressed, then it would get a list-before or list-after field with an empty value. This allows it to remain at the start or end of the list, even if the tiddler that is before/after it is moved.list or does not have a list-before or list-after field, update the tag tiddler’s list to set the tiddler’s new position (as is currently done).In all cases, do NOT change any other tagged tiddlers (i.e., leave their list-before and list-after fields intact).
I’m not certain, but this approach might even be (mostly) backward compatible.
-e
That’s precisely what I was imagining, although I hadn’t yet considered the blank list-before/after yet. I think my technique could be extended to just add one extra dropzone at the top or bottom, and not require a modifier key.
Exactly. that was the goal.
I think that if every place that does this sorting in the core goes through this new process, it will be very close to entirely backward compatible. The only exception would be if custom code depended on the tag tiddler’s list field for other processing, which would be odd, since that would only work right now if they always sorted the tag pill with drag and drop, and never set list-before/after manually. So it sounds low-risk. But it would require the users to get used to a slightly different UI.
I will try to update that tiddler to also show the top and bottom drop-zones. I should have time this evening.
Are we close enough to a design that I should open a GH issue? I’m not ready to commit to creating a PR for it. There’s a lot of code in that area I know nothing about. But I would like to try if I can.
I absolutely love this as the easiest and most intuitive element here.
For things like stylesheets, it’s just so often that I care not a whit about anything except making a certain tiddler drop to the end (so that it trumps whatever other old stuff or plugin-installed stuff was interfering). And with cascade conditions, it’s so often that my interest is in nothing more than making sure that this new custom condition not be stuck (as it is by default) at the end of the cascade. (And usually if I’m building the condition, I’m already in control of making it conditionally savvy enough to work well at the top of the cascade.)
]In some prior back-and-forth, I thought I recall @jeremyruston offering some reason why it really needs to happen that if there’s reordering in the tagpill, that existing list-before and list-after fields needed to be zapped in the process. But I no longer recall the details.
But I did just think of a wrinkle. And it might be why we have the current drop-everything-and-rebuild-the-entire-list mechanism in the first place.
If we have
| Template1 | {list-before: 'Template2'} |
|
| Template2 | {} |
and we now drag Template2 to the list-before drop-zone of Template1, we would get:
| Template2 | {list-before: 'Template1'} |
|
| Template1 | {list-before: 'Template2'} |
It would be nice to prevent this, by removing the list-before from Template1. But that would contradict
In all cases, do NOT change any other tagged tiddlers (i.e., leave their
list-beforeandlist-afterfields intact).
Perhaps that’s not a concern. We could certainly cause the same ambiguity by manually updating our list-before/after fields, and the algorithm will still manage. But it does feel untidy.
I misread a little bit. My idea is slightly different from this:
I would expect the core list fields to remain intact, and only work with list-before / list-after for our custom tiddlers. Plugins could override the tag tiddler’s list field, but I would expect them to usually use list-before/after. The locking mechanism in my sample UI is meant to make it a very deliberate move to adjust any shadow tiddler’s placement. Move your own tiddlers amidst the existing shadow ones, but don’t (often) move those shadows around one another. Obviously you could unlock, and move, say, $:/core/ui/ViewTemplate/tags above $:/core/ui/ViewTemplate/title. I imagine that would be rare, but we need to support it.
So I would expect that the ones in the list field of the tag tiddler would be locked, and not likely to move. And even if they are, I think we can still use the list-before/after fields for them. The sorter needs to already be able to deal with this, so I don’t think it’s an additional burden.
But unlocking can be accompanied by a “You’re overriding a shadow. Proceed?” message, to help drive the point home.
There is another idea I mentioned in the conversation that would not be backwardly compatible: turning the list-before/after fields into list fields (in theory unordered sets, not actually ordered lists, but that’s an implementation detail.)
With this I might include
title: Template 1
list-before: $:/core/ui/ViewTemplate/body [[Template 2]]
title: Template 2
list-before: $:/core/ui/ViewTemplate/body
Then I know that Tempate 1 should appear before both Template 2 and body.
The advantage of this over the following:
title: Template 1
list-before: Template 2
title: Template 2
list-before: $:/core/ui/ViewTemplate/body
is that we could now drag Template 1 to another wiki, and expect it to simply work, and be placed in the right spot, *regardless of whether we also dragged Template 2.
But I wouldn’t focus on this now, as I think we can do the backwardly compatible version on its own, and this will be much more controversial.
I am in favour of changing the default behaviour. I do not really like it, that enlist and enlist-input do not see the before-* fields.
See the problem with list-before/after if we use [enlist{$:/tags/ViewTemplate!!list}] or [{$:/tags/ViewTemplate!!list}enlist-input[]] we get a different list than with [[$:/tags/ViewTemplate]tagging[]]
All of those inconsistencies should also be “fixed” if the behaviour is changed in a backwards incompatible way.
Also see: Extended Listops Filters all of them will probably need to know about the new behaviour.
The list-before/after handling is hardcoded in wiki.js at exports.sortByList().
There needs to be a solid concept, before anything is coded. Otherwise we will cause more inconsistencies as we have at the moment.
I don’t know how to fix that. The enlist operator should not know where its data comes from. If we can update tagging[] in a backward compatible manner, and use it—or its underlying implementation—everywhere the core needs to order a list of tags, I think that’s probably the best we can do. If we are using enlist somewhere we should be using tagging[], then we can fix those, with a risk to backward incompatibility. But I don’t think we can change them in user-land.
The only thing I see as a possibility, but I think it’s a terrible one, would be to change property access to handle list differently. Uggh!
But the big thing here is that the changes we’re proposing should not break much of anything, and I believe shouldn’t require any update to tagging[]. Essentially, we have two methods to modify the order of tag lists, two methods that will work together. All we’re doing is changing which of the two is used by default by the drag-and-drop sorting.
I have being giving this a little through and think we have the filter operators to process the list-before and list-after fields so if these values remained we could always capture the intent of the tiddler author(s). Although I am yet to do this.
That is not unexpected. This “sort by” highlights the direction I was thinking and rather than change the existing mechanisms we can find a way to override them, So often our lists are seen in different orders such as by title, by a date or status. Its on the UI where this becomes an issue, so why not improved how that works.
Get the part of the core that iterates the tag to use an alternate sortby list if it exists. The user altering the order can snapshot the alternate sort order.
The order can be
Or more tentatively: We’d just change (at first) whether users can opt-in to prioritizing the “bottom-up” (child-driven) method of adjusting the order of tag-children. (It may not even be the default for the plugin, if we go with using “option/alt” with a drag to invoke the bottom-up approach, say.)
It makes sense to me that we’d develop a proof of concept as a plugin and get a good number of people playing around with it, and postponing any change to the core until after people experience the possibilities of this approach.
I’m a bit confused here. The point of the “bottom-up” or “surgical reordering” approach is that there is no list — it’s not just about where the list is, but rather that the list-before and list-after fields work in a parallel-distributed fashion to accomplish the determination of order. And it’s a good thing if the tiddlers in my cascade condition can carry something like their appropriate position (for lining up right under the cascade’s tag), even if I import them without importing the tag-home tiddler with its (messy and crufty with baggage from that other wiki) list order field.
Perhaps I’m missing something about your suggestion?
I am wondering if we can divide the list that results after applying “order requests” (list-before/after) from that which results from a bespoke drag and drop.
But wonder if what needs to change is the place where the tags are iterated in the core UI.
Outside the core we can use any method we wish to manage the sort order.
Perhaps this is true. But ordering something that’s not leveraged by the core is usually merely cosmetic (unless we’re talking about cascade-like functions managed outside the core, which could be a good playground for solutions…).
All my pain-points have to do with tag-based cascades that are hooked by the core. So, bottom-up bids for sort-order posotion — in the case of tiddlers that do have a core-cascade-role tag (or stylesheet tag, etc.) — is the main use-case I see.
(Even the ordering of view template elements is less dire; if I import a template and it by default falls to the bottom position under $:/tags/ViewTemplate, at least I often notice the problem, which is essentially cosmetic, right away. And imported stylesheets, if they default to the bottom wherever the list field of $:/tags/Stylesheet has been actively tweaked… at least imported items are defaulting to the dominant position, so it’s easy to troubleshoot… But if I import a cascade condition (as part of a bundle, say) and it is dumped (by default) at the bottom of the pecking order because contents of the tag’s list field all get priority, I might go about my business for a while before realizing that something I imported is not taking effect. Reconstructing what it was, and which was the tag under which it was supposed to dovetail with other conditions (etc.) is a different level of nuisance. And if I then use drag-n-drop within the cascade’s governing tagpill to “fix” the order, then any previous work at setting up other tiddlers with "Hello my name is Conditionally-Use-Caption-If-Available, and I belong before $:/config/ViewTemplateTitleFilters/default" … is suddenly lost.)
Yes, this is why I am interested in this issue. I mentioned previously, a card solution I am playing with, Display cards on tiddlers according to different conditions, so each tiddler has multiple roles/cards.
I have a single tiddler on the $:/tags/ViewTemplate list-after $:/core/ui/ViewTemplate/body which powers a cards solution. It effectively provides an alternative to the $:/tags/ViewTemplate mechanism, but within it. This is allowing me to play with display-filters, delist, and arguably tag order as discussed here.
So for this issue I wait with baited breath to see if anyone comes up with a superior approach to this tag ordering.
I have set list-before="" for these cases, it is somewhat opinionated that it gets the first go in a cascade, but a few times I have given list-before="$:/config/ViewTemplateBodyFilters/default" which reliably exists. I am dumping it at the top of the list 
For stylesheets I would set list-after=""
Using the above I have not yet come across gaps in what I can set. I have not being forced to manually reorder.
If tiddlers are stored in a plugin or shadow tiddler with a list-before or list-after request even if these fields are deleted, after a manual drag and drop) the original value will exist inside the shadow tiddler. Perhaps a mechanism to re-sort based on these shadow values would be helpful.
