The feature to add [[#tags]] entered in the text to become titles, is not unlike what is available through the tags pill. Although I personally would like a form of tag that does not fill the tags field on tiddlers, but can mostly be used in similar ways, like testing if they are in the text, listing them including for selection, and listing the tiddlers they tag.
I am the designer of reimagin tags which extends the tag drop down a lot, So I thought of extending hashtags further, but there is something simple and deterministic about the current title driven process.
We could always create a user interface element to add any hashtag as a tag on the tiddler automagicaly or manually, or via its insertion via an editorToolbar button, but for now at least I donāt think it necessary.
You may also find another organising method flags of interest. In effect no-touch tags.
I donāt think we should be concerned with more options and features for āmarkingā or organising tiddlers. We will just use what we need when we need it. As I am writing a book this idea of hashtags or content tags appeals because it can be used to mark my content, in tiddler, its more specific.
We do start with plugins, but if it was ever in the core it could require a config item be set, to appear or even just using [[#tagsname]] and it becomes visible.
Thatās the point. tags are already āspecialā in every way. Technically they are not a tiddler field. JS internally, they are an array. The same is true for the list field.
If tags are listed, they use the list-field of a tag-tiddler to provide a custom sort order. There is no other field, that can do that.
Every other field that contains a Title list internally is a string, with special formatting to allow abc [[title with spaces]]. So internally there is a fundamental difference, between tags, list and any other field.
For every tag in the system, the core manages indexes. So if we want to know, which tiddlers are tagged eg: HelloThere, the access is very fast, since the index knows that tiddler list already.
Letās assume we have 3 tiddlers a, b and c, which contain different ātag-likeā titles in my-field
title: a
my-field: a [[a a]]
+
title: b
my-field: b
+
title: c
my-field: a
A) The filter to list tiddlers that contain āaā in my-field would need to look as follows.
List all tiddlers that have the ātag-likeā field ā¦ has<field>
This functionality needs to read every tiddler in the wiki
This can become a problem if we have 10000++ tiddlers
Make sure we do not list draft tiddlers ā¦ !is[draft]
Filter the results :filter[...]
In the filter read my-field ā¦ get<field>,
which returns the whole title-list string
Separate the title-list string into a real list we can iterate over ā¦ enlist-input[]
Match the title list against the āsearch stringā ā¦ match<tagLike>
Return the tiddler title only if it matches
join the found elements with a coma +[join[, ]]
Example B)
Get tiddlers with tag ā¦ tag<tag>
This function directly reads form the index, which is automatically updated, whenever a tiddler changes
Make sure we do not list draft tiddlers ā¦ !is[draft]
join the found elements with a coma +[join[, ]]
I think it is clear that there is a huge performance benefit. ā¦
Conclusion Tag-likes
So to make ātag-likeā titles fast, if they are in fields, we would need a more generalised indexing system.
It would need to allow us to speed up the field:my-field[] operator in a way, that is works like a tag.
May be field:my-field:tag-like<tagLike>
Disadvantage: Every tag-like field would need quite some memory, to store the index.
Thatās a completely different part of the core code. links[] and backlinks[] are also very expensive.
To find links and backlinks the whole tiddler text needs to be parsed.
Thatās the most expensive functionality we have in TW
So the parse-tree is cached independently from filters
2 indices need to be created one for links, that other for backlinks
They are created āon demandā when they are needed the first time
Once created, they are cached
But these caches is completely destroyed frequently, so it still can be work intensive
Conclusion links, backlinks
IMO these two functionalities have room for improvement.
If you use [[#tag]] this mechanisms are automatically active.
They are not active for tag-like values stored in eg: my-field
I would not use the existing link, backlink mechanisms here
I would use a trie like index, similar to the one, I am using for my āaliasesā and ābackaliasesā handling in my uni-link plugin.
My trie-structure is similar to the Aho-Corasick functionality that is used now for the freelink-plugin. (Wich is optimised to handle ānon Latinā character sets)
Those 2 trie-indexes could probably be used to create a more generic mechanism that could also be used for very performant tag-like field link and backlink handling.
It would probably also be able to handle the existing tag indexes.
I am not sure if it could replace the existing links and backlinks indexes. A lot of experimenting would be needed.
I just stumbled upon the listed-operator, which uses a global-cache. So it is cached ā¦ But ā The main problem here is that the global cache is destroyed with every UI interaction.
Iāll update my post accordingly soon. Need to dig a bit deeper.
Please take note I reopened this thread to implement hashtags which I could also call content tags as they are placed within the text field. My proposal is to simply use titles as links [[#tagname]] and make use of existing mechanisms. I have demonstrated a number of custom functions to make handling the # and other title prefixes easier to use.
This is quite simple yet functional and does not involve turning them into regular or alternative field tags. yet itās quite easy to locate tiddlers containing them, if the current tiddler has one or more, or test if it contains a particular one. you can open any content tag and see where it is mentioned.
as Mario points out this leverages existing optimisations
So I would appreciate some feedback on this approach specificaly before going too far in another direction and if you are give a reason by comparing with what I proposed.
post script
there is no real need to turn my proposed content tags into regular tags, I am not sure what that would add, just reduce their specificity.
I think this was the primary motivation behind @pmarioās musings (and Iād have the same concerns)ā¦
This may be of less concern if the wiki(s) where youāre using your links/backlinks-based approach arenāt particularly large or text-heavy, of course.
Marioās comment is a technical performance review, specifically in the context of other fields, that is something we may need to contend with, however out of the box we are not trying to use fields beyond the text field, @pmario as responding to using alternative fields, I am explicitly excluding them in my current solution.
When dealing with [[#hashtags]] I quickly reduce the results of links[] or backlinks[] with prefix[#]
I believe the majority of activity relating to hashtags may only be on the current tiddler, as it is focused on ācontent tagsā.
Something else that is often forgotten is many uses of links and tags for that matter are āon demandā only. My example hashtags are just titles, ie links and leverage the caching and optimisations in the core, and only need to be iterated, when, for example one is looking for all instances of that hashtag. In a similar way when testing for hashtags within the current tiddler you are only referencing the current tiddler.
In many ways this solution benefits from the nature, structure and history of performance improvements in TiddlyWiki.
If someone can demonstrate a performance hit using this approach, we can look at that when it happens. However I would be surprised.
Other approaches I did not follow
I was looking at the parsing, and $link widget, to code features for hashtags, and may do so in the future, however I realised I can satisfy 90% of the requirements without doing so. I could also take the macro route such as <<# hashtag>> but then I need to search for these throughout the wiki.
The macro model could permit setting anchors for navigation, turning off display (for printing) and other dynamic features.
Finally this could also be implemented through template transclusion like {{#hashtag||_userTag}} where user tag provides a link with more button features.
Note: I am looking at adopting the prefix _ for transclusion-template tiddlers
Thatās too late. To list all links every tiddler has to be parsed and the parse-tree has to be searched for a link-widget call. Performance here is OKish.
Backlinks are the culprit as we know from many discussions here in the forum. The algorithm to find backlinks needs to search every tag in every tiddler, if it is linked there.
As I wrote, there is room for improvement. The different indexers have been created organically, when we experienced performance problems. So they are implemented according to the distinct issue that needed to be solved. So they are all slightly different.
IMO we should create a more generic functionality now, that we know what actually causes performance issues. ā But thatās a lot of work.
But it is true that this performance requirement is only when trying to find all occurrences, and this typically needs a particular tiddler to be visible. ie there is no background performance issue unless say you leave a sidebar tab open that iterated a hashtag.
As @pmario said, this is true of links[] but not of backlinks[], which you were proposing to useā¦
Iām not sure how you could avoid this issue unless you sacrifice this functionalityā¦ and IMO, hashtags donāt offer much benefit if you canāt find all occurrences of a given hashtag in a given set of tiddlers. which (in your proposed solution) would require backlinks, search:text, or similar operators. So the performance issue seems inescapable without separate improvements to backlinks indexing. Thusā¦
I did understand your intent. What I was implying (and should perhaps have stated more directly) in my previous post was that others are proposing field-based alternatives because they see the performance issues currently inherent to backlinks[] as a substantial barrier to adopting your approach.
Edit: Unless youāre saying that functions like your find.userTag would only incur performance costs when viewing a tiddler (or template) where they were used? Thatās technically true, but strikes me as a strong incentive not to use themā¦
This is a common use pattern, keeping the need to do something in the context in which is is needed and not globalising it.
Why does it?
strike YOU as a strong incentive not to use them
I donāt follow
Perhaps I misread this, @pmario can help us here. I believe his reply was in the context of links and backlinks within additional fields, as I suggested giving links and backlinks a fieldname as a parameter. In turn I was responding to other users raising that. I am not wedded or interested in that now.
No I was not saying that, find user tag is when you are searching for the occurrence of the use of that tag and all.userTags when trying to find all of them. This is when you are trying to collate them, I donāt envision this to be a common need. But I would put it on the hashtag tiddler, like on a tag tiddler, automatically if needed, if title has prefix.
userTags.here is similar to links, and the tags[] operator but only returns those links[] within the currentTiddler tiddler body that have the nominated prefix eg #. We could use this in a viewTemplate to find, and perhaps test for the presence of a particular content tag, then if it exists do something. I am not suggesting we need to, only that we can, just as we can with tags.
The key value I see is allowing a form of annotation within a tiddler body with view and edit modes, something you care about mostly when viewing/or editing that specific tiddler. For example an author I may want to use [[#expand]] to indicate I need to add more here, or [[#reword]]. I expect once done I delete them.
Additional tips,
if these hashtags are in content to be posted to X or something you can use ~ or hide links within the output template.
You can hide the visibility using prettylinks such as [[some in place text|#tag]] but still find the tiddler and list the #tag. [[some text to reword|#reword]] ctrl-L is really good for applying this.
Mario can clarify, but I understood his comments re: links and backlinks to be reflective of their current status (and potential performance impact) ā i.e., backlinks[] used to retrieve links from text fields (only).
If nothing else, his technical overview aligns very closely with my personal experiences using backlinks in large wikis (actually for a very similar purpose to the hashtag approach youāve outlined here). I concluded that, while link syntax is certainly easy to type, collating backlinks from even a subset of a large, highly linked wiki was so slow as to be virtually unusable. I ultimately switched to a system much like this oneā¦
ā¦ where all text-field internal [[links]] that matched a certain filter pattern would be automatically converted to list-items in a nominated field whenever I saved the tiddler.
I realize this is exactly what you didnāt want to hear, but it does work and itās reasonably performant for all the filtering Iād want to do (equivalents of userTags.here, find.userTag, and so on) whereas the links-based filtering really was not.
If itās mostly maintaining a separate field that you object to, I suppose you could, if you liked, also hack the $link widget to include an inline x button ā something like [[#hashtag]] ā #hashtag [x], which you could click in view or preview mode to automatically remove the tag from the field.
To a few of your other pointsā¦
Really? This is pretty much the primary (only?) function of hashtags as Iāve seen them used: to make it easy to collate and browse posts on a particular topic.
This isnāt a use that would ever have occurred to me, but I can see the utility of having your annotations pre-styled in an eye-catching format. But surely you can do this already, without any supporting functions or ViewTemplates, just by typing hashtags as missing links?
I suppose Iād gathered from the example functions you gave earlier that you were mostly interested in using in-text hashtags to collate content to be presented elsewhere ā either on the same tiddler or on a separate hashtag tiddler, backlinks-style. If this isnāt your end-goal, then youāre right: you wouldnāt have to worry about backlinks performance.
On the other hand, if hashtag collation is part of the intended package (and IMO, most people would expect this sort of feature)ā¦
Because, as discussed above, it will be very slow.
This is tapping into the cached list of titles not backlinks.
In the first wiki in which I want to use it, there are not many tags on tiddlers, other than those that add it to the contents. So If I used the above approach to add them to the tags field would be fine.
The solution I was building had another objective, to avoid overuse of tags by introducing another method, as my flags (no-touch tags) solution also does. In my case I expect a few hashtags to be used, they will be added to content needing work, then when the work is done removed, So I dont think the number will accrue too much.
Mario details the issue with non-tags fields but suggests the expensive list and backlinks is already optimised against the text field, although has its limitations. He mentions;
So we have room to innovate.
[Edited]
Perhaps there is value using the very limitations we fact to define different functions, such as local #hashtags (just links) for listing and consumption in a tiddlers text.
List these #thashtags found using links[]prefix[#] on the current tiddler in the story. Whilst we would need backlinks to find these, we donāt. Instead allow hashtags to be globalised with a click, by adding it to the tags. It thus becomes visible as tags[]prefix[#] ie we list only hash tags that are also tags.
: 
