Recursive filter operators to show all tiddlers beneath a tag and all tags above a tiddler

I have created a filter operator taggingtree[] to collect all tiddlers within the tree beneath a tag (or a list of tags), i.e. all its children, the children’s children, and so on. I’ll post the complementary filter to get all of a tiddler’s (or a list of tiddlers’) parent tags, as well as the parents’ tags, and parents’ parents’ tags and so on, in a forthcoming reply to this post.

My use case ist this: In a very large wiki, search results are often overwhelming. Therefore, I build a list of all tags that appear within the tag hierarchy above the search results tiddlers and put these into a select element. The select element is then used to narrow down the search results to the selected topic which can be either broad (by selecting a tag in an upper level of the hierarchy) or narrow (by selecting a tag near the bottom). The present filter is used to build a list of all tiddlers beneath the selected tag which is then combined with the search results using the :intersection filter run prefix.
Showing all wiki tags in the select element is not useful when there are a lot of them, as in my case (1,000+). Showing only the immediate tags of the search results doesn’t help much if the tag hierarchy is very deep and one doesn’t know where exactly to look, i.e. it’s not much more useful than the results list itself.

Imagine you could limit search results on tiddlywiki.com to anything below Filters, be it filter operators or filter run prefixes and whatnot, without getting results for widgets or releases or whatever.

This works especially well if you put a button in the ViewToolbar that populates the select element (or rather the tiddler tied to it) with the current tiddler and then puts the cursor into the search box. This way I can start a search in the hierarchy below that tiddler with a single button click. This has already saved me more time than was needed to implement it.

Here is the taggingtree filter code for copy-pasting:

/*\
title: $:/core/modules/filters/taggingtree.js
type: application/javascript
module-type: filteroperator

Filter operator returning all tiddlers that are tagged with the selected tiddlers
as well as those tagged below in the tag hierarchy

\*/
(function(){

/*jslint node: true, browser: true */
/*global $tw: false */
"use strict";

/*
Export our filter function
code adapted from $:/core/modules/filters/tagging.js
*/
exports.taggingtree = function(source,operator,options) {
	var results = [];
	source(function(tiddler,title) {
		// start the recursion for this title
		getTdidlersRecursively(title,results,options);
	});
	return results;
};

function getTdidlersRecursively(title,results,options) {
	// get tagging[] list at this level
	var intermediate = options.wiki.getTiddlersWithTag(title),
			t,p;
	// remove any that are already in the results array to avoid loops
	// code adapted from $tw.utils.pushTop
	if(intermediate.length !== 0) {
		if(results.length !== 0) {
			if(results.length < intermediate.length) {
				for(t=0; t<results.length; t++) {
					p = intermediate.indexOf(results[t]);
					if(p !== -1) {
						intermediate.splice(p,1);
					}
				}
			} else {
				for(t=intermediate.length-1; t>=0; t--) {
					p = results.indexOf(intermediate[t]);
					if(p !== -1) {
						intermediate.splice(t,1);
					}
				}
			}
		}
		// add the remaining intermediate results and traverse the hierarchy further
		$tw.utils.pushTop(results,intermediate);
		intermediate.forEach(function(title) {
			getTdidlersRecursively(title,results,options);
		});
	}
	return;
}

})();

taggingtree.js.tid (1.7 KB)

Usage is basically identical to tagging[], except that it will generally yield more results. There is a loop filter to avoid infinite recursion.

Caveats:

  • I’d rather have done this in wikitext. There is a related discussion on GG, and for a while I went with @Mark_S’ proposal for nested subfilters. That only works for a predefined hierarchy depth, though. Comparing with my comments then, my hierarchy depth has increased by at least 4 levels since that time, so that code must be updated regularly to keep up with my taxonomy mania.
    Also, it is about 15 % faster than the nested subfilters approach (in a test which yielded ≈ 3,600 result tiddlers, using count[] to avoid the DOM creation overhead when timing) and probably orders of magnitude faster than kin.
  • I’m not a JS expert. The code above is mostly copy-pasted from other modules in TW. I’ve given credit in the code comments. I do not know in detail how some of it, e.g. the tiddler iterator source, works. If there are ways to improve the code or make it more conforming to TW standards, please let me know.

See also the preceding discussion here in the forum for why I don’t just use the kin filter or a recursive macro (mostly speed issues), where @TW_Tones also proposed a macro solution.

Have a nice day
Yaisog

3 Likes

While it’s “standard” that the filter operators are “all lowercase”, you don’t need to use that convention in your code. eg: gettiddlersrecursively should be changed to getTdidlersRecursively(), to improve readability.

Function names in JavaScript usually use “camelCase” … It’s not the same as for wikitext “CamelCase”.

In JavaScript in general, if a function starts with an uppercase letter, it means that it is a constructor as in new $tw.Tiddler(...) … Constructors only work in combination with new

1 Like

@pmario: Thanks, I have changed it directly in my previous post (and this one).

As promised, here is the complementary operator to get the whole tag hierarchy above a tiddler or a list of tiddlers:

/*\
title: $:/core/modules/filters/tagstree.js
type: application/javascript
module-type: filteroperator

Filter operator returning all the tags of the selected tiddlers
as well as the tags of those tags, and so on

\*/
(function(){

/*jslint node: true, browser: true */
/*global $tw: false */
"use strict";

/*
Export our filter function
code adapted from $:/core/modules/filters/tags.js
*/
exports.tagstree = function(source,operator,options) {
	var tags = {};
	source(function(tiddler,title) {
		// start the recursion for this title
		getTagsRecursively(tiddler,tags);
	});
	return Object.keys(tags);
};
	
function getTagsRecursively(tiddler,tags) {
	var intermediate = [],
			t, length;
	// get the tags
	if(tiddler && tiddler.fields.tags) {
		$tw.utils.pushTop(intermediate, tiddler.fields.tags);
	}
	// remove any that are already in the tags object to avoid loops
	if(intermediate.length !== 0) {
		if(Object.keys(tags).length !== 0) {
			for(t=intermediate.length-1; t>=0; t--) {
				if(tags[intermediate[t]] === true) {
					intermediate.splice(t,1);
				}
			}
		}
		// add to result object
		for(t=0, length=intermediate.length; t<length; t++) {
			tags[intermediate[t]] = true;
		}
		// recurse
		intermediate.forEach(function(title) {
			getTagsRecursively($tw.wiki.getTiddler(title),tags);
		});
	}
	return;
}

})();

tagstree.js.tid (1.5 KB)

Now I have to stop procrasting and do stuff.

Have a nice day
Yaisog

2 Likes

Good work on the filter operator. I will check it out. Once I finish a new version of the wikitext macro above, with looping protection, we will have to test its performance against the operator.

  • Note in my examples I did not use the KIN operator myself.

Nice work @Yaisog

Will it be possible to replace kin-filter in the tw-locator with this?

In my wiki, tw-locator and kin-filter are always the most CPU-consuming part.

1 Like

I’m using these filters to great effect with TiddlyMap for task management. Thanks @Yaisog - faster than Kin. Although Kin is awesome too.

Hi, @Yaisog I only just noticed the tagstree operator finds the items tagging the current one and up (not down)

  • Did you make one that lists all the tiddlers below as in the for TableOfContents? Since this is possibly the most needed to treat complex hierarchies as a single set?
  • This would make things easier for @paulgilbert2000 who is designing some complex hierarchies.
1 Like

One operator is posted in post #1 in this thread and the complementary operator in post #3.
I think you need taggingtree.js from post #1?

Thanks you, sorry, my mistake, and thank you.

In my mind these are good enough to submit to core, if not with minor changes arising from peer review. They are sufficiently light weight I think.

  • Would you consider submitting them?

Also you may be interested in the various nearby discussions (if not already involved) because access to a whole tree in an operator, offers a simplified subset of what the KIN operator offers.

  • Running map after getting a full tree of tiddlers is an easy way to collate a lot of values.

Before I’ll make a PR, I would like to hear what @jeremyruston thinks about adding these filters to the core. I’m not 100% sure that this is core material. After all, these can be easily installed from files.
If he is positive, I’ll make a PR.

Have a nice day
Yaisog

Hi @TW_Tones,
A wikitext solution of nearby was developed by @Yaisog see: Nearby Neighbors: A WikiText Solution - Tips & Tricks - Talk TW (tiddlywiki.org)

1 Like

Apologies @Yaisog I commented on a related GitHub ticket that @kookma opened, but didn’t link it here. The comment was:

Hi @kookma it would indeed be useful to have something like the kin operator in the core. My preferred route is not just to provide a special operator for treewalking, but to extend the filter syntax so that it is straightforward and safe to write recursive functions. Thus, I think the first step is actually the implementation of user defined functions in #6666. That gives us named functions with named parameters, which are the building blocks of recursion in functional languages.

3 Likes

I have being experimenting with @Yaisog’s two operators and yet to fully understand or confirm their operation.

Some examples;

;tags tree about
{{{ [tagstree[About]!is[system]!is[missing]] }}}

;tagging tree about
{{{ [taggingtree[About]!is[system]!is[missing]] }}}
  • note missing tiddlers are included (helpful but not expected)
  • Its listing tiddlers I do not expect

I am getting a different result with the same filter in a list

;tagging tree about list
<$list filter="[taggingtree[About]!is[system]!is[missing]]">
   <$link/> -  {{!!caption}}<br>
</$list>

I think it may be confused as to if it is a selection constructor or a selection modifier

  • I will return when I understand them better.

[Edits]

  • They also follow any tag including system tags
  • It, taggingtree looks more like a filter to retrieve “all related tiddlers” for the nearby project, not just return a list as in a TOC

In your first example, you use it in an inline filter. This would return only the first result.

Also, like I wrote in the OP, "Usage is basically identical to tagging[]". Hence, they don’t take a parameter. The input to the filter determines either the tag(s) for which all downstream children are found (taggingtree) or the tiddler(s) for which all upstream tags are found (tagstree).

Just like tags[] and tagging[] they will return system tiddlers. This is intended, to keep them simple and universal.

2 Likes

When filtered transclusion (aka, an "inline filter) occurs directly in a wikitext context, it renders ALL matching items. It is only when filtered transclusion is used as a widget param value that it only returns the first result.

Thus:
{{{ [enlist[A B C D]] }}} renders as four link items, “ABCD”,
but
<$text text={{{ [enlist[A B C D]] }}}/> renders as just “A”

-e

1 Like

Thanks @Yaisog I have not being so error prone before, sorry, now its working.

  • I was using mistakenly the parameter, so the input to the filter was [all[tiddlers]]
  • now I use [[tiddlername]taggingtree[]] and / or [<currentTiddler>tagsyree[]]

I am learning new things here every day. :grinning:

I find taggingtree and tagstree conceptually straightforward and fast. I think it should be in core!

1 Like

Can you make it a plugin? Or you can allow me to make it a plugin using GitHub - tiddly-gittly/Tiddlywiki-WikiText-Plugin-Template: Simple plugin template, with Github Pages demo site. , and I will add it to the plugin library to receive update.

Sure, I can make it a plugin. Give me a day or so to get some documentation up…