[tw5] Complex search - data inside transcluded tiddlers, produce outer pages instead

[caderoux]

It’s a while since I posted, so I will recap our use case. I produce a data dictionary of our cardiovascular data warehouse in TiddlyWiki during our build process.

A particular set of main tiddlers concern “Observations”, while other sets are “Measurements”, “Procedures”, “Devices” etc.

Each of these have a certain granularity they are grouped at for documentation - these are what I would call the Measurement tiddlers. So all the Mitral valve Doppler measurements are grouped in one tiddler that has the title “Mitral Valve | Doppler”, and that page is made up of sections. There might be many many other Measurement tiddlers like “Aortic valve | Area” and “Pulmonic valve | Regurgitation”, each holding the documentation for a grouped set of the things:

Within each Measurement, a page follows a template and the template transcludes tiddlers which represent the appropriate chunk of the page which is generated by the build process. And also transcludes tiddlers which represent the human additions of documentation to the wiki. Each of these, whether auto-generated or manually edited, are loaded as system tiddlers to keep them out of the way of the form:

$:/ascend/SOMEPREFIX:Name=Mitral valve｜Doppler

Where SOMEPREFIX might be XText.Measurement for the main auto-gen table of information, and Text.Measurement.summary for some fragment that the user can edit.

And then the <> macro puts all this together with transcludes to make the “Mitral valve｜Doppler” tiddler:

$:/ascend/Text.Measurement.preamble:Name=Mitral valve｜Doppler ← user tiddler

$:/ascend/XText.Measurement:Name=Mitral valve｜Doppler ← auto-gen tiddler

$:/ascend/Text.Measurement.summary:Name=Mitral valve｜Doppler ← another user tiddler

$:/ascend/XText.Measurement.stcodes:Name=Mitral valve｜Doppler ← another auto-gen tiddler

So all those prefixed tiddlers are designated elements of the tiddler with that Name.

So if we want to change the template for Measurements, we can change it in one place without having to rebuild every Measuement tiddler if the underlying tables and bits are still the same. The inner bits can be re-used in other pages and the auto-gen stuff never tramples on the human-gen stuff. All wonderful.

The problem comes in searching. Some of the content we want to be able to search is inside the transcluded tiddlers, but we would like search results for be able to show a list of basically the final outer tiddlers (the non-system ones).

We can generate the outer tiddlers completely in our build process if necessary, but that means the human part also needs to go through the earlier parts of the build process and kind of defeats the benefits. We could generate search text that is hidden during the build process on the main tiddlers, also needing to bring in the human template earlier.

Each of these elements does mainly go to one tiddler, so it would be possible to go from a fragment to a tiddler - even if they were remixed on a custom tiddler, we could tell the difference, so what I’m looking for I think is maybe a redirect from a tiddler to the tiddler that transcludes it or some other hints to make the search function more useful without losing a lot of the benefits of the transclusion system by generating whole tiddlers again.

Thanks in advance,

Cade

[pmario]

Hi Cade,

Can you show a screenshot of the search dialogue, where it finds content in a system tiddler? … Is this possible to share?

I’ll do a quick summary, to see if I do understand it right. … Everything works will except of search.

Search finds content from auto created “inner” / shadow tiddlers.

But clicking the link should open the “outer” template which basically shows a transcluded list of auto-created and user-created content.

I think it should be possible to create a “link-catcher” configuration the filters out the prefix and links to the name only.

So one of your “outer” tiddlers is: “Mitral valve｜Doppler”

This tiddler exists. So it is a real tiddler. right?

Do your system tiddlers eg: “$:/ascend/XText.Measurement.stcodes:Name=Mitral valve｜Doppler” have a caption-field? or do they show up with the full title in the search results?

As written an example of a search result dropdown would be needed.

-mario

[caderoux]

The system tiddlers do not show up in the search normally (I think because they are system tiddlers). But the content we are looking for is inside those tiddlers.

The “Mitral valve｜Doppler” tiddler does exist as a normal tiddler, its only content is the <<measurement>> macro which uses the template to make all the content, using fields in the tiddler and transcludes of tiddlers based on name

This is one of the data table tiddlers used in it that contains useful search content:

That table of data is fully generated by the build, and itself can transclude user-entered comments using that trans-edit macro. For instance, the “$:/ascend/Text.Measurement.View.description:Name=Mitral valve｜Doppler” is the user-entered tiddler of comments about the View column in the Measurement table for Mitral Valve | Doppler measurements. Such tiddlers may or may not exist. In “edit-mode” the informatics specialist can click on “Edit” or “Missing” and either edit them or create any missing notes on that item. So that content also, we would like to show up in the search. And we can tell from the name of those fragment tiddlers which the primary Measurement tiddler it eventually bubbles up to.

I probably picked a bad example because there aren’t any comments on that data. This is the final render of the page. I circled most of the transcludes and you can see the inner transcludes.

So for instance, if someone right now searches for “wave peak”, they don’t get any useful results because it’s in the system tiddlers, but we’d like it to show this tiddler as a result, because the content contains it. If one of those “Missing” user-edited transcludes contains some content, we’d also like those to be searched and this main tiddler to show up in the results.

I hope that makes it a little more clear.

Thanks,

Cade

[caderoux]

I left off a search example:

Right now “wave peak” only shows these other tiddlers, the changelog we generate between versions which is not broken up this way because it doesn’t need a lot of interspersed user content, and some other generated content related to coding systems for data interchange which have it as a literal and again, aren’t structured to be interspersed with user content (at the moment)

[pmario]

Thx for the info. This helps to understand the structure a bit better.

[caderoux]

I see. In the version of the user manual we ship, we disable editing, and hide a lot of stuff in TW, so that the users don’t get confused. They do actually have access to advanced search still, apparently I wasn’t thorough enough.

In this case, we probably need to make the regular search be more like the advanced search and take some actions to suppress the ones in the ST Codes, because that’s redundant. Or switch the fragment tiddlers back to regular tiddlers and suppress them in other places where seeing the underlying fragments might be confusing to users.

I guess I can probably figure out ways to do all that by myself.

Bot to figure out a way to only show the top level results. Is there a way to switch out the search result to make it link to a different tiddler? In the case above, I would want to only see Mitral valve|Doppler and Tricuspid valve|Doppler

Thanks,

Cade

[Scott_Sauyet]

In this case, we probably need to make the regular search be more like
the advanced search and take some actions to suppress the ones in the ST
Codes, because that’s redundant.

You might check out how the relink plugin (1) to see how it finds the
tiddlers that transclude (the text of) the current one. You could
probably implement your own search on top of that. (Theoretically, this
would have to be recursive, but your data might not require that.)

– Scott

[pmario]

I do have something to experiment with. !!! Backup first !!!

Edit: https://tiddlywiki.com/#%24%3A%2Fcore%2Fui%2FDefaultSearchResultList which contains the default-filter search.

If you mess it up. Just delete the tiddler. The shadow (default) will take over again.

It has 2 fields, which contains the 2 filters used to create the search results.

Change them to:

first-search-filter: [search:title<userInput>] :map[split[:Name=]nth[2]] +[sort[title]limit[250]]

second-search-filter: [search<userInput>] :map[split[:Name=]nth[2]] +[sort[title]limit[250]]

------ Explanation

first- … searches for titles only.

second- … is a full text search

In your first post you wrote about the different tiddler titles. All the relevant titles seem to have 1 thing in common: :Name= … So this can be used to extract the needed title.

:map.. applies the following filter to every input title and if they pass the filter write the result to the output

..[split[:Name=].. will split the tiddler title into 2 elements using :Name= as the separator.

…nth[2]] uses the second element, which is eg: Mitral valve | Doppler

You wrote that STCode is redundant. … Since the standard search does title deduplication they are ignored automatically.

Hope that helps. If you have more system tiddler titles … the mechanism should be similar. … but the filter can become much more complex. …

-mario