Help with templates, fields, and searchability

I think your imported tiddlers look fine. The links seem to work.

The only thing that seems to have happened, is that your search function does not work anymore.

I did download empty.html from tiddlywiki.com and I did re-import all your tiddlers from your commonthroat.html into empty.

Save and reload, then the search function worked for me.

I personally would define a modified: 20250518 field for every tiddler. So they are visible in the Recent tab.

See docs about the TW date format for more details.


I personally would not change the text portion. – I would add new fields that may be useful in the future for sorting.

Just some thoughts.
-Mario

Interesting. The search seems to work for me. Everywhere I’ve tried it, on whatever instance of TW, it gives you a list of titles and body text that match your search.

I’m among those who would absolutely go with field-structure. Data architecture is beautiful. Flattening data dimensions into ordered text beneath headings is… not.

I suspect you’ll eventually run into situations where search results will be unmanageably cluttered if you don’t have field-structure… For example, let’s find a word whose definition includes “weak” … Oh no! — weak is part of the text of so many tiddlers because it’s a common string in the pronunciation field! Being able to search within a specific field solves such problems.

Another advantage of creating real data structure from the start is that it’s so easy then to come up with on-the-fly variants to your UI: You can make one template for a quick-reference “glossary” that filters according to some condition of interest and formats each definition with a hanging indent but omits etymology and usage notes, for example, while another template foregrounds usage notes up top, puts etymology in italics at the bottom, etc., ordering the fields and applying nice visually satisfying css to them according to whatever the your purpose requires…

You can get a plugin that helps you search in specific (or any filtered subset of) fields. Here’s one:

EDIT TO ADD: You asked about multi-line fields. Yes, you can absolutely have multi-line and multi-paragraph fields:

and here’s a small demo I made.

Let us know what you decide and how it’s going!

I’ve already run into that problem. I have another lexicon for a different language spoken by the same critters that I started just earlier this year that’s much more of a green field for my TW explorations. The first thing I did was come up with a more compact, and more importantly search friendly, phonetic transcription system, as can be seen here.

Regarding the “put everything in fields” concept, it’s more or less what I was complaining about earlier. Wikitext (rather the HTML rendered from it) is already structured. Why not lean into that structure? Again, TW is by no means the only software guilty of this.

I find fields to be super-powerful in their relation to TiddlyWiki’s filters and templates. But if wikitext structure hits your sweet spot, then I certainly will cease evangelizing for fields!

An approach I use often with data from an external source is to (when necessary) write some code to parse the input format into a convenient representation, then write an additional bit to convert that into JSON tiddlers with title, tags, and text fields plus whatever else seems useful. I tried that with your data.

The JS Code creates JSON tiddlers like this:

    {
        "title": "HKrqFdqg",
        "tags": "lexicon",
        "entry-id": "581",
        "pronunciation": "long rising strong growl, chuff, huff, late high weakening whine, huff, short low weak growl",
        "etymology": "[[HKrg]] (notch) + [[qFdqg]] (ear)",
        "part-of-speech": "Noun",
        "definition": "ear notch (derogatory term for a firearms enthusiast)"
    },

I save the output to a file and drop it into an empty wiki, drag your Hello! tiddler over and, since there is no text content in any of these new tiddlers, add a ViewTemplate that looks like this:

title: viewtemplates/lexicon
tags: $:/tags/ViewTemplate
list-after: $:/core/ui/ViewTemplate/body

<% if [<currentTiddler>tag[lexicon]] %>
<dl>
<$list filter="pronunciation etymology part-of-speech definition usage usage-notes notes note note- related-terms derived-terms lore entry-id" variable="field-name">
<% if [<currentTiddler>has<field-name>] %>
<dt><$text text = {{{ [<field-name>search-replace:g:[-],[ ]titlecase[]] }}} /></dt>
<dd><$wikify name="content" text={{{ [<currentTiddler>get<field-name>] }}} output="html"><<content>></$wikify></dd>
<% endif %>
</$list>
</dl>
<% endif %>

This template is fairly naive, just including a definition list for all the custom fields, in an order I guessed might be close to what you’d want. (It also points out that you have fields of note, notes, and note-, which seems odd!) The resulting tiddlers look something like this:

You can see this result at http://scott.sauyet.com/Tiddlywiki/Demo/Commonthroat.html.

This is far from a complete solution. But it might be a start on the way there.

If it’s close and there’s something in my process you want to tweak but don’t know how, just let me know. The code for the conversions is very clear to me, but could well be obscure, or just gobbledygook to others.

2 Likes

I would be more inclined to use them in the way you suggest if TW made it easier out of the box to work with them. Something like prefixing a search with field_name:value would search for that value only in that field. I know the advanced search can do that, but why push such a powerful feature further up the learning curve? Same thing with not allowing line-breaks in the easy-to-access field entry boxes when creating a new tiddler.

My other issue with using fields for my use case is that fields are not hierarchical. Dictionaries usually distinguish between polysemy (a single word having multiple related meanings) and homophony (two unrelated words that sound the same). Polysemous meanings are grouped under one heading, while homophonous meanings are separated.


I will check that out. I’m currently normalizing those obsidian markdown files precisely to turn them into a single monolithic JSON file using Python.

OK, I’ve reorganized my Outlander wiki to put everything in fields. Now I need to figure out how to make those fields searchable.

I’ve implemented your view template here. It seems to work well. Entering words this way would certainly make porting the lexicon to other software much easier. Still need to figure out how to make them searchable and how to quickly enter new entries. Obsidian’s biggest strong suit was that very little got in the way between my ideas and the page. I didn’t have to think of what button to push to format a level 1 heading or start a table. It was all done right there in the body of the note, without the verbosity of HTML markup.

One difference between my parsing and yours was that I further parsed the definition section to extract the part-of-speech. I think that’s useful, but YMMV.

I would not expect to create a tiddler and add the fields manually. I would create an entry form. Here is a sample, adapted from a question of mine from last year:

EntryForm.json (2.9 KB) (download this and drag the resulting file onto any wiki.)

For searchability, another topic discusses way to modify the default search, although I’ve personally only altered its output. But you can also write a custom search using the search Operator, which allows you to search in specific fields or in all fields.

I’m trying to strike a balance between granularity and ease of entry.

Another thing I hadn’t considered is the ability to track backlinks and missing entries (links to missing tiddlers).

Supporting the argument for fields it is the best way and the skys the limit when searching based on a fieldname and value or part there of.

Although I understand if the content you populate your tiddlywiki was insufficently sophisticated to use fields there will be a desire to search the unstructured content at least until you find a way to improve its structure by moving to fields. The fact is there are a range of methods to enhance search in tiddlywiki so you could possibly use a search method that can check the distance so search for “Etymology and ear” and perhaps ask that they be no more than N characters apart and you will have at least a fuzzy type of search (I dont recall yet where this distance check was but one exists), we can look for it if you want it.

To search the whole wiki based on the content of a given fieldname is trivial, even different partial matches are possible.

  • Add to that the ability to filter out based on something like object-type first and you have already applied a scope.
1 Like

How does one set that up? I’m terrible at front end stuff. I’m getting flashbacks to when I tried building an Access database at work. I read a whole textbook on DB design and everything. I loved the back end stuff, like tables and key fields and relationships, but when it came time to building a front end so that anyone besides me could use the DB effectively, I collapsed.

2 Likes

I’m throwing my hands in the air regarding fields. If we’re supposed to put everything in fields, we shouldn’t have design a bunch of HTML forms from scratch to add and look up such tiddlers.

If you’ve truly given up, then ok.

If you’re looking for suggestions, there are people willing to help, but I think we’d need actual questions.

Fields are one way of doing things. I have some wikis where almost everything is handled with fields and templates, and there’s little text content. I have others where I touch almost no fields except for the big-Ts, title, tags, and text. It depends upon my need.

2 Likes

The issues I have are:

  1. You can’t enter newlines into the standard field entry form when creating a tiddler, making it hard to make field-heavy tiddlers out of the box.
  2. The search box only searches title and text values, not any of the other fields, making it hard to search for field-heavy tiddlers out of the box.
  3. The solutions to these problems require a lot of verbose markup.

The example of a tiddler entry form given above, while impressive, is beyond arcane to me. Yes I’ve tried looking up the documentation, but I get confused. It’s the same sort of befuddlement I feel when people try to use YAML as a programming language.

The other big reason I’m moving away from fields is that they’re not hierarchical. I mentioned above that i have to distinguish between polysemy and homophony. The word Dbr can mean both to meditate and to research. That’s polysemy, one word with shades of meaning. B is both the verb to attempt and the preposition from, that’s homophony, two separate words that sound the same. These are indicated differently. Homophony in particular would be hard with fields, as you now have one entry–one tiddler-- with multiple parts of speech, so I can’t just have “partOfSpeech” as a single field.

Don’t get me wrong, TW still succeeds at the stuff that attracted me to it in the first place. It has case sensitive links and it’s very easily sharable, but the learning curve is steep.

EDIT: Oops, I missed the fact that @Springer had already provided a demo of multi-line fields above! I’ll leave this for posterity since I did try to outline the mechanics a little further.

If you would like multi-line fields, this one has a pretty easy fix. Here’s a package you can drag and drop into your wiki to try it out; a basic explanation follows. multiline fields.json (701 Bytes)

The edit widget used for a given field name is controlled by a cascade, as shown below:

Here, #2 shows the default core tiddler that will be used to render each field in the editor. #1 is an alternative that takes precedence over the default (since it appears higher in the list). To make $:/my/ui/EditTemplate/fieldEditor/textarea, I cloned the default tiddler and very lightly tweaked it to use a textarea rather than a single-line input.

As written (and as presented in my sample file above), this will affect all fields in the editor. If I wanted to change only a subset of fields, I could use a filter like

[<currentField>match[long]then[$:/my/ui/EditTemplate/fieldEditor/textarea]]
~[<currentField>match[select]then[$:/my/ui/EditTemplate/fieldEditor/select]]

Then, assuming that I had created $:/my/ui/EditTemplate/fieldEditor/textarea and $:/my/ui/EditTemplate/fieldEditor/select tiddlers containing the appropriate widgets, the long field would be globally rendered with a textarea, the select field would be rendered with a dropdown (based on the one in that template tiddler), and all other fields would use the default input.

No you can’t not out of the box. But many have needed this, and there’s plenty of discussion on it, including several solutions: https://talk.tiddlywiki.org/search?q=multiline%20fields.

Again, this is a fairly common need, but not common enough to make it to the core: https://talk.tiddlywiki.org/search?q=searching%20in%20other%20fields.

That is one drawback of using a generic toolkit rather than a dedicated tool; you need to do the customization.

Many users would look at that and see relatively simple markup. It depends on how much time you want to invest in it. And it takes time, I’m afraid, to get good at it.

I really hope people are only using that as a markup language! If they’re trying to program logic with it, that would be really scary!

No, neither are SQL columns. Columns are a very good analogy to fields, and tiddlers to rows, with certain tags mapping to tables. I don’t know if you know SQL, but I imagine the spreadsheet concept of rows and columns is familiar. SQL tables are also not hierarchical. Nor are spreadsheets. And columns could never be, not directly. But people model hierarchical relationships in SQL all the time. And in spreadsheets. We can do that too in TiddlyWiki by making hierarchical groups of tiddlers. It pays to remember this:

From The Philosophy of Tiddlers:

The purpose of recording and organising information is so that it can be used again. The value of recorded information is directly proportional to the ease with which it can be re-used.

The philosophy of tiddlers is that we maximise the possibilities for re-use by slicing information up into the smallest semantically meaningful units with rich modelling of relationships between them. Then we use aggregation and composition to weave the fragments together to present narrative stories.

TiddlyWiki aspires to provide an algebra for tiddlers, a concise way of expressing and exploring the relationships between items of information.

If I were to try to model this, just on the little bit I know, I would probably have one tiddler for each Term and a number of separate tiddlers for each of its Definitions

title: down
tags: Term
pronunciation: / daʊn /
title: def/down/1
tags: Definition
term: down
part-of-speech: adverb
example: to come down the ladder.

from higher to lower; in descending direction or order; toward, into, or in a lower position

These definitions are a selection of the more than 50 definitions of “down” found on dictionary.com

definitions
title: def/down/1
tags: Definition
term: down
part-of-speech: adverb
example: to come down the ladder.

from higher to lower; in descending direction or order; toward, into, or in a lower position
title: def/down/2
tags: Definition
term: down
part-of-speech: adverb
example: He fell down.

on or to the ground, floor, or bottom:
title: def/down/20
tags: Definition
term: down
part-of-speech: preposition
example: They ran off down the street.

in a descending or more remote direction or place on, over, or along
title: def/down/22
tags: Definition
term: down
part-of-speech: adjective

being at a low position or on the ground, floor, or bottom.
title: def/down/36
tags: Definition
term: down
part-of-speech: noun

a downward movement; descent.
title: def/down/41
tags: Definition
term: down
part-of-speech: verb-transitive
example: He downed his opponent in the third round.

to put, knock, or throw down; subdue:
title: def/down/42
tags: Definition
term: down
part-of-speech: verb-transitive
example: to down a tankard of ale.

to drink down, especially quickly or in one gulp:
title: def/down/45
tags: Definition
term: down
part-of-speech: verb-intransitive

to go down; fall.
title: def/down/46
tags: Definition
term: down
part-of-speech: interjection
example: Down, Rover!

(used as a command to a dog to stop attacking, to stop jumping on someone, to get off a couch or chair, etc.):

def/down/1

I would apply a ViewTemplate to all Term tiddlers like this:

title: Term ViewTemplate
tags: $:/tags/ViewTemplate

<% if [<currentTiddler>tag[Term]] %>

!! Pronunciation

{{!!pronunciation}}

!! Definitions

<$let prefix={{{ [<currentTiddler>addprefix[def/]addsuffix[/]] }}}>
<table>
<tr><th>id</th><th>part of speech</th><th>definition</th></tr>
<$list filter="[tag[Definition]term<currentTiddler>]">
<tr>
<td><$link><$text text={{{ [<currentTiddler>removeprefix<prefix>] }}} /></$link></td>
<td><$link to={{!!part-of-speech}}>//{{!!part-of-speech}}//</$link></td>
<td>{{!!text}}</td>
</tr>
<% if [<currentTiddler>has[example]] %>
<tr><td colspan="2">&nbsp;</td><td class="example">{{!!example}}</td></tr>
<% endif %>
</$list>
</table>
</$let>

While that might look like gobbledygook to you, we can help you break this down into meaningful chunks should you want to learn this technique.

In the end, we could get a tiddler like this:

You can try this out by downloading the following and dragging the resulting file onto any TW; I’d suggest tiddlywiki.com:

Definitions.json (3.2 KB)

Open the down tiddler to see the results.


But again, this may not be for you. TiddlyWiki is not a system to display linguistic information. It is a general-purpose tool that may allow you to build such a system for yourself.

2 Likes

Some possibly helpful information

The standard search in the sidebar is designed to not overwhelm people so only searches the afformentioned fields rather than all fields.

To place things in the side bar we use the tag $:/tags/SideBarSegment so place this in a tiddler
<<tag $:/tags/SideBarSegment>> and click on the tag pill to see the items in the sidebar.

  • You will see $:/core/ui/SideBarSegments/search is the tiddler that delivers the standard search. Open thius tiddler

The standard search is quite complex code because of the need to respond with each character entered immediatly after the third is entered by default.

  • However to cut a long story short there is a tiddler $:/core/ui/DefaultSearchResultList and if you edit it you will see some fields containing filters.

first-search-filter the “title maches” filter [!is[system]search:title<userInput>sort[title]limit[250]]

  • This only searches title as this is the most importiant way to search, and most meaningful

second-search-filter the “All maches” filter
[!is[system]search<userInput>sort[title]limit[250]]

  • This does not specify any fieldnames so uses the default for the search operator which says the fieldlist defaults to tags, text and title if blank, see below;

So if you alter second-search-filter it will search all fields (however only if !is[system] ie not if a system tiddler begining $:/)

I suggest to alter the filter to;
[!is[system]search:*<userInput>sort[title]limit[250]]
which will search all non system tiddlers fields.

Keep in mind you can use similar filters in the advanced search filters tab.

Documentation

The search operator uses an extended syntax that permits multiple fields and flags to be passed:

[search:<field list>:<flag list>[<parameter>]]
  • field list: a comma delimited list of field names to restrict the search
    • defaults to tags, text and title if blank
    • an asterisk * instead of the field list causes the search to be performed across all fields available on each tiddler
    • preceding the list with a minus sign - reverses the order so that the search is performed on all fields except the listed fields
  • flag list: a comma delimited list of flags (defaults to words if blank)
  • parameter: filter parameter

These are used to generate the search dropdown

I did create the field-search plugin. Which allows you to define which fields you want to search in.

I did create the field-editor plugin, which allows you to switch between fields you want to edit in multi-line mode. It will also install the field-view plugin

More details can also be found at: [Intro] Multiline Field Editor Plugin