Help with templates, fields, and searchability

#1

I am new to TW, so new that I don’t know what questions to ask, so forgive me for info dumping. I don’t know what information is necessary and what isn’t.

I have a large (~780 notes) Obsidian vault that I am attempting to port to TW to make it more shareable online. It contains the lexicon of a constructed language I’m developing.

The markdown files have a more or less identical structure patterned after Wiktionary entries. The file name is the lemma (dictionary form) of the word. Each entry has at a minimum a definition marked by a level 1 heading. A pronunciation, etymology, and other sections such as usage notes and derived terms may or may not be present, each also marked with level 1 headings. The content of most sections is somewhat loosely formatted. All definitions have a part of speech as a level 2 heading, but may contain number lists, bullet lists, or separate paragraphs. Etymologies and derived terms sections usually contain links to other entries.

Obsidian uses markdown, which can be accommodated by TW using a plugin. It would seem straightforward to simply drag and drop the relevant .md files on top of TW to import them, and indeed that works well enough, but I’m slowly learning that that isn’t how you’re supposed to do things if you want to search, sort, and parse tiddlers. Instead, you’re encouraged to put info in fields.

I can parse these sections into fields by using the level 1 headings as field names and use wikitext for the values, but how do I present these tiddlers? How do I make them searchable? The search box on the sidebar only looks at titles and body text. How do I create new entries given that the field value boxes on the new tiddler creation menu don’t allow line breaks?

Permit me a digression to complain about a recurring problem I see in note taking software that TW also suffers from. I see this problem in Obsidian and Trillium as well. All these let you enter nice markup to structure a document with headings, lists, tables, etc, but they don’t take that structure into account when searching or parsing. I can’t search for text under a specific heading. I can’t search if a specific heading exists, I can’t transclude the text under a specific heading, or a specific table etc.

2 Likes
#2

Apologies, I can’t help you with any of that.

I do suggest, though: if some thing is worth transcluding, there is wisdom in having that thing exist in its own dedicated tiddler. That is a componentization approach.

Distinct things that should be easy to find, best to not bury them inside other content.

#3

That’s how I assume things should be. Use the text for the actual content and leave fields for metadata.

#4

Hi, Welcome!

Do you have an example of 1 or 2 markdown files, so we can see the structure?

#5

Sure, things can be put in the fields of a tiddler.

There are other things, though, that are better suited in dedicated tiddlers for the purpose of reuse via Transclusions.

For example:

image
image1393×762 97.8 KB

The text for the testimonial by Joe Armstrong exists in a tiddler dedicated to that purpose.

The “Testimonials and Reviews” template could have all of the testimonials existing in the body of that one tiddler. But then it is not possible to have links to any one specific testimonial (because, as you already described, that feature does not exist.)

Better for “Testimonials and Reviews” to just be about the display of various testimonials that exist in their dedicated tiddlers.

My original comment was about content in dedicated tiddlers (for which links are easy), and not about putting content in fields. Content in fields is useful for the purpose of transcluding content, but not useful for links to the content, whereas content in the body of a tiddler is good for both Transclusion and links.

#6

The text between dashes is YAML-formatted frontmatter. That’s the sort of stuff I’d expect to put into a field. The links are in markdown format, but Obsidian also accepts wiki links, and I converted them using a plugin before importing them to TW.

The reason the in-language text looks weird is that the “Romanization”, such as it is, is case sensitive, which won’t work with markdown links (since they’re based on HTML) nor the case-insensitive Windows filesystem. So I had to hack it using the mathematical alphanumeric symbols Unicode block. One of the principle reasons I want to switch to TW is that links are case sensitive.

Here’s a link to the wiki itself if you’re curious.

𝚙𝚛 𝚛𝙳𝙱.md

---
entry id: 712
---

# Pronunciation

short high strong grunt, chuff. chuff, long falling weak whine

# Etymology

[𝚙𝚛](𝚙𝚛.md) (reciprocal coverb) [𝚛𝙳𝙱](𝚛𝙳𝙱.md) (to speak, say)

# Definition

## Verb

To discuss, speak amongst.

# Usage

The topic of discussion may be placed as the object of 𝚛𝙳𝙱. The subject is a plural noun or a noun denoting a group, such as [𝙻𝙼𝚛𝚐](𝙻𝙼𝚛𝚐.md) (moot, assembly) or [𝚛𝙿𝚚𝙿𝙻𝚐](𝚛𝙿𝚚𝙿𝙻𝚐.md) (clergy), or else the other discussion partner may be placed as the object of [𝚙𝚛](𝚙𝚛.md).

𝚛𝙶𝙷𝚐.md

---
entry id: 444
check: false
imported: true
---
# Pronunciation

chuff, long low strengthening growl, short low weak growl

# Definition

## Noun

mech, heavy powered armor
#7

I think your imported tiddlers look fine. The links seem to work.

The only thing that seems to have happened, is that your search function does not work anymore.

I did download empty.html from tiddlywiki.com and I did re-import all your tiddlers from your commonthroat.html into empty.

Save and reload, then the search function worked for me.

I personally would define a modified: 20250518 field for every tiddler. So they are visible in the Recent tab.

See docs about the TW date format for more details.

I personally would not change the text portion. – I would add new fields that may be useful in the future for sorting.

Just some thoughts.
-Mario

#8

Interesting. The search seems to work for me. Everywhere I’ve tried it, on whatever instance of TW, it gives you a list of titles and body text that match your search.

#9

I’m among those who would absolutely go with field-structure. Data architecture is beautiful. Flattening data dimensions into ordered text beneath headings is… not.

I suspect you’ll eventually run into situations where search results will be unmanageably cluttered if you don’t have field-structure… For example, let’s find a word whose definition includes “weak” … Oh no! — weak is part of the text of so many tiddlers because it’s a common string in the pronunciation field! Being able to search within a specific field solves such problems.

Another advantage of creating real data structure from the start is that it’s so easy then to come up with on-the-fly variants to your UI: You can make one template for a quick-reference “glossary” that filters according to some condition of interest and formats each definition with a hanging indent but omits etymology and usage notes, for example, while another template foregrounds usage notes up top, puts etymology in italics at the bottom, etc., ordering the fields and applying nice visually satisfying css to them according to whatever the your purpose requires…

You can get a plugin that helps you search in specific (or any filtered subset of) fields. Here’s one:

EDIT TO ADD: You asked about multi-line fields. Yes, you can absolutely have multi-line and multi-paragraph fields:

and here’s a small demo I made.

Let us know what you decide and how it’s going!

#10

I’ve already run into that problem. I have another lexicon for a different language spoken by the same critters that I started just earlier this year that’s much more of a green field for my TW explorations. The first thing I did was come up with a more compact, and more importantly search friendly, phonetic transcription system, as can be seen here.

Regarding the “put everything in fields” concept, it’s more or less what I was complaining about earlier. Wikitext (rather the HTML rendered from it) is already structured. Why not lean into that structure? Again, TW is by no means the only software guilty of this.

#11

I find fields to be super-powerful in their relation to TiddlyWiki’s filters and templates. But if wikitext structure hits your sweet spot, then I certainly will cease evangelizing for fields!

#12

An approach I use often with data from an external source is to (when necessary) write some code to parse the input format into a convenient representation, then write an additional bit to convert that into JSON tiddlers with title, tags, and text fields plus whatever else seems useful. I tried that with your data.

The JS Code creates JSON tiddlers like this:

    {
        "title": "HKrqFdqg",
        "tags": "lexicon",
        "entry-id": "581",
        "pronunciation": "long rising strong growl, chuff, huff, late high weakening whine, huff, short low weak growl",
        "etymology": "[[HKrg]] (notch) + [[qFdqg]] (ear)",
        "part-of-speech": "Noun",
        "definition": "ear notch (derogatory term for a firearms enthusiast)"
    },

I save the output to a file and drop it into an empty wiki, drag your Hello! tiddler over and, since there is no text content in any of these new tiddlers, add a ViewTemplate that looks like this:

title: viewtemplates/lexicon
tags: $:/tags/ViewTemplate
list-after: $:/core/ui/ViewTemplate/body

<% if [<currentTiddler>tag[lexicon]] %>
<dl>
<$list filter="pronunciation etymology part-of-speech definition usage usage-notes notes note note- related-terms derived-terms lore entry-id" variable="field-name">
<% if [<currentTiddler>has<field-name>] %>
<dt><$text text = {{{ [<field-name>search-replace:g:[-],[ ]titlecase[]] }}} /></dt>
<dd><$wikify name="content" text={{{ [<currentTiddler>get<field-name>] }}} output="html"><<content>></$wikify></dd>
<% endif %>
</$list>
</dl>
<% endif %>

This template is fairly naive, just including a definition list for all the custom fields, in an order I guessed might be close to what you’d want. (It also points out that you have fields of note, notes, and note-, which seems odd!) The resulting tiddlers look something like this:

image
image1061×804 29.7 KB

You can see this result at http://scott.sauyet.com/Tiddlywiki/Demo/Commonthroat.html.

This is far from a complete solution. But it might be a start on the way there.

If it’s close and there’s something in my process you want to tweak but don’t know how, just let me know. The code for the conversions is very clear to me, but could well be obscure, or just gobbledygook to others.

2 Likes
#13

I would be more inclined to use them in the way you suggest if TW made it easier out of the box to work with them. Something like prefixing a search with field_name:value would search for that value only in that field. I know the advanced search can do that, but why push such a powerful feature further up the learning curve? Same thing with not allowing line-breaks in the easy-to-access field entry boxes when creating a new tiddler.

My other issue with using fields for my use case is that fields are not hierarchical. Dictionaries usually distinguish between polysemy (a single word having multiple related meanings) and homophony (two unrelated words that sound the same). Polysemous meanings are grouped under one heading, while homophonous meanings are separated.

I will check that out. I’m currently normalizing those obsidian markdown files precisely to turn them into a single monolithic JSON file using Python.

#14

OK, I’ve reorganized my Outlander wiki to put everything in fields. Now I need to figure out how to make those fields searchable.

#15

I’ve implemented your view template here. It seems to work well. Entering words this way would certainly make porting the lexicon to other software much easier. Still need to figure out how to make them searchable and how to quickly enter new entries. Obsidian’s biggest strong suit was that very little got in the way between my ideas and the page. I didn’t have to think of what button to push to format a level 1 heading or start a table. It was all done right there in the body of the note, without the verbosity of HTML markup.

#16

One difference between my parsing and yours was that I further parsed the definition section to extract the part-of-speech. I think that’s useful, but YMMV.

I would not expect to create a tiddler and add the fields manually. I would create an entry form. Here is a sample, adapted from a question of mine from last year:

EntryForm.json (2.9 KB) (download this and drag the resulting file onto any wiki.)

For searchability, another topic discusses way to modify the default search, although I’ve personally only altered its output. But you can also write a custom search using the search Operator, which allows you to search in specific fields or in all fields.

#17

I’m trying to strike a balance between granularity and ease of entry.

Another thing I hadn’t considered is the ability to track backlinks and missing entries (links to missing tiddlers).

#18

Supporting the argument for fields it is the best way and the skys the limit when searching based on a fieldname and value or part there of.

Although I understand if the content you populate your tiddlywiki was insufficently sophisticated to use fields there will be a desire to search the unstructured content at least until you find a way to improve its structure by moving to fields. The fact is there are a range of methods to enhance search in tiddlywiki so you could possibly use a search method that can check the distance so search for “Etymology and ear” and perhaps ask that they be no more than N characters apart and you will have at least a fuzzy type of search (I dont recall yet where this distance check was but one exists), we can look for it if you want it.

To search the whole wiki based on the content of a given fieldname is trivial, even different partial matches are possible.

  • Add to that the ability to filter out based on something like object-type first and you have already applied a scope.
1 Like
#19

How does one set that up? I’m terrible at front end stuff. I’m getting flashbacks to when I tried building an Access database at work. I read a whole textbook on DB design and everything. I loved the back end stuff, like tables and key fields and relationships, but when it came time to building a front end so that anyone besides me could use the DB effectively, I collapsed.

2 Likes
#20

I’m throwing my hands in the air regarding fields. If we’re supposed to put everything in fields, we shouldn’t have design a bunch of HTML forms from scratch to add and look up such tiddlers.