Tiddlywiki Indexer: Create Subject/Author Indexes

#1

From Wikipedia
An ‘index’ (plural: usually indexes , more rarely indices ; see below) is a list of words or phrases (‘headings’) and associated pointers (‘locators’) to where useful material relating to that heading can be found in a document or collection of documents. Examples are an index in the back matter of a book and an index that serves as a library catalog.

Tiddlywiki Indexer

Tiddlywiki indexer contains two parts:

  1. an index-entries tiddler, where important words or phrases are stored

    • this is simple (here a plain text) tiddler
    • default title for this tiddler is WikiIndexEntries

  2. a macro to create the index

  • here it is called wiki-indexer
  • the indexer indexes the Wiki based on preset fields, by default title, tags, caption are indexed
  • the scope of indexing can be set, the default is [all[tiddlers]!is[system]]
  • this macro is modular and one can customize it

NOTE: By default wiki-indexer will index the whole wiki.

Example

  • In tiddlywiki.com the WikiIndexEntries was created as below (few words/phrases will be indexed.)
Android Action ActionWidget Article
Bug
Command Concept Community Customize Chrome
Operator
Definition Debug
Edition Editor Edge
Feature Field Filter Firefox Font [[Filter Run Prefix]]
Google
Html
Icon Image Import Internet IOS
JavaScript
Keyboard
List Listops Linux Learning
Mathematics Mechanism Message Modules Mac Macro
Plugin Platforms
Reference Release Resource
Safari Saving SVG Stylesheet
Toolbar Table Tag Tutorial Template
Video Variable
Widget Windows Wikitext
  • Then in another tiddler the indexer was called as
<<wiki-indexer scope:"[all[tiddlers]!is[system]]"  fields:"title,tags,caption">>

This produces:

img_848_msedge

NOTE Wiki indexer will be published in the next update of Thinkup. I would appreciate ideas, comments, …

7 Likes
Tiddlywiki Indexer Plugin: Public Release
#2

Feedback: To claim indexing a whole wiki, the text field must be offered, too, I believe.

EDIT: Ah, I see, the fields list can be changed. Good stuff @Mohammad

#3

Quite true! You can simply pass any field you like, for example

<<wiki-indexer 
  scope:"[all[tiddlers]!is[system]]" 
 fields:"title,tags,caption,text, description, keywords"
>>

I just excluded text in experimental version, as with my entry selection for tiddlywiki.com it produces too much results. A good selection of entries is needed!

#4

Very interesting.

Do I understand it right that you manually create the entry tiddler. This is the (black+bold) headline and under this all the listed items under it are tiddlers that contain that entry? So you would get the same results if you typed the entry word into the search tool, assuming the search tool searches the same fields. Right?

#5

Yes, like LaTeX, an index-entries tiddler holds all important words, phrases

True! You can set the fields and the indexer searches only in those fields and if finds the word/phrase, then that tiddler will be listed

Yes!

The indexer also have pagination in alphabetical order!

#6

Cool. I like this.

I have no idea if the following are good ideas so FWIW:

A different UI would be to put the A, B, C etc in the sidebar. Clicking a letter opens a tiddler dedicated to showing the “results” (and, like in you demo, its contents change depending on which index letter you click).

Or maybe, somehow, the resultslist could open in the regular “search popup”.

I can imagine that it would be useful to be able to quickly add an entry, at least for the first period of time one uses it. One UI for that might be a little text field and an OK button next to it that appends the entered entry into the list.

1 Like
#7

Interesting! I like the idea! I think using the sidebar is more Tiddlywiki way!

Very cool! I like this!

Mat, go for this, I think your idea is more natural and closer to published indexes in books!
If I can help let me know!

#8

@twMat
More inputs:

  1. I read the index-entries and enlist them
  2. Next I look for entries in search scope (default is all non system tiddlers) and given fields
  3. If step 2 has non-empty output, I show the entry and all items have that entry in given fields
#9

When this settles down, it’s going to very useful.

:heart:

1 Like
#10

Mohammad,

Thanks for looking at establishing a solution for this important need within tiddlywiki knowledge and information management.

A few suggestions;

  • When listing more than N tiddler titles for a given keyword provide a more, or pagination method so we can see all keywords under a “index letter” at a view, then click to open.
  • Allow for more than one index-entries tiddler eg; use tiddlers tagged $:/tags/index-entries, this would allow keyword sets to be imported from different sources to support indexing.
    • One of these index-entries tiddlers can be generated from an analysis of all text fields and the exclusion of “stop words”.
    • Perhaps later users can choose one or more of Scope and/or the index-entries to see alternative “curated indexes”, for example a list of species, would only index species names in tiddlers.

The following ideas need not all be included in this tool, but I hope you could design it such that the future uses become possible.

Using index-entries in creation and curation
Your solution so far extracts the keywords from the search of nominated fields, however for larger repositories more use can be made of index-entries, not just for the index you demonstrate here.

  • Perhaps we can provision a method to capture from the index view found keywords and choose to add them to a named index-entries tiddler?
  • With index entries tiddlers , we can actually use them to suggest values, for example a keywords field in a new or tiddler under revision. That is as we build index keywords we can then use these to classify content “moving forward”. That is the content generates the index, the index informs the content, the content adds to the index.
  • one could enter / import a list of keywords not necessarily already in the wiki, then use this to classify existing tiddlers and then build the index. For example we may already have animals and species but we may then add a taxonomy.

Arguably index-entries tiddlers can be developed that people will share both to explore existing content or to help them organise the content.

Much of what I suggest here is more about tiddlywiki meta concepts, which can be hard to communicate or apply across wikis, but by ensuring your indexer provides some generic methods then we encourage a de facto standard we can leverage and share, to make use of these meta concepts.

1 Like
#11

Is this similar to core/modules/indexers like backlinks-index.js ?

Seems it is a visual index widget for user to read, but not an indexer for filter operators like backtranscludes to read?

#12

I see! There are some good indexers for tags, fields, backlinks there! I have almost zero knowledge of JS, if I had I could use this !
But would you please explain me if what I did above can be done through such indexers?

2 Likes
#13

Hi @TW_Tones,
I like all your comments! Part of them were implemented, part were not!
I think most of them can be implemented!
I will update you here!

2 Likes
#14

Mamma mia! Do you ever sleep?

That looks brilliant! But you so far ahead of me: I ran out of time to enjoy it.
Hopefully, later I may have time enough!

Best wishes
TT

1 Like
#15

Oh my, this looks really great! Amazing work, Mohammad, you have really been on a killer winning streak with plugins lately!

Q: Are the entries case-sensitive, or will they catch both Job and job?

I may be asking for the moon with this suggestion, but if eventually there were a way to search only headers in the text field [!! This is an h2 header with the word debug] but not the rest of the text field, that might weed out a lot of unhelpful results from the text field.

Anyway, wow, I can’t wait to try this.

1 Like
#16

You have all the controls! now it looks in case insensitive, but you can force it to be casesensitive!

1 Like
#17

I went for a walk, and this idea generated more thoughts.

  1. You may need to find a way to ‘user-proof’ one aspect of this. Specifically, users may think that by tagging or adding keywords to a tiddler, that those words will show up in the indexing, just as they do in list widgets and other things. But there is a second step of adding those tag words or keywords to the index tiddler. If they forget that step, then their activity of adding tags and keywords will not produce what they want. I am not sure what to do about it, but the thought occurred to me and I thought I better mention it.

  2. Not having seen the plugin yet, I am wondering about multiple word entries (e.g., “Media criticism”). Could this be adjusted to handle those? Or maybe it already does?

  3. A thought for the future of this plugin, or a separate iteration of it: I wonder if it would be possible to hack it to use other parameters than the letters of the alphabet, for example, broad domains. Example:

‘Rest and recreation’ Games Hobbies “Media criticism” Movies Television

Where ‘Rest and recreation’ is the domain, and it takes the place of a letter of the alphabet.

2 Likes
#18

Thanks a lot for once again sharing a great tool @Mohammad !

As requested, here are some suggestions :

  • A search field to filter the index (full-text search and/or search in the index itself).
  • A way to display results for certain topics only. Hard-coded in the macro call or with a generated list of toggleable checkboxes for each main topic.
  • Several ways to order the list : alphabetical, number of backlinks, according to a field
  • Support for cross-references (e.g : " Felidae. See Cats")

And some inspirations/guidelines I’ve found :

1 Like
#19

@DaveGifford
Thank you for your feedback!

These can be added to index as [[multiple words]] and the exact word can be searched!

I think you can do all of this, I polish the code and soon I share a minimal working version!

Your comments and references are much appreciated! I added them to my todo list and I will try to see how I can implement if not all of them at least part of them! Thank you

2 Likes
#20

Comes up as page not found. Let us know if you find a new version of that. Sounds interesting.