[tw5] Ranked list of tiddlers according to number of backlinks

Hi,

I am interested in being able to get an overview of my wiki to identify tiddlers which seems to function as important “hubs” in my knowledge base - for instance if a tiddler has ten backlinks then it might be more important than one with say 2 backlinks - or perhaps I have neglected a tiddler because I have missed places which should ideally link to it.

I use the “giffmex” bi-directional back link plugin so the required data is already in place.

It’s all speculative, I am just interested in seeing if tools can show relationships in my knowledge base which is 20MB which perhaps I cannot always appreciate.

I do not regularly code in Tiddlywiki so I started off with existing code to hack…

https://groups.google.com/g/tiddlywiki/c/j5oVXLCErE8/m/okRgDg9GCgAJ

My experimental tiddler code now looks like this…

<$list filter="[all[]]" variable=“tiddler”>
<$list filter="[backlinks[]count[]]" variable=“count”>
<$reveal type=“lteq” state="$:/temp/Testing!!minimum" text=<> >
<$reveal type=“gteq” state="$:/temp/Testing!!maximum" text=<> >

<> <$link to=<> />

</$reveal>
</$reveal>
</$list>
</$list>

It’s a good start - it displays a list of all my tiddlers with the number of backlinks for each tiddler displayed - the next step would be to order the list so that tiddlers with the most backlinks appear higher in the list.

I looked at the idea of adding a new field to each tiddler to hold the number of backlinks - if so then so far have not found out how to add a new field (if it does not already exist) and write the value ‘count’.

So far performance is not an issue.

I am interested in any existing plugins or attempts to extract this kind of information from large wikis - no point re-inventing the wheel!

Thanks
Jon

I found a useful lead from Eric Shulman using subsort
https://groups.google.com/g/tiddlywiki/c/hZBWTSo8eCw

Using this I can order in the filter directly.

My filter looks like this
<$list filter=’[!is[system]!tag[Journal]sortsub:integerlimit]’>

And the macro sub looks like this
\define sub() [backlinks[]count[]]

Thought I would leave this for the next person.
Jon

2 Likes

Thanks a lot for sharing this ! sortsub is a difficult filter for me to understand, examples like yours are great.

Jon,

My filter looks like this
<$list filter=’[!is[system]!tag[Journal]sortsub:integerlimit]’>

you can use :sort as a simpler solution

https://tiddlywiki.com/#Sort%20Filter%20Run%20Prefix

3 Likes

Many thanks Mohammad - I will try that out.

Findings: Like many interesting problems I do not have a completely clear picture what I need but it is becoming clearer.

At the moment I have two new buttons on my side bar - there will be one when I have added some kind of control to rank in increasing or decreasing - until then I am happy to prototype with two buttons.

Already I am finding some good ideas in tiddlers that are over one year old and yet had zero back links and so were only really ‘discoverable’ through tags - I thought for instance I had discovered something new recently but it turns out that the seed of the idea was over a year old - my earlier tiddler had some good ideas that I had become disconnected with. Of course older tiddlers often lack some tags and links precisely because they are old and written at a time when those additional tags and similar tiddlers did not yet exist. Many of my tiddlers although single topic contain 500 to 1000 words.

Backlinks are a very good indicator but they do not tell the whole story - they are asymmetric to my ranking code because I only detect the backlinks and not links in the regular text entry of the tiddler so sometimes there is a really useful tiddler that has a lot of backlinks and clustered around that useful tiddlers perhaps even more useful ones that have no backlinks because I originally linked in one direction.

I have played around also with star ratings plugin $:/plugins/tobibeer/rate - where the field to contain the integer star rating is held by an integer ( 1 to 5 ) as a dynamically added field in the tiddler - by this I mean that the field is added to the tiddler when you first assign a star rating - it also disappears if you later change your mind and give that tiddler a zero rating.

Also the upvoting tool from Kooma $:/plugins/kookma/vote - if I understand this one correctly it does not actually add a field to the tiddler but instead maintains it’s own table of votes for all tiddlers which is held in a special tiddler? I stand to be corrected on this one :slight_smile: If my understanding is correct then I did find this one less useful - I would have preferred a field added to the tiddler being voted on so that code written by me could then be independent on Kooma’s code and then simply display tiddlers in the main story river according to how my own filters interact with the vote number stored in the tiddler.

Finally I have my own tags called simply “1” and “2” - these I assign to tiddlers I find particularly useful a bit like “silver” and “gold” medals - currently I do not have “bronze” - there is no “3”.

Are these lots of different ways of solving the same problem? Yes certainly but that does not mean that they are redundant, each has it’s strengths and weaknesses.

For instance the strength and weakness of both the stars plugin and the upvoting plugin are that you do not have to go into edit mode just click on the widget - this does also mean that accidental voting may occur - this would be easy to do on my mobile phone for instance - consequence could be that one of my most useful tiddlers disappears to the bottom of the pond again.

There is psychology here as well - for instance I have resisted adding “3” and then “4” and “5” to my own tagging system because I know that once a tiddler has been assigned “5” ( lowest score in my case ) then I will probably never look at it again and so it loses the opportunity to be reconsidered in the light of changes to my understanding of the area my knowledge base concerns, this is infact why I also have a button on my sidebar called “random 10” - it simply shows a random ten tiddlers on my story river so that every so often I shake the dice and give myself the opportunity to reconsider tiddlers that would otherwise receive no attention.

Possibilities also include ranking on the lines of “importance by association” - tiddlers that are ( for instance ) linked to high ranking tiddlers are themselves considered of high rank but not necessarily assigned a high ranking score directly - so a dynamic decision at the time the story river is populated.

For the time being then I advocate quite a number of different approaches - the whole issue of assigning importance to tiddlers is complex and at least as complex as the way the knowledge base has evolved and been interlinked.

What I do feel is that it would be very nice if the standard wiki - out of the box so to speak facilitated ranking by having user available fields - probably just integers provided for the purpose or if official support was provided for the kind of dynamic fields that tobibeer provided - it would be great to have functionality capable of visiting every tiddler according to a filter and updating the various rankings by means of user registered functionality or simply published frameworks for this.

The goal for a support framework, a tool kit if you like would be to

  1. Simplify plugins written with the purpose of assigning integer rank to tiddlers and have a common way of storing the rank. I do not favour rank being stored in special tiddlers because I think it is less accessible to developers having a rudimentary knowledge of Tiddlywiki - for instance in my case I can write filters and add pill buttons to my side bar to filter on criteria I consider of interest but I do not have the technical expertise to write the above plugins mentioned above - I can usually tweak them after study of the code but that is all.

  2. Provide methods for updating ranking not just on a single tiddler when someone has voted on it but on all tiddlers - for instance the ranking based on the number of backlinks may require periodic refresh although I do not believe it would necessary require automatic update - I would happily press a button once a day to update.

The justification for wanting common ground - a best practices toolkit is to try and ensure that the methods for assigning, updating and reading rank are similar and suited to as many ways of working as possible - for instance I found it much easier to create buttons for the tobibeer ranking system ( to display on the story river ) because I just had to react to a field stored in the tiddler containing a simple integer - I found this less easy with Kooma’s voting system because as far as I understand it the rankings for all tiddlers are stored in a special tiddler rather than the actual tiddler being ranked. I think if the storage method and the methods of access, update and so on have a common feel then we might see more and more interesting ideas develop which can be adapted to suit the individual user. Ideally I would like to focus on the algorithm that calculates rank ( if rank is not a simple user assignment as it is in the case of upvoting and stars ) and probably develop several different models in the attempt to better understand the connections and priorities in my own knowledge base. It is very unlikely that one view of the world will satisfy every user and every wiki - we all think and organise differently but I do feel a vanilla toolkit to facilitate the housekeeping activities would enable non-expert coders in Tiddlywiki to focus on the meat of the problem rather than the housekeeping.

Sorry it’s so long but I think ranking is an absolutely critical part of any large knowledge base wiki particularly if the knowledge base is not focused on a hard science where there may be fewer alternative views or perspectives on a particular area of study where the individual may find their viewpoint is in constant development and the knowledge base is constantly in revision, review and mining for new connections and explanations.

Sorry reading through my last post I see some areas where I was not clear.

I am suggesting each tiddler would have a number of integer slots for the use of any plugin that wants to assign some level of rank either by direct user intervention or by algorithm ( counting backlinks for instance ) and a toolkit of functionality to support these fields.

Plugins wanting to use this system would need a mechanism to negotiate which ranking slot they were assigned to use.

A general vanilla toolkit to carry out and facilitate as much of the housekeeping as possible without dictating how rank is calculated or indeed used - a neutral framework which makes as few assumptions as possible about the inside knowledge of the ranking system - by means of analogy anyone who earns a wage or has a bank account is probably an indirect user of financial and accounting software so we all benefit from the same ground level software infrastructure but for the most part the software does not involve itself with how we earn the money or spend the money - so I see ranking of tiddlers a little like money - you can count how much and order in terms of the amount but everyone will have a different idea where that money came from and where it might go ad so it is with how we value and rank tiddlers.

1 Like

Some advantages and disadvantages of the upvoting way of working…

The upvoting plugin has an upper limit of +99 and a lower limit of -99 with colour coding so that negative votes are a different colour which explains why it shows rank visually as a number on the tiddler instead of a row of stars !!!

There is great freedom here but it shares the same issue as the star ranking system which is what I call voting inflation.

You see it when judges award marks out of ten for a performance, once a judge as awarded a perfect ten then the judge is in trouble if a better performer comes along next.

One solution to this problem is to apply an ageing principle so that in the case of upvoting each time a new vote on a particular tiddler is cast the total number of pre-existing votes is calculated, this existing total is then eroded a little by applying a factor - say 0.95 - each tiddler apart from the one being voted on right now is then assigned a share of that diminished total based on it’s previous share - so every vote apart from the new one keeps it’s relative position with respect to the others but non of the older voting scores can be a perfect ten - only the latest ranking or re-ranking can receive a perfect ten. It’s a crude model I know - in reality it would need work but the principle is there to overcome voting inflation which occurs because perhaps we cannot actually really compare the latest tiddler with every other tiddler we should really compare it with to vote accurately - instead we end up using upvoting as a way to push something a little higher or lower in the rankings having a vague awareness it deserves to be higher or lower but without having the time to really go into it in detail - the hope is that although the decision is made without full consideration, if up voting or down voting occurs frequently enough then it will eventually reflet our wishes - I suppose most of the time, if we were looking at a story river ordered by vote then we are just judging each tiddler relative to ones in the same neighborhood. I suppose voting inflation occurs if we tend to favour moving items down rather than up but in some cases it is simply easier to move the deserving tiddler up rather than move some of it’s neighbors down?

Sorry when I wrote

One solution to this problem is to apply an ageing principle so that in the case of upvoting each time a new vote on a particular tiddler is cast the total number of pre-existing votes is calculated, this existing total

Of course I meant the total of all existing scores not the number of all existing scores - if there were 100 tiddlers each with a score of 10.0 then I am referring to the total score of 1000.0