Composing titles that appear at the end of the "All" listing

[jeremyruston]

The “All” listing in the sidebar sorts punctuation before letters. Somebody asked on Mastodon whether there was a character that would result in a tiddler being placed at the end of the list, other than starting the title with “Z”.

To investigate, I wrote a quick snippet of JavaScript that sorts the first 2000 or so Unicode characters. The result looks like this:

€‚ƒ„†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ­͏҈҉ֽ֑֖֛֢֣֤֥֦֧֪֚֭֮֒֓֔֕֗֘֙֜֝֞֟֠֡֨֩֫֬֯
ׅׄ؀؁؂؃؄؅ؘؙؚؐؑؒؓؔؕؖؗ؜ـۖۗۘۙۚۛۜ۝ۣ۪ۭ۟۠ۡۢۤۧۨ۫۬܏̸̧̨݄݈̲̖̗̘̙݀݃݇݉݊̓̓҆̔҅́́̀̀̆̂̌̊͂̈̈́̋̃̇̄̍̎̒̽̾̿͆͊͋͌͐͑͒͗͛҄҇݁݅̕̚͝͞
̶̷̵̴ְֱֲֳִֵֶַָׇֹֺֻּֿׁׂًٌٍَُِّْٰܑ̡̢̛̜̝̞̟̠̩̪̫̬̯̳̺̻̼͇͈͉͍͎͓͔͕͖͙͚݂݆̣̤̥̦̭̮̰̱̹ٕٟٖٜܱ̅̉̏̐̑҃ٓٔٗ٘ٙٚٛٝٞܰܲܳ͘͜͟͢͠͡ͅ
ܴܷܸܹܻܼܾܵܶܺܽܿ…  _-֊,՝،؍٫٬;;؛:։؞܃܄܅܆܇܈!¡՜?¿՞؟܉.۔܁܂··؝܀'"«»()[]{}§¶@*٭/\&#%٪؉؊՚՛՟־׀׃׆׳״܊܋܌܍`´΄˜^¯˘˙¨΅˚˝
¸˛ʹʹ͵ʺ˂˃˄˅ˆˇˈˉˊˋˌˍˎˏ˒˓˔˕˖˗˞˟˥˦˧˨˩˪˫ˬ˭˯˰˱˲˳˴˵˶˷˸˹˺˻˼˽˾˿°҂֍֎؈؎؏۞۩©®϶+±÷×<=>¬|¦~؆؇ːˑ¤¢$£¥֏؋0٠۰߀1١۱߁¹½¼2
٢۲߂²3٣۳߃³¾4٤۴߄5٥۵߅6٦۶߆7٧۷߇8٨۸߈9٩۹߉aͣAªáÁàÀăĂâÂǎǍåÅǻǺäÄǟǞãÃȧȦǡǠąĄāĀȁȀȃȂæÆǽǼǣǢȺɐɑɒbBʙƀɃɓƁƃƂcͨCćĆĉĈčČċĊ
çÇȼȻƈƇɕdͩDďĎđĐðÐȸdzʣDzDZdžDžDŽʥʤɖƉɗƊƌƋȡeͤEéÉèÈĕĔêÊěĚëËėĖȩȨęĘēĒȅȄȇȆɇɆǝƎəƏɛƐɘɚɜɝɞʚɤfFʩƒƑgGǵǴğĞĝĜǧǦġĠģĢɡɢǥǤɠƓ
ʛɣƔˠƣƢhͪHʰĥĤȟȞħĦʜƕǶɦʱɧʻʽiͥIíÍìÌĭĬîÎǐǏïÏĩĨİįĮīĪȉȈȋȊijIJıɪɨƗɩƖjJʲĵĴǰȷɉɈʝɟʄkKǩǨķĶƙƘʞlLˡĺĹľĽļĻłŁŀĿljLjLJʪʫʟƚȽ
ɫɬɭȴɮƛʎmͫMɱnNńŃǹǸňŇñÑņŅnjNjNJɴɲƝƞȠɳȵŋŊoͦOºóÓòÒŏŎôÔǒǑöÖȫȪőŐõÕȭȬȯȮȱȰøØǿǾǫǪǭǬōŌȍȌȏȎơƠœŒɶɔƆɵƟɷȣȢpPƥƤɸqQȹʠɋɊ
ĸrͬRʳŕŔřŘŗŖȑȐȓȒʀƦɍɌɹʴɺɻʵɼɽɾɿʁʶsSˢśŚŝŜšŠşŞșȘſßʂȿʃƩƪʅʆtͭTťŤţŢțȚʨƾʦʧŧŦȾƫƭƬʈƮȶʇuͧUúÚùÙŭŬûÛǔǓůŮüÜǘǗǜǛǚǙǖǕ
űŰũŨųŲūŪȕȔȗȖưƯʉɄɥʮʯɯƜɰʊƱvͮVʋƲʌɅwWʷŵŴʍxͯXˣyYʸýÝŷŶÿŸȳȲʏɏɎƴƳȝȜzZźŹžŽżŻƍƶƵȥȤʐʑɀʒƷǯǮƹƸƺʓþÞƿǷƻƨƧƽƼƅƄʔɂɁˀʼʼn
ˮʾʕˤʿˁʡʢʖǀǁǂǃʗʘʬʭαΑάΆβϐΒγΓδΔεϵΕέΈϝϜͷͶϛϚζΖͱͰηΗήΉθϑΘϴιͺΙίΊϊΪΐϳͿκϰΚϗϏλΛμµΜνΝξΞοΟόΌπϖΠϻϺϟϞϙϘρϱΡϼσϲΣϹςͼϾͻ
ϽͽϿτΤυΥϒύΎϓϋΫϔΰφϕΦχΧψΨωΩώΏϡϠͳͲϸϷϣϢϥϤϧϦϩϨϫϪϭϬϯϮаАӑӐӓӒәӘӛӚӕӔбБвВгГѓЃґҐғҒӻӺҕҔӷӶдДԁԀђЂԃԂҙҘеЕѐЀӗӖёЁєЄжЖӂӁ
ӝӜԫԪҗҖзЗӟӞԅԄԑԐѕЅӡӠԇԆиИѝЍӥӤӣӢҋҊіІїЇйЙјЈкКќЌқҚӄӃҡҠҟҞҝҜԟԞԛԚлЛӆӅԯԮԓԒԡԠљЉԉԈԕԔмМӎӍнНԩԨӊӉңҢӈӇԣԢҥҤњЊԋԊоОӧӦөӨ
ӫӪпПԥԤҧҦҁҀрРҏҎԗԖсСԍԌҫҪтТԏԎҭҬћЋуУўЎӱӰӳӲӯӮүҮұҰѹѸфФхХӽӼӿӾҳҲһҺԧԦѡѠѿѾѽѼѻѺцЦҵҴчЧӵӴԭԬҷҶӌӋҹҸҽҼҿҾџЏшШщЩъЪыЫӹӸ
ьЬҍҌѣѢэЭӭӬюЮяЯԙԘѥѤѧѦѫѪѩѨѭѬѯѮѱѰѳѲѵѴѷѶҩҨԝԜӏӀաԱՠբԲգԳդԴեԵևզԶէԷըԸթԹժԺիԻլԼխԽծԾկԿհՀձՁղՂճՃմՄյՅֈնՆշՇոՈչՉպՊջՋռ
ՌսՍվՎտՏրՐցՑւՒփՓքՔօՕֆՖՙאבגדהוװױזחטיׯײכךלמםנןסעפףצץקרשתءٴ۽ٵٶٷٸآأٲٱؤإٳݳݴئاٮبٻپڀݐݑݒݓݔݕݖةتثٹٺټٽٿجڃڄچڿڇحخځ
ڂڅݗݘݮݯݲݼدذڈډڊڋڌڍڎڏڐۮݙݚرزڑڒړڔڕږڗژڙۯݛݫݬݱسشښڛڜۺݜݭݰݽݾصضڝڞۻطظڟعغڠۼݝݞݟفڡڢڣڤڥڦݠݡٯقڧڨكکڪګڬݿڭڮگڰڱڲڳڴݢػؼݣݤلڵڶڷ
ڸݪم۾ݥݦنںڻڼڽڹݧݨݩهھہۂۃۿەۀوۥۄۅۆۇۈۉۊۋۏݸݹىيۦیۍێېۑؽؾؿؠݵݶݷےۓݺݻܐܒܭܓܔܮܖܕܯܗܘܙݍܚܛܜܝܞܟݎܠܡܢܣܤܥܦܧݏܨܩܪܫܬހޙޚށނރޜބޅކއ
ޢޣވޥމފދޛތޘޠޡލގޤޏސޝޞޟޑޒޓޔޕޖޗޱަާިީުޫެޭޮޯްߊߋߌߍߎߏ͸͹΀΁΂΃΋΍΢԰՗՘֋֌֐׈׉׊׋׌׍׎׏׫׬׭׮׵׶׷׸׹׺׻׼׽׾׿܎݋݌޲޳޴޵޶޷޸޹޺޻޼޽޾޿

Based on that, I don’t think that there are any easy-to-type symbols that will be sorted after the letters, but there are one or two unusual characters that work: for example, ǂǁΔʘ

For those interested, the JavaScript snippet is:

var codes = [];
for(var code=32;code<2000;code++) {
	codes.push(String.fromCharCode(code));
}

codes.sort((a, b) => a.localeCompare(b))

for(var pos=codes.length-100;pos>=0;pos = pos-100) {
	codes.splice(pos,0,"\n");
}

var s = codes.join("");

console.log(s);
copy(s);

[Yaisog]

You shouldn’t have stopped at 2,000. All the good symbols (arrows, math, geometrical) start just above 2,000…
See e.g. https://symbl.cc/en/unicode/blocks/geometric-shapes/

[jeremyruston]

@Yaisog

All the good symbols (arrows, math, geometrical) start just above 2,000…

Good point, I just tried up to 20,000 but sadly, all the good symbols seem to sort before the letters.

[pmario]

With Windows 11 you can use the Clipboard History if you type Windows-key v to get a popup like this and select “Symols” tab

Edit: uups. Sorry the symbol I did mention here is a different one that Jeremy posted – which does not sort at the end of All-tab :/. … BUT the concept is OK;
Then select Supplemental symbols. … At pos (1) you can scroll right for fast selection

have fun!
mario
PS: Info about the Clipboard History

PPS: Clearing the history use the “Clear all” button

[Yaisog]

Ever since some point in the Windows 10 lifecycle, you can also press Win+. (period) to get a popup where you can select all sorts of smileys and symbols – once you know which ones are sorted after Z…

[mwiktowy]

Sorting Unicode seems to be not exactly straight-forward.

https://unicode.org/reports/tr10/

[TW_Tones]

I have researched this and my notes follow, I think we need an additional sort operator, however note a suggested work around below.

The sort operators, to my knowledge the only way to sort in TiddlyWiki, they are defined in $:/core/modules/filters/sort.js

• which then calls sortTiddlers = function(titles,sortField,isDescending,isCaseSensitive,isNumeric,isAlphaNumeric)
• Which is defined in $:/core/modules/wiki.js.
• I can’t pretend to understand how this Javascript works but I observe Operators using the following parameters are not defined;
  • isCaseSensitive=false,isNumeric=true,isAlphaNumeric=true
  • isCaseSensitive=true,isNumeric=true,isAlphaNumeric=true
  • however I created and tested these and they are by definition not what we need for this use case.

We need to extend sorting in tiddlywiki
Whilst we may wish to revert to a sort order by character position, even if it were only ASCII we don’t want that, because as a rule normally a sort will map a-z and A-Z next to each other.

• See ASCII Sort order Chart, note there are non-alphanumeric characters we could use before numbers, lower/upper case, after Uppercase, and after lowercase. But we need a sort algorithm that honours these, but also handles the a-z and A-Z part correctly eg;
  • ! before letters and numbers
  • > before a-z and A-Z
  • ~ after a-z and A-Z

The method used may need to take account of Unicode’s multi-byte nature if we are to sort the unicode as well.

• Noting there are also other a-z and A-Z character sets that deserve this extra treatment as well.

I would be confident there are libraries and functions out there that can honour the use of special characters before and after 0-9 and a-z and A-Z character sets, as well as the standard practices.

• This may deserve another and seperate sort operator to be defined because it will be insufficient to make use of the existing sortTiddlers function in $:/core/modules/wiki.js

Workaround;

• When wanting a list that puts items with a selected character eg; _ at the top or bottom eg: `~’ of a list you can fake it with three lists, whose result looks like one.

1. has prefix _ sort and present in first list
2. !has prefix _ or ~ and sort as desired
3. has prefix ~ sort and present in last list

Additional note:

• We can currently “convert a character number to a character” see [charcode Operator](file:///C:/Scratch/tiddlywiki-sort-operators.html#charcode%20Operator) but we can not do the reverse “converts a character to a character number”.
• I expect a “primitive function” to do this may be needed for an improved sort and I would request that one also be made available as an operator.

[Hans_Wobbe]

I use a lot of exotic characters. For me, the use of “Win+.” (period) to trigger the selection of Smilies+…, combined with the use of the Maurycy’s AutoComplete plugin is very useful.

[TW_Tones]

Since you @Hans_Wobbe have reactivated this thread I see was not yet documented a full solution, but I have found one, at least for non-Unicode.

I also mention @jeremyruston as the original poster.

We now have the Sort Filter Run Prefix and specifically its string suffix.

<$list filter="[tag[Home]]  :sort:string[get[title]]">

</$list>

I have tested the above and observe it is effectively using an ASCII sort but also puts lowercase before uppercase.

So the order tiddlers depends on the first letter according to its place in the ASCII table (with useful differences).

The following lists them all and the order that is used;

<$list filter="[range[32],[126]]" variable=ascii>
   <<ascii>> {{{ [charcode<ascii>] }}}<br>
</$list>

Thus tiddlers beginning;

• space + ! " # $ % & ' ( ) * + , - . / all come before numbers
  • Of course $ is used for system tiddlers so you can exclude those beginning $:/ with !is[system]
  • Some may find # and -, asterix * and ( ) useful here above numbers
• the numbers appropriately sorted eg 1 then 10, then 100
• then : ; < = > ? @ follow numbers, but proceed a-z and A-Z
• There are some characters between a-z and A-Z that also end up proceeding a-z and A-Z [ \ ] ^ _ when using the above sort method. ie :sort: string[get[title]]
• Then we get the a-Z and A-Z sorted
• Then finaly after all alphabetics the following symbols will sort below the alphabetics { | } ~

So a simple guide would be to use as the first character (or in a sortfield)

• ! to place something at the top
• . at the top just above numbers
• 0-9 Numbers, honors number schemes
• _ after numbers before alphabetics
• a-z Alphabetics lowercase first then uppercase for each letter
• ~ for after Alphabetics
• Then finaly Unicode values will come after that, not researched in detail by me (yet)

[CodaCoder]

@TW_Tones

a-z Alphabetics lowercase first then uppercase for each letter

Other way around. A:65 .. a:97

[TW_Tones]

@CodaCoder

Other way around. A:65 .. a:97

Sorry not sure what you mean?

• The lowercase is coming before upper case in this string based sort.
• Would you prefer the otherway?
• I have no problem which is first only that they appear near each other.

Here is the result from my test data

[CodaCoder]

@TW_Tones

Sorry not sure what you mean?

You said:

a-z Alphabetics lowercase first … then uppercase for each letter

Your $list enumerates ASCII codes 32 .. 126, therefore, ASCII code 65 (uppercase ‘A’) comes BEFORE ASCII code 97 (lowercase ‘a’). So, it’s uppercase first, then lowercase.

And you said it twice, so I’m not sure what you’re referring to, but your code (and ASCII itself) is pretty clear.

[TW_Tones]

The list widget enumerates the ASCII characters so you can see what they are, I then go on to explain the sort order as a result of using the filter run :sort: string[get[title]] behaves.

@TW_Tones

So the order tiddlers depends on the first letter according to its place in the ASCII table (with useful differences).

• The emphasis is on with useful differences

It is not as simple as the ASCII table because we would not want a lowercase a to come after an uppercase Z for example.

• The key point here is it not only satisfies the OT but also the need for other punctuation characters that appear before after numbers and alphanumetics in the sort, or the number 10 to come before 1 etc…
• Now by choosing the right character as a prefix to a tiddler title, you can ensure where it comes in the order of things.
• The main relationship to the ASCII table is you can see the cluster and order of symbols available in each relative position.

Here is a test sample of tiddlers you can drop on tiddlywiki.com to test different sort methods. character-prefixed-titdlers.json (1.8 KB)

[CodaCoder]

@TW_Tones

I then go on to explain…

No, no, that is not the point. Please, pay attention, Tones

@CodaCoder

Your $list enumerates ASCII codes 32 .. 126, therefore, ASCII code 65 (uppercase ‘A’) comes BEFORE ASCII code 97 (lowercase ‘a’). So, it’s uppercase first, then lowercase.

Your code

<$list filter="[range[32],[126]]" variable=ascii>
   <<ascii>> {{{ [charcode<ascii>] }}}<br>
</$list>

Try it on TW .com ← Uppercase FIRST.

[TW_Tones]

@CodaCoder see my screen shot above, the result has modes before Modes.

As I said it is not important the order of a-z and numbers, in the ASCII table when you use the recommended sort method the result is what I have stated.

• it becomes aA - zZ and 1-9 before 10-19 as is expected normally.
• It is clearly handled internally perhaps by the javascript function that is being used.

Yes the ASCI List does return
! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~

• Note I removed backtick, and would not recommend | [ ] { } as tiddlywiki tells you.

[TW_Tones]

@CodaCoder have you not seen what I am saying?

I did not deserve the “Please, pay attention, Tones” perhaps that was for you?

[TW_Tones]

For any future testing here is some test data;

sample-character-prefixed-tiddlers.json (17.6 KB)

Drop on empty and use to list

<$list filter="[tag[Prefixed tiddler set]]  :sort:string[get[title]]">

</$list>

I have now found numbers failing to sort correctly. 1 10 100 come together. as do 2 20 and 200

• Odd I missed this

[Scott_Sauyet]

@TW_Tones

CodaCoder see my screen shot above, the result has modes before Modes.

It has “modes” before “Modes uppercase”. But that’s different. It’s similar to how “AB” sorts before “ABC”. If you try “Modes” and “modes”, you’ll see that upper-case sorts first.

As to numbers, that’s a much harder sorting problem. Normal sorting is character-by-character lexicographic. To handle numbers properly, you’d have to chunk the digits together and convert them to numbers. It’s an interesting problem that’s certainly been solved, but I’ve never looked for (or tried to write) a solution.

Hmmm…

[Scott_Sauyet]

@Scott_Sauyet

To handle numbers properly, you’d have to chunk the digits together and convert them to numbers. It’s an interesting problem that’s certainly been solved, but I’ve never looked for (or tried to write) a solution.

Never mind, it’s been solved, and is built into the language:

Intl.Collator - JavaScript | MDN

This is probably too new to be allowed in the TW core, but could probably be a plugin.

[Scott_Sauyet]

@Scott_Sauyet

This is probably too new to be allowed in the TW core, but could probably be a plugin.

Again, never mind. This is built in:

https://tiddlywiki.com/#sortan%20Operator

<$let my-list="ABC-12 ABC-3 ABC-20 ABC-3.141592 ABC-2.71828 ABC-3-XYZ">
''Unsorted'': <$text text={{{ [enlist<my-list>join[, ]] }}} /><br/>
''sort'': <$text text={{{ [enlist<my-list>sort[]join[, ]] }}} /><br/>
''sortan'': <$text text={{{ [enlist<my-list>sortan[]join[, ]] }}} /><br/>
</$let>

unsorted	ABC-12	ABC-3	ABC-20	ABC-3.141592	ABC-2.71828	ABC-3-XYZ
sort	ABC-12	ABC-2.71828	ABC-20	ABC-3	ABC-3-XYZ	ABC-3.141592
sortan	ABC-2.71828	ABC-3	ABC-3-XYZ	ABC-3.141592	ABC-12	ABC-20