There's still no TiddlyWiki5 highlighting for the CodeMirror 6 plugin

,
#1

Hello TiddlyFolks,

While building the CodeMirror 6 plugin I read a lot about Syntax Highlighting and that custom Highlighting can be achieved for unsupported languages.

Now, TiddlyWiki5 language is such an unsupported language.

The challenge is, that TiddlyWiki5 language will need a mixed parser like the CodeMirror 6 Markdown parser (that’s my personal verdict).

A mixed parser parses uses the Lezer parser and other parsers and combines them.

TiddlyWiki5 uses:

  • HTML syntax
  • TiddlyWiki5 markdown-like syntax
  • Widget syntax
  • Filter syntax
  • CSS syntax
  • Codeblocks within WikiText
  • Macro syntax
  • …

We would need to understand, how we can make the future CodeMirror 6 TiddlyWiki5 parser split a TiddlyWiki5 syntax text into semantic tokens that can then be used for Syntax Highlighting (and autocompletion, too).

I’m writing these lines because I hope that with the power of more brains we can achieve something.
I have the knowledge how to integrate the parser, how to generate a parser from a lezer.grammar file and “put it into the library”.

What I don’t have is the correct approach, that’s why I ask if interested Forum members could leave their comments, maybe we find a suitable approach :slight_smile:

That’s all for now,
best wishes from sunny Southtyrol,
Simon

2 Likes
#2

I don’t have any real experience with this, but I’m pretty happy with @joshuafontany’s vscode syntax highlighter. I don’t know if the models are similar enough that you might gain some insight from it.

The mixed content certainly makes this a challenging problem!

1 Like
#3

The mechanism used for VSCode is completely different to CM6. You might get some insight, how it could look like. But the code can not be used.

#4

I’m not surprised. Thanks for the information.

#5

@BurningTreeC aka Simon, Like others here we are very keen to see a highlighter eventuate. I just though I would point out incase you did not think of (unlikely) this approach.

Tiddlywikis own parser is driven by wikitext rules under $:/core/modules/parsers/wikiparser/rules/ I just created a new one myself by cloning one of these. In many ways they document tiddlywikis syntax as html/javascript elements.

  • Perhaps using this will ease the development of a syntax highlighter including the nessasary regular expressions.

On a related issue it would be great if oneday we could contribute a parser to the great pandoc.org and allow tiddlers to be converted to and from other formats starting with wikitext, and where nessasary quote widgets that will not work in other formats.