TiddlyWiki syntax/grammar for the CodeMirror 6 plugin

Developers
, ,
#1

Hi TiddlyWikians,

the CodeMirror 6 plugin I’m developing is in a pretty nice state and I’m looking into the inner workings of language modes for CodeMirror 6

CodeMirror 6 uses a Parser system called Lezer

I have already created an extremely limited tiddlywiki-language mode (just the integration) and that mode is used for tiddlers with the type text/vnd.tiddlywiki

That just means I know the very basic starting steps

I hope for some support by people in this community in regards of creating lang-tiddlywiki even if it’s just emotional support :sweat_smile::sleepy:

For people who are interested I’d lay open the steps how I clone the codemirror repository from GitHub, install the dependencies, create the tiddlywiki-lang folder with the basic functionality and finally, how I build codemirror.js from everything

This topic is also meant for discussion and ideas, so feel free to comment :slight_smile:

With best wishes from Southtyrol,
Simon

1 Like
#2

Hi @BurningTreeC

I had a look at the docs for Lezer and the example JavaScript grammar. It seems to be a familiar type of grammar notation.

However, the Markdown parser for Lezer turns out not to use the Lezer declarative language, but is instead a custom TypeScript parser that integrates with Lezer:

This is an incremental Markdown (CommonMark with support for extension) parser that integrates well with the Lezer parser system. It does not in fact use the Lezer runtime (that runs LR parsers, and Markdown can’t really be parsed that way), but it produces Lezer-style compact syntax trees and consumes fragments of such trees for its incremental parsing.

That’s very encouraging! It suggests that we can stay away from Lezer itself, and instead use the built-in TiddlyWiki parser, and just worry about converting the parse tree to the Lezer-compatible format that CodeMirror requires.

#3

Hi @jeremyruston

That sounds really encouraging! :+1:

I’ll have to think about the consequences it brings to the game

#4

Hello @jeremyruston

Do you have an idea how to start to use the built-in TiddlyWiki parser?

Thank you,
Simon

#5

Look at this

And how I use nodes https://github.com/tiddly-gittly/wikiast/blob/master/packages/wikiast-util-to-wikitext/src/astSerializer/image.ts

#6

You can try transform wikiast to lezer ast like I do, and you can try PR to wikiast package to put them together.

#7

Thank you @linonetwo

you seem to have more knowledge than me in this direction

I don’t know how to integrate this in the current codemirror engine.js

#8

I think you just need to find out where is accepting lezer’s parsing result, and you write a wikiast-util-to-lezer-ast, to transform wikiast to lezer ast (so you don’t need to use lezer any more!). Just short-circuit the lezer parser.

Just like how I make slate wysiwyg editor work.

#9

Yes @linonetwo ,

I’m trying to figure out WHERE to short-circuit the lezer parser
And then I can worry about transforming wikiast to lezer style

1 Like
#10

I just want to clarify that if type is blank text/vnd.tiddlywiki is the default. So a tiddler is text/vnd.tiddlywiki unless otherwise set.

#11

Hi @TW_Tones ,

yes, that is already handled :slight_smile:

#12

Hello @jeremyruston

Am I right with the assumption that the tiddlywiki parser needs to parse the whole document at once?

So it cannot use streams of text, correct?

I’m currently tinkering around, trying to do in Javascript what the lang-markdown package does.
I could create something that crashed when it called startParse() but I’ve seen in the docs that startParse, createParse and parse must be implemented and these are probably the functions where we can use the tiddlywiki parser

#13

I’m now at the point where I can transform tiddlywiki’s parse tree to a lezer tree

Does anybody have an idea how to do that?

#14

That’s correct, yes.

It might be helpful to look at some example Lezer trees, if you’ve found any?

#15

Yes @jeremyruston

I’ve already looked at lezer trees
They are not too complex
What I’d have to do is recursively cycle through the tiddlywiki parse tree and create a lezer tree from it
I have a function that cycles through the parse tree but I need a way to create that new object from it containing a lezer tree for each child and subchild and so on…

#16

Hi @BurningTreeC – can you link to some examples?

#17

Hi @jeremyruston

Currently I can only post a link to Lezer Reference Manual

There you can see that the parse tree must be converted to a lezer Tree containing an array of children which are also lezer Trees (and their children must be lezer Trees and so on)

So what I’m trying to do is cycling trough the parse tree and converting each node to a lezer tree. Where I’m struggling is putting it all together

I cannot post an example what I’m doing because I’m currently away from my desk AND I really doubt that I’m doing this correctly

Where I’m sure is that I’m in the correct function (createParse) Lezer Reference Manual returning the resulting lezer tree in advance()

#18

Just copy some code from GitHub - tiddly-gittly/wikiast: Wiki AST transformation for WYSIWYG editor and advanced dynamic widgets in Tiddlywiki. , it is similar to GitHub - syntax-tree/mdast-util-to-nlcst: utility to transform mdast to nlcst , a very standard way for ast transformation, easy to write, because you only transform a single node in a file.

And easy to test, you can see I’ve written many tests, you can’t write this without unit test, otherwise you will have many regressions.

#19

Hi @linonetwo - could you point me to the specific code you intend?

#20

Just read all code in this folder https://github.com/tiddly-gittly/wikiast/blob/master/packages/wikiast-util-to-slate-plate-ast/src/index.ts there is no much. And I have some note about them

https://wiki.onetwo.ren/AST转换成另一种AST