Some technical details that may help us to extend the possibilities for wikitext based editors

#1

@Mohammad,

This info is related to: Section Editor Plugin: First Stable Release - #37 by Mark_S

As you started the section-editor discussion I was thinking about an open pull-request at GitHub: Add more start/end parser ranges by SmilyOrg · Pull Request #4977 · Jermolene/TiddlyWiki5 · GitHub

It would extend the TiddlyWiki internal parse-tree with some “start / stop” information that would make your “sectioning / replace” mechanism much easier if some of this info would be exposed to the eg: search and search-replace operators.

Since the tiddler parse-tree is cached, accessing this info should be very fast and it would make sure, that the wikitext filters use the same parsing rules as the core. eg: !.class heading text and so on.

The parse-tree for

! heading 1

some text

! heading 2

looks like this:

[
    {
        "type": "element",
        "tag": "h1",
        "attributes": {
            "class": {
                "type": "string",
                "value": ""
            }
        },
        "children": [
            {
                "type": "text",
                "text": "heading 1"
            }
        ]
    },
    {
        "type": "element",
        "tag": "p",
        "children": [
            {
                "type": "text",
                "text": "some text"
            }
        ]
    },
    {
        "type": "element",
        "tag": "h1",
        "attributes": {
            "class": {
                "type": "string",
                "value": ""
            }
        },
        "children": [
            {
                "type": "text",
                "text": "heading 2"
            }
        ]
    }
]

The parse-tree with the PR active would look like this:

[
    {
        "type": "element",
        "tag": "h1",
        "attributes": {
            "class": {
                "type": "string",
                "value": ""
            }
        },
        "children": [
            {
                "type": "text",
                "text": "heading 1",
                "start": 2,
                "end": 11
            }
        ]
    },
    {
        "type": "element",
        "tag": "p",
        "children": [
            {
                "type": "text",
                "text": "some text",
                "start": 13,
                "end": 22
            }
        ],
        "start": 13,
        "end": 22
    },
    {
        "type": "element",
        "tag": "h1",
        "attributes": {
            "class": {
                "type": "string",
                "value": ""
            }
        },
        "children": [
            {
                "type": "text",
                "text": "heading 2",
                "start": 26,
                "end": 35
            }
        ]
    }
]

The character count starts with 0 and line-feeds \n are also counted. … As you can see, if the numbering would be optimized it would be simple to find the 2 heading level-1’s or any other element that we want to find. …

just to let you know
-mario

2 Likes
Section Editor Plugin: First Stable Release
#2

Many thanks @pmario - I will also follow up the GitHub!

#3

Adding start/end ranges is the direction I went in as well when I was still considering taking the streams as a list editor concept further (replace every list with a streams like UI).

Somewhere in one of my local branches I’ve added ranges for a lot of the wikiparser rules that are missing this information, including lists which was tricky to do with the way it is written. It is on the pile of things to find the time to clean up and organize, as I recall it needs some core changes as well some of which are in the PR linked in the top post.

Note that adding parse tree ranges is also the first step towards integrating third party visual/block based editors.

#4

I would add this could lead to being able to auto render qualified html id attributes to such numbered elements alowing people to link to them within the tiddler or apply styles progressivly. The anchor and link is long sought after as is easy slider/expansion.

#5

I think a streams-like editor should not necessarily limited to lists. I can think about a possibility to change the bullet-point to different icons.

  • ! … HEADING
  • * … UL - LI
  • # … OL - LI
  • > … BLOCKQUOTE
  • t … plain text which is converted to paragraphs
  • p … single PARAGRAPH
  • … and so on

Where a right-click can change the wikitext-type that is stored in 1 node. My idea is limited to “block parser elements” at the moment.

In my idea with this mechanism it would be possible to create a “Word-like” document outliner.

Since the parsing info is provided by the core it shouldn’t matter too much if the starting / endpoint is a single tiddler or multiple tiddlers.

#6

An other idea was, that it should be simple to create a list of all eg: H1 headings to see if they are consistent eg:

  • Some headings may end with a full-stop, some headings don’t.
  • Some headings use Title Case … some do not.

and so on. …

1 Like
#7

I only mentioned streams and lists as the context in which I worked on parser start/end ranges. It is not otherwise relevant to this discussion.

What you describe is a block based editor which is a very different and distinct concept which I have intentionally steered clear of… but anyway this thread was about improving the parser as far as I understood it, so on that note:

Yes adding the missing start/end ranges is necessary for a block based editor, hence my reference to the possibility of integrating third party block based editors (rather than re-inventing the wheel). This is something that SmileyOrg has worked on, which led to the PR in question. I believe they even have a very early prototype. I am not sure if they are on this forum.

#8

@pmario brief comment. I think the focus on parsing detail is very useful. TT

1 Like
#9

see my The quickest and easiest ever todolist in tiddlywiki ? Requires 5.2.0 - #2 by TW_Tones for a simple way to handle line prefixes, somewhat relevant/inspired by here.

#10

Referring to Hide Tiddler Body

and

Hi @jeremyruston
Referring to above discussions and some PRs on GitHub, I highly appreciate to have a look and let us know your opinion!

Having a simple solution without touching core tiddlers is very important as TW 5.1.23+ specially 5.2.0 allows to write simple wikitext and parse the tiddler content (text field = tiddler body) and display in custom format!

See recent tools like Streams, Section Editor, Shiraz Quick table, Interactive todos show the need for simple procedure to render the tiddler body in custom ways!