Is TiddlyWiki written in itself?

lytex · March 28, 2023, 10:43pm

Joe Armstrong was investigating this concept of building a Quine in his tiddlywiki, but I don’t know if this means interoperability with another hypothetical backend:
https://joearms.github.io/#Index

If tiddlywiki was a programming language, then this would be a different question

I don’t think tiddlywiki has to be a full programming language, but it may be close. It “only” would have to be able to build tiddlywiki itself based on some building blocks. The code could be provided by the blocks themselves (be it javascript, erlang, or emacs lisp for example) and then have an executor that would actually create/update/read/delete tiddlywiki components

Charlie_Veniot · March 28, 2023, 10:55pm

Depends on what you mean by “written in itself.”

TiddlyWiki is not at all like a “Bootstrapping” compiler.

As a “quine”, you can alter it and have it output its new (or even unchanged) “self”, but that is “self-replicating”.

The very core of TiddlyWiki, you may override with your own pieces, but you aren’t changing the core. You’re just setting up a “please use my version instead of the core version”. But the core version remains. If that were done for every version of TiddlyWiki, every new version would still be carrying all previous versions of every overridden core tiddler and we would have ever-ballooning-in-size versions of TiddlyWiki.

The only way to change the core in a TiddlyWiki instance is by opening up that TiddlyWiki instance with some editor (even Notepad), to alter the core. Then you have a new TiddlyWiki instance with a brand new core, but TiddlyWiki didn’t create a new version of itself. It can’t.

New TiddlyWiki bits and pieces can be created, core TiddlyWiki bits can be overriden via tiddlers, but it can’t output a version of TiddlyWiki with a brand new core (i.e the javascript stuff that allows everything to happen.).

pmario · March 29, 2023, 7:12am

TiddlyWiki is a Quine. … When you save it, it effectively does build itself. That makes it possible to have different download buttons at tiddlywiki.com

There is a big green Download Empty button, which uses a configuration which builds an empty version, using parts of tiddlywiki.com

If you click the “Check button” in the right sidebar it will use the actual content from the page and it will save / download itself – plus – the new content

So yes. … TiddlyWiki is “written in itself”

lytex · March 29, 2023, 9:29am

I think the meaning of " TiddlyWiki is written in itself” is not clear, and what I’m aiming at is easiness of reimplementation independent of the browser.

Taking a look at the developer documentation:
https://tiddlywiki.com/dev/#TiddlyWiki%20Core%20Application

If one were to reimplement the inner parts (say the bootprefix and the microkernel) so that the data model is the tiddler data model, all public functions that the plugins use are implemented, and the functions drawing and editing the user interface are written in some other language and provide an alternative interface (say emacs lisp, and the interface is drawn in emacs itself) then everything would work?

One would need at least:

Some way of running emacs-lisp in the browser (although not as an interface, as only the plugins and the public function interface/wrapper would be running in javascript) and make calls to emacs to edit the actual interface
Working with javascript objects and modifying them to keep the illusion that there is a browser, possibly having to deal with recursive objects

Edit: Now that I think of it, the plugins probably assume there is a DOM, so you would have to replicate the entire DOM and at this point you are basically implementing a browser in emacs

pmario · March 29, 2023, 11:39am

Theoretically yes … The problem with “in theory” is, that it always works “in theory”, but in praxis it does not. … As so often the devil is in the details. Having one sentence of specification is a bit “sparse”

But

The TW data model is simple. It basically is a JSON structure, which is wrapped in an HTML <script> tag, that is embedded into an HTML file, which contains the TW core-code and core-UI.

If you

open https://tiddlywiki.com/empty.html
create a tiddler
- named eg: my-new-tiddler
- tagged: tag-1
- field named: field-1 value: value-1
- text
  
  some text
  field-1: {{!!field-1}}"
save / download the file empty.html
open it in your favourite text editor eg: VSCode

you will see several HTML comment markers

<!--~~ Raw markup ...
<!--~~ Static styles ...
<!--~~ Static content ...
<!--~~ Ordinary tiddlers ... ← Explained in more details later [A]
<!--~~ Library modules ... ← position [B]
<!--~~ Boot kernel prologue ...
<!--~~ Boot kernel ...
<!--~~ Raw markup for the bottom ...

If you search for and jump to: <!--~~ Library modules … you’ll see a structure like this:

{"created":"20230329111008703","text":"some text\n\nfield-1: {{!!field-1}}","tags":"tag-1","title":"my-new-tiddler","modified":"20230329111824745","field-1":"value-1"}
]</script><div id="storeArea" style="display:none;"></div>
<!--~~ Library modules ~~-->

The big blob of text above the my-new-tiddler is the TW core-plugin, which also is a tiddler.

The pretty-printed structure of the wiki store looks like this:

<script class="tiddlywiki-tiddler-store" type="application/json">[
[
	{
		"title": "my-new-tiddler",
		"created": "20230329111008703",
		"modified": "20230329111824745",
		"tags": "tag-1",
		"field-1": "value-1"
		"text": "some text\n\nfield-1: {{!!field-1}}",
	},
	{
		"title": "minimal-viable-tiddler",
	}
]</script>

So … If your UI can search for <script class="tiddlywiki-tiddler-store" type="application/json"> and parse a JSON array, that contains tiddler objects you are able to read every tiddler-store from every tiddlywiki.html file.

Important: It’s allowed to have several stores wrapped in <script class="tiddlywiki-tiddler-store" type="application/json"> tags.

If you are able to create a JSON “tiddler store” and inject it into empty.html using the described markers, you are able to create a valid file-TiddlyWiki, which can be opened and saved in the browser.

See: How do TWs JSON Formats Look Like

The template $:/core/templates/tiddlywiki5.html responsible to create a file-tiddliwiki is at GitHub
and html-template at tiddlywiki-com

have fun!
mario
also see: TiddlyWikiPharo and critical code/data literacy curriculum - #8 by pmario

my-new-tiddler.json (169 Bytes)

pmario · March 29, 2023, 11:47am

I forgort to mention … the <div id="storeArea" style="display:none;"></div> is the tiddler store for older TWs prior to v5.2.0

{"created":"20230329111008703","text":"some text\n\nfield-1: {{!!field-1}}","tags":"tag-1","title":"my-new-tiddler","modified":"20230329111824745","field-1":"value-1"}
]</script><div id="storeArea" style="display:none;"></div>
<!--~~ Library modules ~~-->

This format is still valid, but much more complex to maintain with an external editor.

<div id="storeArea" style="display:none;">
<div created="20230329111008703" field-1="value-1" modified="20230329111824745" tags="tag-1" title="my-new-tiddler">
<pre>some text

field-1: {{!!field-1}}</pre>
</div>
</div>

pmario · March 29, 2023, 11:59am

Now the more complex stuff. …

… If you want to render tiddlers in a “native way” with macros, variables and especially transclusions you either have to re-use the tw-javascript-rendering-engine or you’ll need to create a TW wikitext parser.

The existing regexp-based parser is in the works since about 2013 and the development is still ongoing. … So “here will be dragons …” and here too

Scott_Sauyet · March 29, 2023, 1:40pm

Remember that wikis based on this core, and the core itself, all assume the existence of the Web layer underneath. That means that you would have to recreate all of HTML/JS/CSS, including such subtleties as the difference between buttons and links, the complexities of CSS selectors. It’s a giant job. The latest JS spec alone is well over 250,000 words long. The HTML spec is over half a million. Just the list of CSS Recommendations, Candidate Recommendations, Proposed Recommendations, Working Drafts and so on is huge, never mind the actual content.

So figuring out what precisely you would want to support from the underlying platform is a large job. Implementing it is comparable to the process of creating a new web browser from scratch.

So best of luck. If you’re looking for help with this, sure, I’ll be available eventually, but just remember that I’m out of office for the next few… centuries.

lytex · March 29, 2023, 2:07pm

Yes, the idea is not to implement a spec-compliant web engine, “only” the wikitext parser. The thing is, as you say, that core (and probably plugins as well) assume there is a HTML/DOM object underneath. To what extent (and where in the code, and how much) do they assume it exists?

However if this is true:

excluding plugins, the UI and its internals should be increasingly easier to implement in another environment without HTML, JS, CSS

There is also this answer to help identify the different parts of the so-called “wikitext”:

What We Can Call Tiddlywiki WikiText Scripting

To me, WikiText (TwText) is TiddlyWiki’s markdown syntax. It is just the stuff for formatting text, the substitute for HTML and CSS (the stuff for look and feel).

Widgets, that’s scripting. TiddlyWiki Script, or TW Script.

The curly bracket syntax for transclusion, whether double or triple curly brackets, that’s also TW Script, but in a convenient short hand.

Same for the square bracket syntax for links. Short hand notation for TW Script.

Although, for example, input elements are HTML elements, I consider them the visual elements of the scripting language because they “do” things

All of the filter operators, I see them as TiddlyWiki Query Language, or TW Query Language, or TwQL.

Oh yeah, and then there are “metaprogramming” things: pragmas, for example. (TwMetaProgramming)

Scott_Sauyet · March 29, 2023, 2:36pm

But that is layered atop the Web space. wikitext is not an entirely new language. It’s an extension of HTML.

Everywhere. The core button $:/core/ui/Buttons/new-tiddler includes an HTML <span>

<span class="tc-btn-text">
<$text text={{$:/language/Buttons/NewTiddler/Caption}}/>
</span>

and elsewhere has CSS defining display information for tc-btn-text.

It’s all wrapped in a ButtonWidget which includes lines like

	domNode = this.document.createElement(tag);
	this.domNode = domNode;
	// Assign classes
	var classes = this["class"].split(" ") || [],
		isPoppedUp = (this.popup || this.popupTitle) && this.isPoppedUp();
	if(this.selectedClass) {
		if((this.set || this.setTitle) && this.setTo && this.isSelected()) {
			$tw.utils.pushTop(classes, this.selectedClass.split(" "));
			domNode.setAttribute("aria-checked", "true");
		}
		if(isPoppedUp) {
			$tw.utils.pushTop(classes,this.selectedClass.split(" "));
		}
	}

This means you’d have to implement the interface for Document, including its createElement method, which would have to return an element, typed according to the tag supplied to it. Somewhere in that returned value’s hierarchy, you would have to have an appropriate setAttribute method. You would need to duplicate the JS this behavior, it’s Array prototype, including the split method, it’s behavior for || and &&, and so on.

Or not. You could say that you just need to supply the same interface as the ButtonWidget from some other stack, but you’ll need to remember that that interface right now is to generate DOM elements with certain styles and behaviors attached. If you did this for every JS Widget, macro, etc, my guess is that you would still have to create a huge swath of the web stack. And every upgrade might require more, because the core will certainly not restrict itself to some arbitrary subset of the DOM that you happen to have implemented.

But just the fact that wikitext is an extension of HTML and not an entirely separate language should be enough to point out the difficulties.

CodaCoder · March 29, 2023, 2:41pm

I’m not convinced by that conclusion

Why? For what purpose?

lytex · March 29, 2023, 3:22pm

Sorry, then I’m confused about what wikitext is and what html is.

In this example->Generating dynamic links, the wikitext I assume it is:

[[link to myself|Linking in WikiText]]

and then the underlying html would be:

<$link to="Linking in WikiText">
  <$text text="link to myself"/>
</$link>

Are both valid wikitext?

Charlie_Veniot · March 29, 2023, 3:26pm

The following:

[[link to myself|Linking in WikiText]]

is indeed shorthand/equivalent to :

<$link to="Linking in WikiText">
  <$text text="link to myself"/>
</$link>

All of the above is referred to as “WikiText”. I prefer say the first example is TwText and the second is TwScript.

Regardless, each thing above is TiddlyWiki syntax that gets eventually rendered as HTML “anchor” elements:

<a ...

pmario · March 29, 2023, 7:23pm

TW parsers convert the wikitext into an AST, wich is called the parse-tree

The parse-tree is used to create widgets which then create the widget-tree, which is already very close to HTML

The rendering-engine converts the widget-tree into HTML output

You can have a closer look at your second example on yourself. … It’s the same pipelien. parser → parse tree → widget-tree → html code.

If you use my example, you will see a bigger difference between parse-tree and syntax-tree. … If there are transclusions, the widget-tree “resolves” them recursively. The parse-tree only parses the content of the current tiddler.

some text
field-1: {{!!field-1}}

jeremyruston · March 30, 2023, 8:14pm

Thanks @lytex for an intriguing question – and welcome to the community.

Thank you! It’s nice to remember Joe. He was an inspiration to me, and I was honoured by his enthusiasm for TiddlyWiki. His untimely passing was a great tragedy that I still feel keenly.

The most important philosophy of TiddlyWiki is that the purpose of recording information is to re-use it, and TW proposes that the best way to optimise information for reuse is to cut it up into the smallest semantic units. One can then use links, lists and tags to weave the fragments together into multiple overlapping narratives.

The observation about TiddlyWiki being written in itself relates to some secondary principles:

that the individual needs of users are all different, and so it is not respectful to try to impose the same user interface on every user
that users are not generally in a position to know up front what their ideal user interface looks like
that the most efficient way for users to figure out precisely what they need is by iteratively modifying the user interface themselves

We address these needs by making wikitext powerful enough to be able to implement TiddlyWiki’s entire user interface.

To address your specific questions:

That’s right. As others have pointed out, the target platform of TiddlyWiki is the browser, and so even if TiddlyWiki were to be rewritten in a different language it would still need a web browser (or an emulation of one)

The only available implementation of the TiddlyWiki primitives is currently JavaScript, targeting HTML.

I am not familiar with the internals of Jupyter Notebook, but given the fact that it manifests itself as an interactive web page it must include JavaScript in its implementation. If that’s true, then the simplest path to integration might be to integrate the existing JS implemention of TiddlyWiki.

lytex · April 6, 2023, 8:14am

Thank you all for your patience, understanding and support, especially @pmario for helping me visualize the process and digging into the internals of TW

After studying TW and Jupyter architectures I’ve come up with some objectives/principles:

Two distinct integrations
There are two types of integrations: TW into Jupyter, which means writing a Jupyter extension, and Jupyter into TW, which means writing a TW plugin
Have the TW editor bar on each Jupyter cell
This is the main result from the Jupyter point of view (I call this “Integrating TW into Jupyter”). In the future it may be interesting to execute code from TW by making calls to IPython kernels, but I want to:
Focus first on markdown cells
Jupyter has also code cells for executing code. Being able to call IPython from TW is what I would call “Integrating Jupyter/IPyhton into TW” but it seems better to first get comfortable with the inner workings of Jupyter
Run TW in server mode
Running TW In this way means all the .tid tiddler files are available, which allows the following correspondence:
1 cell, 1 tiddler
I think a good approach is considering a notebook a particular arrangement of tiddlers and each cell an individual tiddler. I’m considering plugins such as the Rider plugin to be able to edit a transcluded tiddler/cell.
The filesystem and in particular the .tid files can be the shared interface between TW and Jupyter to keep the underlying files synchronized

pmario · April 6, 2023, 12:22pm

It may be a view month in the future, when the transclusion mechanism can do this out of the box, once the parametrized transclusions PR gets merged. …

pmario · April 6, 2023, 12:32pm

Just to be sure. TW needs a plugin to deal with Markdown syntax.

The TW markdown plugin is a bit special too. If it sees syntax, that it does not know it hands the text over to the TW parser. … That makes it possible to use TW wikitext in combination with markdown. … Especially, it allows us to use the TW KaTex plugin to render math formulas.

… But I thin Jupyter has its own math plugin or something similar.

lytex · April 6, 2023, 1:34pm

Sorry, I do not see how these may be related… Can you somehow add a parameter so that a transclusion is editable?

Maybe using this syntax from this post?
{{HelloThere||EditLink}

Thanks, I already knew about the plugin but I didn’t knew it handles syntax when TW doesn’t know what to do

pmario · April 6, 2023, 1:54pm

The parser is identified with the type-field. … The default type for tiddlers is text/vnd.tiddlywiki even if no type is set …

Markdown tiddlers have the type text/Markdown or text/x-markdown. So TW knows what to do

The TW type-field is the official mime-type