Accessing ts-raw without the slot widget?

[TW_Tones]

I am looking at using a custom widget to provide a three line preview of a block of code [TW5] Looking for a way to fold code inside a tiddler - #3 by TW_Tones.

I want to use a custom widget and use the body / ts-raw to contain the “code content” then in my custom widget extract the first three lines for the preview.

The problem is the only way to access the body of the calling widdget inside a custom widget is using <$slot $name="ts-raw"/> but the problem with this is it is a widget. In some ways a widget is an output mechanism not an input mechanism, that is it is difficult to use the output of a widget as an input to something else, the exception being using wikify to set a variable.

Would it not be possible to access ts-raw as a variable, inside custom widgets?

• This would keep the content of the calling widget in a form further manipulation can be actioned on.
• I do not know how we would handle the depth attribute, perhaps at least ts-raw could be a macro or procedure with a depth attribute, (however one that does not need wikification).

The same would be true for any fill widget as well.

[jeremyruston]

Hi @TW_Tones

@TW_Tones

Would it not be possible to access ts-raw as a variable, inside custom widgets?

It would be very useful if that were possible. The problem is that transclusions are processed during rendering, by which point the plain wiki text has been parsed to the internal tree representation we call the parse tree, and the original wiki text is no longer available.

Ultimately I would like to address these cases by offering a way to create new parse rules, but in the meantime I have an idea for a relatively simple hack that might satisfy this and some related use cases.

The idea would be to reuse our block quote formatting. The opening triple backticks can optionally be followed by a token that identifies the language for formatting the text (eg JS, CSS, HTML etc):

```js
console.log("This will be formatted as","JavaScript");

Now, we could allow custom procedures to define the behaviour of custom “languages”. That procedure would then be invoked to render the text block, with the entire text block being passed along without any parsing.

It might look something like this:

\procedure tv-codeblock-language-mycustomlanguage(text)
<!-- Process the text in some way --->
<$text text={{{ [<text>encodeuricomponent[]] }}}/>
\end tv-codeblock-language-mycustomlanguage

```mycustomlanguage
This is some text that will be processed by my custom language procedure
```

(I can’t figure out a way to make Discourse allow me to include the closing triple backticks).
(mario here: MD formats evey text as code block, if it starts with 4 leading spaces or a leading TAB char)

[TW_Tones]

Thanks @jeremyruston

I will have a look at it right away, but I am not sure I understand yet. I was hoping to take advantage of the same content, that in ts-raw in two or three different ways inside a custom widget.

• I believe I can do that with the approach you suggest referring to the text multiple times.

[TW_Tones]

I have looked and this and I am not sure how it pertains to the original request.

• I believe you have more work to do?

I would have thought if the content of the calling widget, is available to the custom widget, that it would have been trivial to also access it within the widget. Since unlike parameters the only way is via the slot widget we are faced with the limitations of making use of slot and its widget output.

• If I understand this correctly what I thought was a “trivial requirement”, it is not trivial.

Not unlike your example, I then revisited using procedures, and I have a nice one where the parameters can refer to a variable (eg block of code in a procedure/pragma) or tiddler title (eg a tiddler containing code).

• These would be present in the pragma above or in a diffierent tiddler.

This made me realise that what could be missing that I am looking for is the ability to encode a block of content, where desired inline in a tiddler, then address it and retrieve, its content elsewhere such as in a parameter, procedure or custom widget.

• Of course this can be done using let set vars and the way we set a parameter;

<$let varname="""
code content
""">

</$let>

• However this has a few disadvantages such as an unwanted leading / training line breaks
• A somewhat cumbersome layout that is harder tro scan with the eyes in text.

Conclusion

If there were a way to simply name a block of text such it that becomes available as a variable, we can address/access (even if only in the in the current tiddler) using multiline content as parameters would be easy.

[CodaCoder]

@TW_Tones

If there were a way to simply name a block of text such it that becomes available as a variable, we can address/access (even if only in the in the current tiddler) using multiline content as parameters would be easy.

Taking a step back, I asked myself, isn’t that a tiddler (text field) ?

Then this stood out: “even if only in the in the current tiddler” – and I was reminded of the virtual-tiddler discussions we (et al) were having.

So the algo would be, build a virtual tiddler, stuff it with the text, reference it elsewhere (taking care not to disappear up your own fundament along the way)

[TW_Tones]

Thanks for the reminder @CodaCoder.

I have being playing with options that may work for this but may also influence virtual tiddlers. You may find it interesting, here is an outline.

Consider a section marked anywhere inside any tiddler <section>content</section>. Optionally use CSS to hide it. Possibly such sections can be inside a custom widget or anywhere.

Now transclude a special template tiddler wherever you need this content.

{{ ||template|section|format}}

• Access current tiddler or a named tiddler.

The template will retrieve the text field then, the content within the named section.

• Transclusions can be used as attribute values
• Transclusions can be used to set a variable
• we can wrap a transclusion in some formatting.
• yes, this transclusion of the template can access content in the tiddler it is in.

There is no question, there are “devils in the details”.

• But perhaps this approach can replace the complexities of ts-raw.

[TW_Tones]

I found a direct answer to the title of this topic myself, making use of $parseTreeNodes $slotFillParseTreeNodes on the parameters widget. Although it will be fiddly until I design some good functions/filters, to extract these values.

Any help would be appreciated.
I need the assistance of someone who knows JSON well and writing filters to extract content. Especially when fill widgets are used perhaps @pmario

• The result is some kind of JSON array from which we can extract the content.
• This can then be assigned to a variable and used as content without the use of the ts-raw slot. As a result, a lot more continued processing can be done with the body of the calling widget, including custom parsing.
• If we want to write a custom widget that responds with or without additional $fill widgets defining slots, it gets more complex. But it can be done because the $fill widgets get nested inside the body results, and splits the body outside the $fill widgets.

I believe this demonstrates an avenue to generate a variable containing the calling widgets content. The objective of this topic.

@jeremyruston

@TW_Tones

Would it not be possible to access ts-raw as a variable, inside custom widgets?

It would be very useful if that were possible. The problem is that transclusions are processed during rendering, by which point the plain wiki text has been parsed to the internal tree representation we call the parse tree, and the original wiki text is no longer available.

• So perhaps the “the original wiki text is no longer available” however it can be reconstructed and set to a variable using $parseTreeNodes or $slotFillParseTreeNodes.

If you want to experiment;

\widget $custom.widget()
<$parameters $custom.widget $parseTreeNodes=tree $slotFillParseTreeNodes=slots>

;Tree
<<tree>>

;Slots
<<slots>>
</$parameters> 
\end

<$custom.widget p1="p1value">
is complex

Body of widget

is complex

More body text?
</$custom.widget>

Which results in

Tree

[{"type":"text","text":"\nis complex\n\nBody of widget\n\nis complex\n\nMore body text?\n","start":190,"end":247}]

Slots

{"ts-raw":[{"type":"text","text":"\nis complex\n\nBody of widget\n\nis complex\n\nMore body text?\n","start":190,"end":247}],"ts-missing":[{"type":"text","text":"\nis complex\n\nBody of widget\n\nis complex\n\nMore body text?\n","start":190,"end":247}]}

• You can see here with the correct use of JSON filter operator we can extract and set a variable containing whatever we want.
  • Are there any examples already in the core? Seems like it should be in the documentation.
• Just add additional $fill widgets to see the impact.
• Further we can experiment with the depth

[CodaCoder]

I’ve just spent ~40 minutes digging around in here. I’m not even sure why

Can you explain why you want your tree and slots to remain unrendered? Because that’s what you are getting from the output: unrendered “stuff” from the call to your widget. Where would that be useful?

And before you doubt me…

I added a simple macro at the top:

\define my-macro()
This is the content of my-macro
\end

I added a call to <<my-macro>> in the call to your widget. Both the tree and slots output contained references to my-macro, but nowhere is mentioned the content of the macro.

Am I missing something? Right now all I see is a complex way to represent raw code, unrendered.

[TW_Tones]

Sure, let me explain.

Note: I am looking for a mechanism not a specific output, my last post only demonstrates it is possible, in wikitext.

If you go to the topic top you can see I am unhappy with my access within a custom widget definition to the body of the calling widget, currently the documented method is <$slot $name="ts-raw"/> which is a widget. It is not suitable for parsing.

\widget $my.custom()
<$slot $name="ts-raw"/>
\end

<$my.custom>

Body of the widget
Could be Wikitext, tiddlywiki script
text
data
HTYML
<$my.custom>

• to do anything with the content of the calling widget beyond simply displaying it requires “wikifcation” of <$slot $name="ts-raw"/> before you can even parse it.
• I wanted the body of the calling widget to be available in a variable. Variables can be parsed, split on new line, searched and manipulated. The output of a widget is much less useful.
  • It gets even more complicated if you are using fill widgets to define slots.

Now Jeremy said the body was not available to the widget, this is still true in the way he phrased it, but the example I gave shows the body of the calling widget is available “encoded inside” the values returned via $parseTreeNodes=tree $slotFillParseTreeNodes=slots variables. Just as parameters are available with $parms=parmater-variable

• Thus all that is left to do is to extract the body.

<$let body="[<tree>.. decodefilter]">

Where tree is like the following (From previous example).

[{"type":"text","text":"\nis complex\n\nBody of widget\n\nis complex\n\nMore body text?\n","start":190,"end":247}]

And the body will be available as a variable I can parse, list, split, transform, wrap, search and replace etc…

\nis complex\n\nBody of widget\n\nis complex\n\nMore body text?\n"

becomes

is complex

Body of widget

complex

More body text?

• I could remove the first “leading” \n if I wanted.

[CodaCoder]

Okay, got it, thanks. That’s much clearer.

So my question/concern remains. You’re going to have to deal with everything that the widget body wikitext can possibly contain. IOW, you’ll need to evaluate (wikify maybe?) everything to ensure you get everything. Your example only contains simple strings/text which are by their nature “ready to go”. Basically, every "type":"transclude" will need wikification… I think.

Unless, of course, you’re happy with a prerendered output like that?

[TW_Tones]

@CodaCoder

You’re going to have to deal with everything that the widget body wikitext can possibly contain

Yes and no.

Yes, Lets say I write a custom widget $.csv.table I will not expect it to contain more than CSV (Comma separated values) and the parsing will be about handling CSV data.

No, I could simply write a custom widget that does two things only to the content;

• Display it raw like code
• Render it as usual

In this case it will just do it slavishly whatever the content. Garbage in Garbage out or Genuis in beauty out.

Then later parameters could specify what to do with it.

[saqimtiaz]

@TW_Tones

So perhaps the “the original wiki text is no longer available” however it can be reconstructed and set to a variable using $parseTreeNodes or $slotFillParseTreeNodes.

@TW_Tones

Are there any examples already in the core? Seems like it should be in the documentation.

The conversion of wikitext to a parse tree is a lossy process, i.e. information is lost, such as pragmas, whitespace and comments. Therefore the conversion of a parse tree back to wikitext is not currently possible in a reliable and universal manner and is thus never used by the TiddlyWiki core. If it suits your purposes the JSON filter operators should give you everything you need to write an appropriate filter.

This topic of wikitext to parse tree round trip conversion often comes up in developer discussions in the context of WYSIWYG editors and the first step towards achieving it would be extending the parser, however there are backwards compatibility concerns that would need to be considered.

[TW_Tones]

I am sure there are bigger picture issues like those that you outline, however in this case if someone chooses to build a custom widget to pass in the body content, they are in effect defining themselves what content is relevant and providing handling.

• It simply allows the body content to be used as a parameter value or variable
  • It makes no other promises.
  • In my limited testing I have seen no losses, although I can see it gets messy as it tries to parse it, when I only want what the ts-raw returns.

I don’t know how the core is handling this, nor do I know how to code a simpler solution so I am not in a position to design a solution, only work arounds.

What I do hope I have done is illustrated why this would be helpful, if the “raw” body could be retrieved in a variable.

[saqimtiaz]

@TW_Tones

What I do hope I have done is illustrated why this would be helpful, if the “raw” body could be retrieved in a variable.

The potential utility of accessing slot content as variables and affordances for manipulating parse trees is well recognized amongst core developers and was discussed a fair bit in the lead up to v5.3.0.

However, as explained above, there isn’t any low hanging fruit at this time in terms of how the core can address these needs in a reliable and universal manner. As such, experimentation and exploration in this area from the community is highly welcome.

[TW_Tones]

Given the issues raised here, I will not pursue this idea which seems easy on the surface but is not under the hood. I will maintain a “watching brief” to see if I can find an effective way around it.

I can however generalise this a little more and restate the requirement and have found a new mechanisium that could be developed. However it relates to previous ideas.

The Problem
It would be helpful as we write a tiddler, be it full of tiddlywiki script or content that we could create sections inline, that is in the appropriate position for the content to be, not pinned at the top and defined inside pragma. That is we can maintain a top to bottom serial stream of information.

• I thought “calling” a custom widget with the body of the widget would be a way to introcuce sections.

<$custom.section>
various content
in various forms and layouts
with and without wikitext
</$custom.section>

• Then using parameters and the design of the custom widget use this content in one or more ways.
• In part because this content was addressable using the ts-raw reference.
• However this has proven limited.

With further consideration I realise what I am interested in is entering content “in line” and in and addressable way such that it can be reused in one or more ways.

This following is an approach for which I have a proof of concept working;

<section>
various content
in various forms and layouts
with and without wikitext
</section>

Then using a special transclusion template which I pass the section name to

{{tiddlername||special-template|sectionname}}

• This template can extract the content between <sectionname> and </sectionname>. Defaulting to the current tiddler.
• Such a transclusion can be used to;
  • Set an attribute to the content of the sectioname
  • Thus set a variable to the content of the section name.
  • Thus the inline content is addressable where ever it exists within the content including anywhere inline.
  • Any subsequent custom parsing or processing can thus be done to the content of that section.

[CodaCoder]

I’m treating your last as an RFC, Tones, so treat my responses accordingly.

@TW_Tones

With further consideration I realise what I am interested in is entering content “in line” and in and addressable way such that it can be reused in one or more ways.

Me too, as you know, I’ve done it and shown it to you. So I’m with you all the way here, until you say…

@TW_Tones

Then using a special transclusion template which I pass the section name to

Now you’ve lost all that wonderful “inlinedness” and sent the processing out of sight elsewhere. That’s fine, of course, in the traditional TW sense, but you haven’t kept to your promise to yourself (or your outlining “wish”).

Then you said (I numbered your bullets) …

@TW_Tones

Such a transclusion can be used to;

1. Set an attribute to the content of the sectioname
2. Thus set a variable to the content of the section name.
3. Thus the inline content is addressable where ever it exists within the content including anywhere inline.
4. Any subsequent custom parsing or processing can thus be done to the content of that section.

1. You don’t need this. I could be wrong, but I’ve never needed it and I do this extensively.

2. Mine are variables (well, transclusions). I’ve never needed to assign them to another variable using $let for example. In essence, I pass the identifier/locator around, not the block itself.

3. Mine are addressable as elements wherever they exist – that, after all, is whole point of them: rejig, reuse, ad infinitum. (Aside, I often wonder why that isn’t infinita – but I’m not an ancient Roman.)

4. That sounds like the job of the surrounding context which of course is always the case anyway.

That’s a lot of theorizing in the abstract. Here’s something concrete:

<about-fission>
Lorem ipsum dolor sit amet, <<some-macro>> consectetur //adipiscing// elit...
</about-fission>

about-fission can be pulled out from its host tiddler and resused elsewhere.

[TW_Tones]

@CodaCoder

I’ve done it and shown it to you.

Can you point me to this please?

I need to review it to ensure it am “on the same page as you”.

[CodaCoder]

Best Practice: Tiddlywiki and Structure of Long Note

I do this ALL the time, though I call them author-elements and I don’t use the technique at the chapter level. In my world, a chapter is a grouping of tiddlers. A chapter-section (chapsec) is a tiddler. Chapsecs belonging to the same chapter have the same number/name in their chapter field. A chapsec might contain zero or more author-elements: <about-fission> Stuff about nuclear fission </about-fission> <about-bilbo-baggins> Stuff about Bilbo Baggins </about-bilbo-baggins> Another tiddler …

Section Editor - Prototype

I think you should use custom elements <mo-section> stuff </mo-section> I appreciate you want to use Hn tags (and they’re ready to go in the toolbar) but setting aside your very own elements seems better somehow (gives you a bit more control over their meaning, too). I’m sure you could add them to the toolbar if you wanted to. Other than that, great piece of work @Mohammad !

Parsing html and other tags

Folks, I know this has being covered before, but just can not locate it. 40+mins searching so far. Someone published the way to parse a tiddler containing tags such as one or more <atag>something</atag> in a tiddler. It also took account of attributes such as <atag some=attribute>something</atag>. Of course TiddlyWiki’s prolific use of “tags” is possibly making it difficult for me. Any pointers appreciated. For anyone interested in why? I want to develop a method where you can use an arbi…

[CodaCoder]

I thought there were more – there may be, but I can’t think if any more search strings to try (right now).

[TW_Tones]

Thanks for those references. Mohamads find macro may be the answer I lost.

• If I can separate the attributes on the open tag.

You are right I have come at this subject a number of times but for different reasons. As you can see at the top of this topic I was asking for access to slots as variables and returning to arbitary html tags because this is not possible.

• It should be possible to address/extract named sections out of a tiddlers text and use it however we wish.
  • In keeping with this Topic as a variable

It does get more complex once we take account of;

• accounting for attributes on the open tag
  • 1st separate it from the content of the section
  • 2nd Make the attributes available for further parsing if desired.
• more than one section with the same name, in the same tiddler (perhaps just append?)
• Nested sections
• Addressing sections in the same and current tiddler (for reuse)

Perhaps this thread should come to an end and a fresh start to “address arbitrary sections in tiddlers”.