Streams Fusion: export from Streams

Plugins
,
#1

ok @JanJo and everyone, 0.0.9-beta released which includes paragraph-based (but flat) fusion and introduces “fuse-to-markdown” features.
Regular install is best through the SQPL

Please test and report back. I note that for some reason, the inline <code> blocks don’t convert well to markdown. If you have ideas how I can fix that, please chime in.

Otherwise, markdown conversion works pretty darn well and could become a big deal (plenty of credits to @saqimtiaz again for the Turndown integration). If you have advice or opinion on extracting that into a separate plugin, please chime in also. I sorta miss how we could install prior versions of this addon directly with drag-n-drop without reloading; extracting the markdown conversion into a separate plugin would restore that by removing the javascript portions of the addon.

3 Likes
Soliciting favorite wikis, plugins, and resources on notes for Grok TiddlyWiki
[ Streams ] : future development
Principles plugin - an alternative layout to record life principles as tiddlers and review them
[ Streams ] : future development
#2

The demo doesn’t seem to work for me. It doesn’t matter which output format I choose, it always flattens to “bullet points”.

I’m using ubuntu 20.04 and FF latest atm.

#3

Very neat @fastfreddy and very pleasing for me personally that you got some value out of exploring the Typewriter demo. I’ve been using the view mode Typewriter editor successfully in an edition with some friends for some time, and its been somewhere on my far too long list of things to do to turn the Turndown stuff into a separate plugin.

Pretty simple really. Separate the Turndown library tiddler and associated macro/widget/filter into a separate plugin. The Typewriter demo also tweaks Turndown to do HTML to wikitext, so your Turndown plugin could handle wikitext <=> HTML <=> markdown for a subset of mark up.

Then in Streams Fusion you can test for the presence of this Turndown plugin - or a macro or filter operator that it provides - to toggle the Markdown related features. A filter operator might be the most versatile, but using a macro or a widget are viable choices too.

I think Streams Fusion deserves its own thread for greater visibility. Let me know if you want me to split off your last post + subsequent replies into a new thread.

#4

What is the Typewriter-Editor? I could not find it in the whole site?

#5

@JanJo It is a demo I posted a long time ago. There wasn’t any interest when it was first shared so subsequent development has been private for specific personal purposes.

2 Likes
#6

You are correct @pmario; release quality issue once again.
Fixed (hopefully) in 0.0.10-beta just released.
Thanks for testing; seems I can’t test sufficiently well myself :wink:

#7

Thanks for the guidance @saqimtiaz !

I agree, should be simple really; any recommendation on testing for the presence of a loaded plugin? Streams-fusion could enable the Markdown export format only when the other plugin is present and enabled.

Should I conclude we should proceed in that direction? I find these parts a bit intimidating and I am afraid my solution would fall short of the ultimate possibility:

Regarding splitting off Streams-fusion, no concern; I trust your judgement on what is best.

Lastly, any leads/insight as to why the inline preformatted code doesn’t seem to take (the funk node in my demo Stream)? When I feed the raw html output (from my plugin) into the Turndown demo site, markdown enders correctly, but not within TiddlyWiki… I tried updating the inner content of $:/plugins/sq/typewriter/lib/turndown-browser-umd.js thinking it was versioning issue, but it doesn’t appear to be the case.

#8

If you are using a macro, then you can check for its presence via a filter. Assuming the JavaScript macro has a name property with value turndown:
[[macro]modules[name],[turndown]]

I think treating the Turndown stuff as a separate plugin that provides macros or filter operators is a good way to have flexibility and reusability.

I recommend grabbing the latest turndown-browser-umd.js from https://unpkg.com/turndown@7.1.1/lib/turndown.browser.umd.js. There is a chance I may have been directly modifying that copy in the demo for the purposes of the Typewriter, since that is the very first demo and very raw.

I am juggling a few too many things at present, but if you can provide a minimal test case of the issue with preformatted code and turndown (independent of Streams or Streams Fusion) I can take a look.

1 Like
#9

Hi @saqimtiaz; I have starting splitting off Markdown-export from Streams-fusion.

Here’s a very very minimal not-working example with inline preformatted code Turndown test — minimal not-working example

When you visit the page, you’ll first see a very simple wikitext tiddler; then just below, the rendering in raw HTML followed by the Turndown conversion using the latest turndown library you linked just above. The inline code block doesn’t carry over into the markdown. However, if I copy-paste the basic HTML (<h1 class="">hello</h1><p>what is <code>funky</code></p>) to Turndown Demo, it renders the inline code block well.

When and if you have a chance… I have other things to keep me busy adapting the two plugins after splitting anyways.

much appreciated…

1 Like
#10

I have only had a very quick look but this seems to be a by product of the way TiddlyWiki renders text. So there are three issues with this code:

<$wikify name="htmloutput" text={{wikitext sample}} output="html">
	<$wikify name="mdoutput" text=<<toMarkdown>> >
		<$text text=<<mdoutput>> />
	</$wikify>
</$wikify>
  1. You are outputting plain text
  2. The output isn’t in block mode
  3. TiddlyWiki requires two line breaks to render a line break, but the Turndown output only adds one between the heading and the = characters. If you were using an action-setfield widget to save this output to a tiddler with type text/html I suspect it would render just fine.

This would probably help with the first two:

<$wikify name="htmloutput" text={{wikitext sample}} output="html">
	<$wikify name="mdoutput" text=<<toMarkdown>> >

		<<mdoutput>>

	</$wikify>
</$wikify>

In such situations it is often worth checking if the data is correct and whether it is a render problem. Useful here is the <$log> widget. For example to see exactly what your macro is outputting:

<$log md=<<mdoutput>>/>

Also, I realize this is just a demo to show the problem but have you given some thought to what you imagine the intended workflow for using the Markdown conversion will be? Converting and displaying in real time? Or converting and saving the output? Users will always find ways to use things in ways you had not considered but it is a good idea to have an intended workflow in your mind.

All of these wikify widgets more often than not suggest a better way to implement can be found, though as I said, it may just well be for expedience at present to show the problem.

1 Like
#11

Hello again, I have updated my test case file with additional tooling and reporting.

Agree the nested <$wikify widgets are not ideal. I’ll think of alternatives.

In terms of workflow, I had a couple of ideas:

  • allow users to export a markdown version of their tiddler, through an additional option on the existing Export tiddler button (see image)
  • similarly, give users an extra option exporting results of an advanced search
  • ideally, offer the option to export to clipboard in markdown
  • ideally, offer the option of converting a tiddler to markdown (and changing its type) if the official Markdown rendering plugin exists
  • ideally, offer to option to clone a tiddler to markdown (same as previous, but not overwrite)
  • (through Streams-fusion) continue offering the ability to fuse Streams into markdown

image
image676×516 27.4 KB

1 Like
#12

https://saqimtiaz.github.io/sq-tw/typewriter.html

Thanks for this Saq, this can help me working with users/colleages who sometimes are afraid of wikitext . There should be a lot of interest for this wysiwyg-editor !

1 Like
#13

Hello @fastfreddy

Thanks so much for developing this exporter.

It’s great, yet I think I’ve noticed an error in the code, and I believe I have fixed it.

I was looking for something to export to markdown bullets, but using “-” rather than “*”.

I was going from an old TW5 Streams version into Logseq.
Logseq it is more or less pre-baked for what I wished to do with this early Streams project (from about a year ago).
But they use the dash, not the asterisk style.

Can you make changes to the md output from the extended Streams Settings/Options Tab, in the Control Panel?

I can’t.

https://fastfreddy.gitlab.io/streams-fusion/#%24%3A%2Fconfig%2Fsq%2Fstreams-fusion%2Fturndown%2Fbullets updates, but it seems to not be called correctly.

I think it is the difference between how you define the macro in:
https://fastfreddy.gitlab.io/streams-fusion/#%24%3A%2Fplugins%2Fphiv%2Fstreams-fusion%2Fmacros
And how it is referenced in:
https://fastfreddy.gitlab.io/streams-fusion/#%24%3A%2Fplugins%2Fphiv%2Fstreams-fusion%2FturndownMacro.js

In the latter it is

bulletListMarker

In the former you use

bullets

I have changed all the “bullets” to “bulletListMarker” in my version of your macros file (linked above), and now it is working well.

I hope this helps you, you really help me with your plugin.

2 Likes
#14

Hi @trisaster ! Welcome to the community and thank you for the bug report (with fix included).
I have a new version released with the same fix you provided (and one other issue I found), for you to try.

Cheers,

1 Like
#15

Yes, that fixed it perfectly.

I’ve been wondering: is it possible to retain the links in the TW5 style double square brackets (which you probably know occurs in other apps also) while ‘fusing’ as markdown?

Do you have ideas why this is so hard to do?

To clarify, on your demo, if I put a link into a stream node, on the demo page, to [[Streams-fusion readme]]

In the conversion, it will become one of these:

  1. Streams-fusion readme
  2. [Streams-fusion readme][1]
    [1]: #Streams-fusion%20readme
  3. [Streams-fusion readme][] or 4. [Streams-fusion readme]
    [Streams-fusion readme]: #Streams-fusion%20readme

I just can’t get it to retain as [[Streams-fusion readme]]

I figured I might need to edit the $:/plugins/sq/typewriter/lib/turndown-browser-umd.js file, but what I’ve found is I can change line 261 (for example):
replacement = ‘[’ + content + ‘][]’;
to all kinds of things, but not:
replacement = ‘[[’ + content + ‘]]’;

For example, this is fine:
replacement = ‘[-[’ + content + ‘]-]’;

So I settled on a hacky work around:
replacement = ‘[[’ + content + ‘]’;
And I manually add the final ‘]’ as a final edit.

And, now think to use:
replacement = ‘[[’ + content + ‘]~’;
And do a find/replace of ‘~’ (find) ‘]’ (replace)

#16

Hi @trisaster ,
It looks like you’re primarily interested in markdown conversion (from your earlier post); please advise otherwise. When outputing to wikitext to clipboard, for example, the hyperlinks retain the [[ ]] markup from my testing.

The markdown conversion is handled by another 3rd party library (turndown), which I suspect you already know from your question.

I have not dived in too deeply into turndown yet, but will see what I can uncover. An elegant way to support your use case would be to add a hyperlink format option and have a matching section of code added to (rather than replacing) the turndown library. I think it may even be possible to invoke the turndown service with special rules but I need to study and experiment with that a bit. I looked into this option ($:/config/sq/streams-fusion/turndown/linkReferenceStyle) but is not quite what we were looking for. Maybe adding an option to ($:/config/sq/streams-fusion/turndown/linkStyle) makes more sense.

#17

Yes, what you write makes sense, and actually that is what I’ve done: Well almost, I’ve changed the output in the inline section of the JS file, rather than adding an option to the plugin. I still can’t get it to accept the double brackets. I wonder if the JS watches the output also, and removes the double square brackets. Just a guess, as I can’t really get my head around workflow.

I was doing okay with parsing wikitext when I was deeper into the TW5 ecosystem, but I never understood JS well at all. The file looks logical (if this is @saqimtiaz’s beautiful work then I’m not surprised it is easy to read), and I can hack it about. But again, I’ve hit the limit of what I can do.

Not to worry, I am fine with my global find and replace solution (hack). Although I’m sure others might enjoy the interoperability of making Streams export to a niche but semi-universal .md style, and play nicely with Roam, Logseq, Obsidian…

#18

I don’t have the opportunity at present to look at this in detail but perhaps the inlineLink rule from the old typewriter demo might provide some hints as it generates tiddlywiki syntax links to tiddlers.

    {
        key : "inlineLink",
        filter: function (node, options) {
          return (
            options.linkStyle === 'inlined' &&
            node.nodeName === 'A' &&
        //	( $tw.utils.hasClass(node,"tc-tiddlylink-external") || $tw.utils.hasClass(node,"tc-tiddlylink")) &&
            node.getAttribute('href')
          )
        },
        replacement: function (content, node) {
            var href = node.getAttribute('href');
            if ($tw.utils.hasClass(node,"tc-tiddlylink-external")) {
                if(href === content) {
                    return href;
                } else {
                    return "[[" + content + "|" + href + "]]";
                }
            } else {
                if(("#" + encodeURIComponent(content)) === href) {
                    return "[[" + content + "]]";
                } else {
                    return "[[" + content + "|" + decodeURIComponent(href.substring(1)) + "]]";
                }
            }
        }
    }

You can also pass custom rules to turndown using turndownService.addRule() where turndownService is the instantiated copy of turndown, see the following documentation:

1 Like
#19

I can’t install streams-fusion on node.js with TiddlyWiki 5.2.2. I tried to install it through the SQ Plugin Library, which works in the the first place. But everytime I close the node.js instance and restart the server with tiddlywiki streams --listen I got the following error:

Error executing boot module $:/plugins/phiv/streams-fusion/turndownMacro.js: {}

$:/plugins/phiv/streams-fusion/turndownMacro.js:21
window.TurndownService = require("$:/plugins/sq/typewriter/lib/turndown-browser-umd.js");
^

ReferenceError: window is not defined
    at $:/plugins/phiv/streams-fusion/turndownMacro.js:21:1
    at $:/plugins/phiv/streams-fusion/turndownMacro.js:57:3
    at Script.runInContext (node:vm:139:12)
    at Script.runInNewContext (node:vm:144:17)
    at Object.runInNewContext (node:vm:298:38)
    at $tw.utils.evalSandboxed ($:/boot/boot.js:601:5)
    at $tw.modules.execute ($:/boot/boot.js:879:15)
    at $:/boot/boot.js:919:30
    at $tw.utils.each ($:/boot/boot.js:155:12)
    at $tw.modules.forEachModuleOfType ($:/boot/boot.js:918:12)
#20

Any ideas @fastfreddy @saqimtiaz how to solve this?