Can TiddlyWiki read external tiddlers from URL parameter?

[papiche]

Like md.htm is able to display markdown file from external url

https://zhlicen.github.io/blog/md.htm?src=https://raw.githubusercontent.com/microsoft/vscode/main/README.md

Does the same feature is available to TW?

I mean, could it be possible to indicate tiddlers.json location so it loads as extra displayed tiddlers…

https://ipfs.copylaradio.com/ipfs/QmcoQ3u6F6bKdyXm84kdLYxrKHXjmPizh2RTzjKX77a9mJ/tiddlers.json

I hope it is possible
Thanks

[twMat]

If I undertand you right, you want to display tiddlers from an arbitrary external TW by including the tiddlers permalink in the current wikis URL.

That would indeed be useful and, pushing it a step further, imagine “aggregating” the content of a wiki by specifying multiple permalinks.

Other than iframes, you may want to check out the Share plugin. This is probably not exactly what you ask for but related.

[Springer]

You might find your solution here:

New Features in TiddlyWiki 5.3.0: HTTP request and deserialize operator

Amongst the new features being introduced in TiddlyWiki v5.3.0 is the ability to make HTTP requests from wikitext. This will allow users to fetch data from other sources and convert them into tiddlers, amongst other things. The new release will also include a deserialize[] filter operator that converts the various data formats that TW understands into a JSON representation of tiddlers. Uses case like loading tiddlers from another wiki will now be entirely possible from wikitext as long as the …

… and that thread points you to this plugin:

https://saqimtiaz.github.io/tw5-plugins-sandbox/#ExternalContent

[papiche]

This is exactly what I was looking for.
This will perfectly fit and ease the Web3 architecture i am setting up (using TW)

muchas gracias

[papiche]

Now testing @saqimtiaz “External Content” plugin that seem it can “do the job”

---

UPlanet

How It Works:

1. Personal Data Holder in IPFS:

• Each TiddlyWiki (TW) is assigned a unique “crypto key” known as the “Web3 Address”: /ipns/$publicKEY. Users record their data in TiddlyWiki, including the use of plugins for running new apps.

2. Registration on UPlanet:

• TWs created on “UPlanet” are registered on a specific location called UMap, represented by a 0.01° piece. These TWs are linked with a “Geospatial key,” forming a kind of blockchain that documents the information they collect.

3. Nightly Blockchain Engine:

• Every night, a “blockchain engine” retrieves and disseminates information among TWs through “data bridges.”
• Information is shared with TWs in the same sector (local) and with specific tags from friends’ TWs located abroad.

4. Data Trust Affinity:

• TWs in sectors and beyond collect tiddlers based on the number of signatures they possess.
• This principle is utilized to refine “data trust” from local to global, creating a decentralized information ecosystem.

For more detailed information, you can refer to the “UPlanet white paper draft.”

Why External URLs?

The External URL is employed to simplify the process of obtaining signatures for Tiddlers. Currently, users need to manually visit all sectors and friends’ TWs and copy the desired content into their own TWs. While functional as a proof of concept, this method is not highly practical.

Improvements with ExternalContent

To enhance usability, it would be beneficial to display “yesterday” data from both “local” and “abroad” sources.

Install in my TW

• Added “External Content” plugin to

• Added

  

• Created a Tiddler with Import “button”

---

1. Test “HTML” mode

\procedure loadMyWiki()
\import [[$:/plugins/sq/ExternalContent/loadWikiActions]]
<$transclude
    $variable="loadWikiActions"
    wikiURL="/ipns/k51qzi5uqu5djfls7kbaf7vplpm0n2665x18nv9ol6xz83aohz2g33srvhx583"
    contentFilter=":[!is[system]tag[G1CopierYoutube]!tag[G1Voeu]!sort[modified]limit[2]]"
    deseralizer="text/html"
    />
\end

<$button actions=<<loadMyWiki>> > load content </$button>

Click & wait a little… Then IT WORKS !!
My TW gets populated with external tiddlers

Todo

In order to respect “signature consistency”,
it needs to add “Tags” from $:/config/NewTiddler/Tags (containing signature).
that plugin make that happens “on import”

You help is welcome to modify $:/plugins/sq/ExternalContent/filters/jsonfiltertiddlers.js to do it also on external imported Tiddlers…

---

2. Test “JSON” mode

I give a a try loading “json”…

\import [[$:/plugins/sq/ExternalContent/loadWikiActions]]
<$transclude
    $variable="loadWikiActions"
    wikiURL="/ipfs/QmcoQ3u6F6bKdyXm84kdLYxrKHXjmPizh2RTzjKX77a9mJ/tiddlers.json"
    contentFilter=""
    deseralizer="application/json"
    />
\end
<$button actions=<<loadMyWiki>> > load from JSON </$button>

couldn’t figure out what is correct deseralizer value to use ?

---

Conclusion and Reflection:

• Overall positive experience testing ExternalContent.
• Acknowledges the need for a few adjustments and an automatic process to distribute local and abroad URLs.
• Likens TiddlyWiki to becoming a kind of “Windows95” for Web3.

[saqimtiaz]

@papiche

In order to respect “signature consistency”,
it needs to add “Tags” from $:/config/NewTiddler/Tags (containing signature).

This plugin is just surfacing the native HTTP handling capabilities of TW. To add tags to all imported tiddlers just modify the loadWikiActions procedure:

saqimtiaz/tw5-plugins-sandbox/blob/e9ebad0708d6c15eb0b8974a3d9328e3b0c0e9af/plugins/sq/ExternalContent/load-wiki-actions.tid#L28-L32

[TW_Tones]

@papiche

couldn’t figure out what is correct deseralizer value to use ?

Use the one for the result you want. I have used the deseralizer on the “browse widget” set to deserializer=“application/json” to import it as JSON, and not “import the tiddlers within”.

• The browse widget also allows you to set the default extensions eg .json, .tid, .txt, .csv

use none or “text/vnd.tiddlywiki” to import tiddlers within.

[saqimtiaz]

@papiche

I give a a try loading “json”…

\import [[$:/plugins/sq/ExternalContent/loadWikiActions]]
<$transclude
    $variable="loadWikiActions"
    wikiURL="/ipfs/QmcoQ3u6F6bKdyXm84kdLYxrKHXjmPizh2RTzjKX77a9mJ/tiddlers.json"
    contentFilter=""
    deseralizer="application/json"
    />
\end
<$button actions=<<loadMyWiki>> > load from JSON </$button>

couldn’t figure out what is correct deseralizer value to use ?

application/json is the correct deserializer to use to import JSON as tiddlers. However, the JSON needs to be properly formed as either an object or an array of objects, where each object has at least a title property.

See: https://github.com/Jermolene/TiddlyWiki5/blob/070327cb5708cfad104e2f325115203b499af631/core/modules/deserializers.js#L19-L41

An overview of deserializers available in the core:
https://tiddlywiki.com/#Deserializers

[TW_Tones]

Of course if the json is non standard you can import as plain text and write your own parser even in wiki text the Jason mangler plugin may help as well, use it to import and generate tiddlers even from csv.

[papiche]

Thanks for all your advice.

I must confess that I am a pretty bad javascript developer and even worse “TW programmer”

I am amazed about the rock solid and expandable environment it is and i wouldn’t like to screw too much things by tweaking too much

---

For now. I am not sure I can add any good code into

@saqimtiaz

https://github.com/saqimtiaz/tw5-plugins-sandbox/blob/e9ebad0708d6c15eb0b8974a3d9328e3b0c0e9af/plugins/sq/ExternalContent/load-wiki-actions.tid#L28-L32

How would you copy to inserted tiddlers “tags” the value that is stored in $:/config/NewTiddler/Tags ?
This will be a good exercise to make my comprehension progress

---

@saqimtiaz

However, the JSON needs to be properly formed as either an object or an array of objects, where each object has at least a title property.

UPlanet is producing a sub-key for every “G1Tag” channel anyone would like to populate.
TW is then producing every day a json export of the last 30 tiddlers containing this tag.

But when I try to import some, nothing happens (or I miss it, color change would help)
I don’t understand what is wrong? It is an array of objects : like this
May be loading time is too long (if data is not requested much, which is the case in that low inhabitants UPlanet)

Or isn’t it deserializer / deseralizer typo problem in button action?

---

There is a case, when tiddler with same title already exists, It could be same or different. I don’t know what would happen in UX.

For UX purpose, I wonder if those “inserted tiddlers” could be in different color (solarized actual css colors would be great) but i don’t know how to do it good and compatible with any template…

---

@TW_Tones

Of course if the json is non standard you can import as plain text and write your own parser even in wiki text the Jason mangler plugin may help as well, use it to import and generate tiddlers even from csv.

As you say, it could be possible to create export / import of any kind.
Even better, add some more…

Before reaching that “level” i still need to explore and understand actual filters…
One of my question is how could I export tiddllers as “static HTML file” using tiddlywiki from command line ?

For now i can just do it from Web browser (then ipfs add command on downloaded file)

So blockchain engine could make this json
becoming html and render “Canon_de_Pachelbel”

if i could know what ‘exporter’ i should put in that command…

tiddlywiki --load ${TW} --output /tmp --render '.' "${PLAYER}.g1tag.json" \
 'text/plain' '$:/core/templates/exporters/JsonFile' 'exportFilter' '[tag[G1Tag]]'

---

Sorry if it is like dummy questions for you.
I prompt this topic to ChatGPT … It does not seem to be right.

---

Gratitude for and and all of your help.

Continuing test and investigations…

[oeyoews]

oeyoews/tiddlywiki-starter-kit/blob/main/tiddlywiki.info

[papiche]

@saqimtiaz

• solved not working “json import”
  this was due to deser(i)alizer typo in example code…

• I also put ‘:’ as leading character for

contentFilter=":[...

and made a “pull request”

---

About “tags” management,
This is my actual attempt to add $:/config/NewTiddler/Tags in newtags

<!-- get the tiddler at that position in the array from the JSON -->
<$let
    tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
    title={{{ [<tiddler>jsonget[title]] }}}
    newtags={{{ [<tiddler>jsonget[tags]] }}} $:/config/NewTiddler/Tags 
>
    
    <$action-log newtags=<newtags> />

    <!-- make sure we have a title for the tiddler and exclude system tiddlers -->
    <$list filter="[<title>!is[blank]]">
        <$action-setmultiplefields
            $fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp tags"
            $values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] <newtags>"
            $timestamp="no"
            />
    </$list>
</$let>

not good yet…

---

@oeyoews

https://github.com/oeyoews/tiddlywiki-starter-kit/blob/main/tiddlywiki.info

Thanks.
“tiddlywiki-starter-kit” looks great ! It is a full TW processing tunnel that produces a “blog” ?
I would need more details to understand how and what i can learn from ?

[saqimtiaz]

Try this:

<!-- get the tiddler at that position in the array from the JSON -->
<$let
    tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
    title={{{ [<tiddler>jsonget[title]] }}}
    newtags={{{ [<tiddler>jsonget[tags]] [{$:/config/NewTiddler/Tags}] :and[join[ ]] }}}  
>
    
    <$action-log newtags=<newtags> />

    <!-- make sure we have a title for the tiddler and exclude system tiddlers -->
    <$list filter="[<title>!is[blank]]">
        <$action-setmultiplefields
            $fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp tags"
            $values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] =[<newtags>]"
            $timestamp="no"
            />
    </$list>
</$let>

---

I appreciate the pull request correcting the typo. The leading : is actually not needed so if you are able to update that PR please do, otherwise I will merge an alternate fix when I get the opportunity.

[papiche]

Thank you

@saqimtiaz
I applied your code guidance
made a this pull request.

…

but…
when I update $:/plugins/sq/ExternalContent/loadWikiActions in my TW
something broken !
button no acting , nor I can see log…

I was warned before modifying $:/ tiddler.
Is it because TW apply plugin version consistency check ?

[saqimtiaz]

I am on the road at the moment and cannot debug properly.

Try this replacing the entire contents of $:/plugins/sq/ExternalContent/loadWikiActions with the following:

\procedure startupConfigTitle() $:/config/sq/ExternalContent/load-on-startup
\procedure loadWikiActions(wikiURL,contentFilter,deserializer)
	<!-- actions invoked after fetching the remote data -->
	\procedure getWikiCallback()
		<!-- actions to turn the remote data into tiddlers -->
		\procedure importTiddlers()
			<$let
				passthroughFilter="[all[]]"
				defaultDeserializer="text/html"
				contentFilter={{{ [<contentFilter>!is[blank]else<passthroughFilter>] }}}
				deserializer={{{ [<deserializer>!is[blank]else<defaultDeserializer>] }}}
				tiddlerJSON={{{[<data>deserialize<deserializer>jsonfiltertiddlers<contentFilter>]}}}
				tiddlers={{{ [<tiddlerJSON>jsonindexes[]] :map[<tiddlerJSON>jsonget<currentTiddler>,[title]] :and[format:titlelist[]join[ ]] }}}
			>
			<$action-log $$filter="wikiURL contentFilter deserializer tiddlers tiddlerJSON"/>
				<!-- iterate over each position in the array -->
				<$list filter="[<tiddlerJSON>jsonindexes[]]" variable="index">
					<!-- get the tiddler at that position in the array from the JSON -->
<!-- get the tiddler at that position in the array from the JSON -->
<$let
    tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
    title={{{ [<tiddler>jsonget[title]] }}}
    newtags={{{ [<tiddler>jsonget[tags]] [{$:/config/NewTiddler/Tags}] :and[join[ ]] }}}  
>
    
    <$action-log newtags=<<newtags>> />

    <!-- make sure we have a title for the tiddler and exclude system tiddlers -->
    <$list filter="[<title>!is[blank]]">
        <$action-setmultiplefields
            $fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp =tags"
            $values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] =[<newtags>]"
            $timestamp="no"
            />
    </$list>
</$let>
				</$list>
			</$let>
		\end importTiddlers
		<!-- actions invoked if there is an error fetching the data -->
		\procedure failureHandler()
			<$action-log status="error fetching the wiki"/>
			<$action-setfield $tiddler={{{ [[$:/temp/http/error/]addsuffix<now [UTC]YYYY0MM0DD0hh0mm0ssXXX]>] }}} text={{{ [[There was an error fetching the wiki ]addsuffix<wikiURL>addsuffix<error>] }}} tags="$:/tags/Alert"/>
		\end failureHandler
		<$list filter="[<status>match[200]]" variable="null" emptyValue=<<failureHandler>> >
			<$action-log data=<<data>> status="succcess" />
			<<importTiddlers>>
		</$list>
	\end getWikiCallback

	<!-- fetch the remote data source-->
	\procedure getWikiActions()
		<$action-sendmessage
			$message="tm-http-request"
			method="GET"
			bind-status={{{ [[$:/temp/http/load-content/]addsuffix<wikiURL>] }}}
			oncompletion=<<getWikiCallback>>
			url=<<wikiURL>>
			var-wikiURL=<<wikiURL>>
			var-contentFilter=<<contentFilter>>
			var-deserializer=<<deserializer>>
			>
	\end getWikiActions

<!--  check if its a tiddlyhost URL and if so map it to the json file /tiddlers.json, also trim any trailing slashes  -->

<$let isTiddlyHost={{{ [<wikiURL>regexp[(?i)^https:\/\/\S+tiddlyhost.com]then[yes]else[no]] }}}
	wikiURL={{{ [<isTiddlyHost>match[yes]] :then[<wikiURL>!suffix[tiddlers.json]trim:suffix[/]addsuffix[/tiddlers.json]] :else[<wikiURL>] }}}
	deserializer={{{ [<isTiddlyHost>match[yes]then[application/json]else<deserializer>] }}}
	>
	<$action-log $$filter="isTiddlyHost wikiURL deserializer"/>
	<<getWikiActions>>
</$let>
\end loadWikiActions


<!--  ARE IMAGES IMPORTING CORRECTLY?   -->

If that does not work for you, you will need to wait until I have a chance to publish an updated version of the plugin that allows specifying optional additional actions to invoke on every incoming tiddler, and that likely wont happen for another 6 weeks or so.

[papiche]

@saqimtiaz

If that does not work for you, you will need to wait until I have a chance to publish an updated version of the plugin that allows specifying optional additional actions to invoke on every incoming tiddler, and that likely wont happen for another 6 weeks or so.

It does work !
Great, i can inter-link TWs with “cross keys” loading.
… Have a nice trip on the road…

i would need and appreciate a “tw programming” lesson

meanwhile, i discovered that clue you dropped

How to change tiddler background color based on content-type

TiddlyWiki, as of a few versions ago, makes it very easy to add additional classes to the page template wrapper or to the tiddler template using filters. Add a tiddler tagged: $:/tags/ClassFilters/TiddlerTemplate with the following text: [<storyTiddler>type[text/markdown]then[md-tiddler]] Then add a tiddler tagged $:/tags/Stylesheet with the text: .md-tiddler { background: <<colour code-background>>; } References: https://tiddlywiki.com/#SystemTag%3A%20%24%3A%2Ftags%2FClassFilters%2FT…

… trying to adapt method to “is_volatile” field

… TW structure understanding evolves: TW $:/ looks like old Windows C:/

[Springer]

@papiche

… trying to adapt method to “is_volatile” field

Yes. In case anyone wants to see such a css solution in action:

Quick-demo TiddlyWiki models

[papiche]

@Springer

see such a css solution in action:

Thank you for this great demo place !

You made me found all i needed for “local / abroad” tiddler status control.

After having imported

1. $:/springer/volatile/css
2. $:/core/ui/ViewTemplate
3. externalTiddler_puzzle
4. $:/publishFilter

css applyed immediately !
did i get enough tids to get “is_volatile” removed when saving ?

I wonder “what is for”, what is inside import.bundle ?
TW tweaking capabilities are really incredible

[papiche]

@papiche

get “is_volatile” removed when saving ?

@Springer
your change in publishFilter makes is_volatile=yes tiddlers not recorded.

In my user case, i would like to give an easy way to “keep it local”,

Actual action is to remove is_volatile field and save Tid.
Any guidance on how I could add a “tid control button” executing that “program” ?

[Springer]

To have your wiki keep a local copy of the imported tiddlers, just remove that $:/publishFilter