Can TiddlyWiki read external tiddlers from URL parameter?

Like md.htm is able to display markdown file from external url

https://zhlicen.github.io/blog/md.htm?src=https://raw.githubusercontent.com/microsoft/vscode/main/README.md

Does the same feature is available to TW?

I mean, could it be possible to indicate tiddlers.json location so it loads as extra displayed tiddlers…

https://ipfs.copylaradio.com/ipfs/QmcoQ3u6F6bKdyXm84kdLYxrKHXjmPizh2RTzjKX77a9mJ/tiddlers.json

I hope it is possible :wink:
Thanks

If I undertand you right, you want to display tiddlers from an arbitrary external TW by including the tiddlers permalink in the current wikis URL.

That would indeed be useful and, pushing it a step further, imagine “aggregating” the content of a wiki by specifying multiple permalinks.

Other than iframes, you may want to check out the Share plugin. This is probably not exactly what you ask for but related.

You might find your solution here:

… and that thread points you to this plugin:

https://saqimtiaz.github.io/tw5-plugins-sandbox/#ExternalContent

This is exactly what I was looking for.
This will perfectly fit and ease the Web3 architecture i am setting up (using TW)

muchas gracias :wink:

1 Like

Now testing @saqimtiazExternal Content” plugin that seem it can “do the job”


UPlanet

How It Works:

  1. Personal Data Holder in IPFS:
  • Each TiddlyWiki (TW) is assigned a unique “crypto key” known as the “Web3 Address”: /ipns/$publicKEY. Users record their data in TiddlyWiki, including the use of plugins for running new apps.
  1. Registration on UPlanet:
  • TWs created on “UPlanet” are registered on a specific location called UMap, represented by a 0.01° piece. These TWs are linked with a “Geospatial key,” forming a kind of blockchain that documents the information they collect.
  1. Nightly Blockchain Engine:
  • Every night, a “blockchain engine” retrieves and disseminates information among TWs through “data bridges.”
  • Information is shared with TWs in the same sector (local) and with specific tags from friends’ TWs located abroad.
  1. Data Trust Affinity:
  • TWs in sectors and beyond collect tiddlers based on the number of signatures they possess.
  • This principle is utilized to refine “data trust” from local to global, creating a decentralized information ecosystem.

For more detailed information, you can refer to the “UPlanet white paper draft.”

Why External URLs?

The External URL is employed to simplify the process of obtaining signatures for Tiddlers. Currently, users need to manually visit all sectors and friends’ TWs and copy the desired content into their own TWs. While functional as a proof of concept, this method is not highly practical.

Improvements with ExternalContent

To enhance usability, it would be beneficial to display “yesterday” data from both “local” and “abroad” sources.

Install in my TW


1. Test “HTML” mode

\procedure loadMyWiki()
\import [[$:/plugins/sq/ExternalContent/loadWikiActions]]
<$transclude
    $variable="loadWikiActions"
    wikiURL="/ipns/k51qzi5uqu5djfls7kbaf7vplpm0n2665x18nv9ol6xz83aohz2g33srvhx583"
    contentFilter=":[!is[system]tag[G1CopierYoutube]!tag[G1Voeu]!sort[modified]limit[2]]"
    deseralizer="text/html"
    />
\end

<$button actions=<<loadMyWiki>> > load content </$button>

Click & wait a little… Then :fireworks: IT WORKS !!
My TW gets populated with external tiddlers :wink:

Todo

In order to respect “signature consistency”,
it needs to add “Tags” from $:/config/NewTiddler/Tags (containing signature).
that plugin make that happens “on import”

You help is welcome to modify $:/plugins/sq/ExternalContent/filters/jsonfiltertiddlers.js to do it also on external imported Tiddlers…


2. Test “JSON” mode

I give a a try loading “json”…

\import [[$:/plugins/sq/ExternalContent/loadWikiActions]]
<$transclude
    $variable="loadWikiActions"
    wikiURL="/ipfs/QmcoQ3u6F6bKdyXm84kdLYxrKHXjmPizh2RTzjKX77a9mJ/tiddlers.json"
    contentFilter=""
    deseralizer="application/json"
    />
\end
<$button actions=<<loadMyWiki>> > load from JSON </$button>

couldn’t figure out what is correct deseralizer value to use ?


Conclusion and Reflection:

  • Overall positive experience testing ExternalContent.
  • Acknowledges the need for a few adjustments and an automatic process to distribute local and abroad URLs.
  • Likens TiddlyWiki to becoming a kind of “Windows95” for Web3.

This plugin is just surfacing the native HTTP handling capabilities of TW. To add tags to all imported tiddlers just modify the loadWikiActions procedure:

Use the one for the result you want. I have used the deseralizer on the “browse widget” set to deserializer=“application/json” to import it as JSON, and not “import the tiddlers within”.

  • The browse widget also allows you to set the default extensions eg .json, .tid, .txt, .csv

use none or “text/vnd.tiddlywiki” to import tiddlers within.

application/json is the correct deserializer to use to import JSON as tiddlers. However, the JSON needs to be properly formed as either an object or an array of objects, where each object has at least a title property.

See: https://github.com/Jermolene/TiddlyWiki5/blob/070327cb5708cfad104e2f325115203b499af631/core/modules/deserializers.js#L19-L41

An overview of deserializers available in the core:
https://tiddlywiki.com/#Deserializers

Of course if the json is non standard you can import as plain text and write your own parser even in wiki text :nerd_face: the Jason mangler plugin may help as well, use it to import and generate tiddlers even from csv.

Thanks for all your advice.

I must confess that I am a pretty bad javascript developer and even worse “TW programmer” :wink:

I am amazed about the rock solid and expandable environment it is and i wouldn’t like to screw too much things by tweaking too much


For now. I am not sure I can add any good code into

How would you copy to inserted tiddlers “tags” the value that is stored in $:/config/NewTiddler/Tags ?
This will be a good exercise to make my comprehension progress


UPlanet is producing a sub-key for every “G1Tag” channel anyone would like to populate.
TW is then producing every day a json export of the last 30 tiddlers containing this tag.

But when I try to import some, nothing happens (or I miss it, color change would help)
I don’t understand what is wrong? It is an array of objects : like this
May be loading time is too long (if data is not requested much, which is the case in that low inhabitants UPlanet)

Or isn’t it deserializer / deseralizer typo problem in button action?


There is a case, when tiddler with same title already exists, It could be same or different. I don’t know what would happen in UX.

For UX purpose, I wonder if those “inserted tiddlers” could be in different color (solarized actual css colors would be great) but i don’t know how to do it good and compatible with any template…


As you say, it could be possible to create export / import of any kind.
Even better, add some more…

Before reaching that “level” i still need to explore and understand actual filters…
One of my question is how could I export tiddllers as “static HTML file” using tiddlywiki from command line ?

For now i can just do it from Web browser (then ipfs add command on downloaded file)
export

So blockchain engine could make this json
becoming html and render “Canon_de_Pachelbel”

if i could know what ‘exporter’ i should put in that command…

tiddlywiki --load ${TW} --output /tmp --render '.' "${PLAYER}.g1tag.json" \
 'text/plain' '$:/core/templates/exporters/JsonFile' 'exportFilter' '[tag[G1Tag]]'

Sorry if it is like dummy questions for you.
I prompt this topic to ChatGPT … It does not seem to be right.


Gratitude for and and all of your help.

Continuing test and investigations…

@saqimtiaz

  • solved not working “json import”
    this was due to deser(i)alizer typo in example code…

  • I also put ‘:’ as leading character for

contentFilter=":[...

and made a “pull request” :wink:


About “tags” management,
This is my actual attempt to add $:/config/NewTiddler/Tags in newtags

<!-- get the tiddler at that position in the array from the JSON -->
<$let
    tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
    title={{{ [<tiddler>jsonget[title]] }}}
    newtags={{{ [<tiddler>jsonget[tags]] }}} $:/config/NewTiddler/Tags 
>
    
    <$action-log newtags=<newtags> />

    <!-- make sure we have a title for the tiddler and exclude system tiddlers -->
    <$list filter="[<title>!is[blank]]">
        <$action-setmultiplefields
            $fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp tags"
            $values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] <newtags>"
            $timestamp="no"
            />
    </$list>
</$let>

not good yet…


Thanks.
“tiddlywiki-starter-kit” looks great ! It is a full TW processing tunnel that produces a “blog” ?
I would need more details to understand how and what i can learn from ?

Try this:

<!-- get the tiddler at that position in the array from the JSON -->
<$let
    tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
    title={{{ [<tiddler>jsonget[title]] }}}
    newtags={{{ [<tiddler>jsonget[tags]] [{$:/config/NewTiddler/Tags}] :and[join[ ]] }}}  
>
    
    <$action-log newtags=<newtags> />

    <!-- make sure we have a title for the tiddler and exclude system tiddlers -->
    <$list filter="[<title>!is[blank]]">
        <$action-setmultiplefields
            $fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp tags"
            $values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] =[<newtags>]"
            $timestamp="no"
            />
    </$list>
</$let>

I appreciate the pull request correcting the typo. The leading : is actually not needed so if you are able to update that PR please do, otherwise I will merge an alternate fix when I get the opportunity.

:pray:
Thank you

@saqimtiaz
I applied your code guidance
made a this pull request.

but…
when I update $:/plugins/sq/ExternalContent/loadWikiActions in my TW
something broken !
button no acting , nor I can see log…

I was warned before modifying $:/ tiddler.
Is it because TW apply plugin version consistency check ?

I am on the road at the moment and cannot debug properly.

Try this replacing the entire contents of $:/plugins/sq/ExternalContent/loadWikiActions with the following:


\procedure startupConfigTitle() $:/config/sq/ExternalContent/load-on-startup
\procedure loadWikiActions(wikiURL,contentFilter,deserializer)
	<!-- actions invoked after fetching the remote data -->
	\procedure getWikiCallback()
		<!-- actions to turn the remote data into tiddlers -->
		\procedure importTiddlers()
			<$let
				passthroughFilter="[all[]]"
				defaultDeserializer="text/html"
				contentFilter={{{ [<contentFilter>!is[blank]else<passthroughFilter>] }}}
				deserializer={{{ [<deserializer>!is[blank]else<defaultDeserializer>] }}}
				tiddlerJSON={{{[<data>deserialize<deserializer>jsonfiltertiddlers<contentFilter>]}}}
				tiddlers={{{ [<tiddlerJSON>jsonindexes[]] :map[<tiddlerJSON>jsonget<currentTiddler>,[title]] :and[format:titlelist[]join[ ]] }}}
			>
			<$action-log $$filter="wikiURL contentFilter deserializer tiddlers tiddlerJSON"/>
				<!-- iterate over each position in the array -->
				<$list filter="[<tiddlerJSON>jsonindexes[]]" variable="index">
					<!-- get the tiddler at that position in the array from the JSON -->
<!-- get the tiddler at that position in the array from the JSON -->
<$let
    tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
    title={{{ [<tiddler>jsonget[title]] }}}
    newtags={{{ [<tiddler>jsonget[tags]] [{$:/config/NewTiddler/Tags}] :and[join[ ]] }}}  
>
    
    <$action-log newtags=<<newtags>> />

    <!-- make sure we have a title for the tiddler and exclude system tiddlers -->
    <$list filter="[<title>!is[blank]]">
        <$action-setmultiplefields
            $fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp =tags"
            $values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] =[<newtags>]"
            $timestamp="no"
            />
    </$list>
</$let>
				</$list>
			</$let>
		\end importTiddlers
		<!-- actions invoked if there is an error fetching the data -->
		\procedure failureHandler()
			<$action-log status="error fetching the wiki"/>
			<$action-setfield $tiddler={{{ [[$:/temp/http/error/]addsuffix<now [UTC]YYYY0MM0DD0hh0mm0ssXXX]>] }}} text={{{ [[There was an error fetching the wiki ]addsuffix<wikiURL>addsuffix<error>] }}} tags="$:/tags/Alert"/>
		\end failureHandler
		<$list filter="[<status>match[200]]" variable="null" emptyValue=<<failureHandler>> >
			<$action-log data=<<data>> status="succcess" />
			<<importTiddlers>>
		</$list>
	\end getWikiCallback

	<!-- fetch the remote data source-->
	\procedure getWikiActions()
		<$action-sendmessage
			$message="tm-http-request"
			method="GET"
			bind-status={{{ [[$:/temp/http/load-content/]addsuffix<wikiURL>] }}}
			oncompletion=<<getWikiCallback>>
			url=<<wikiURL>>
			var-wikiURL=<<wikiURL>>
			var-contentFilter=<<contentFilter>>
			var-deserializer=<<deserializer>>
			>
	\end getWikiActions

<!--  check if its a tiddlyhost URL and if so map it to the json file /tiddlers.json, also trim any trailing slashes  -->

<$let isTiddlyHost={{{ [<wikiURL>regexp[(?i)^https:\/\/\S+tiddlyhost.com]then[yes]else[no]] }}}
	wikiURL={{{ [<isTiddlyHost>match[yes]] :then[<wikiURL>!suffix[tiddlers.json]trim:suffix[/]addsuffix[/tiddlers.json]] :else[<wikiURL>] }}}
	deserializer={{{ [<isTiddlyHost>match[yes]then[application/json]else<deserializer>] }}}
	>
	<$action-log $$filter="isTiddlyHost wikiURL deserializer"/>
	<<getWikiActions>>
</$let>
\end loadWikiActions


<!--  ARE IMAGES IMPORTING CORRECTLY?   -->

If that does not work for you, you will need to wait until I have a chance to publish an updated version of the plugin that allows specifying optional additional actions to invoke on every incoming tiddler, and that likely wont happen for another 6 weeks or so.

1 Like

It does work ! :+1:
Great, i can inter-link TWs with “cross keys” loading.
… Have a nice trip on the road…

i would need and appreciate a “tw programming” lesson

meanwhile, i discovered that clue you dropped

… trying to adapt method to “is_volatile” field

… TW structure understanding evolves: TW $:/ looks like old Windows C:/ :wink:

Yes. In case anyone wants to see such a css solution in action:

Thank you for this great demo place !


You made me found all i needed for “local / abroad” tiddler status control.

After having imported

  1. $:/springer/volatile/css
  2. $:/core/ui/ViewTemplate
  3. externalTiddler_puzzle
  4. $:/publishFilter

css applyed immediately !
did i get enough tids to get “is_volatile” removed when saving ?

I wonder “what is for”, what is inside import.bundle ?
TW tweaking capabilities are really incredible

1 Like

@Springer
your change in publishFilter makes is_volatile=yes tiddlers not recorded.

In my user case, i would like to give an easy way to “keep it local”,

Actual action is to remove is_volatile field and save Tid.
Any guidance on how I could add a “tid control button” executing that “program” ?

To have your wiki keep a local copy of the imported tiddlers, just remove that $:/publishFilter