-
solved not working âjson importâ
this was due to deser(i)alizer typo in example code⊠-
I also put â:â as leading character for
contentFilter=":[...
About âtagsâ management,
This is my actual attempt to add $:/config/NewTiddler/Tags in newtags
<!-- get the tiddler at that position in the array from the JSON -->
<$let
tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
title={{{ [<tiddler>jsonget[title]] }}}
newtags={{{ [<tiddler>jsonget[tags]] }}} $:/config/NewTiddler/Tags
>
<$action-log newtags=<newtags> />
<!-- make sure we have a title for the tiddler and exclude system tiddlers -->
<$list filter="[<title>!is[blank]]">
<$action-setmultiplefields
$fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp tags"
$values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] <newtags>"
$timestamp="no"
/>
</$list>
</$let>
not good yetâŠ
Thanks.
âtiddlywiki-starter-kitâ looks great ! It is a full TW processing tunnel that produces a âblogâ ?
I would need more details to understand how and what i can learn from ?
Try this:
<!-- get the tiddler at that position in the array from the JSON -->
<$let
tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
title={{{ [<tiddler>jsonget[title]] }}}
newtags={{{ [<tiddler>jsonget[tags]] [{$:/config/NewTiddler/Tags}] :and[join[ ]] }}}
>
<$action-log newtags=<newtags> />
<!-- make sure we have a title for the tiddler and exclude system tiddlers -->
<$list filter="[<title>!is[blank]]">
<$action-setmultiplefields
$fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp tags"
$values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] =[<newtags>]"
$timestamp="no"
/>
</$list>
</$let>
I appreciate the pull request correcting the typo. The leading : is actually not needed so if you are able to update that PR please do, otherwise I will merge an alternate fix when I get the opportunity.
Thank you
@saqimtiaz
I applied your code guidance
made a this pull request.
âŠ
butâŠ
when I update $:/plugins/sq/ExternalContent/loadWikiActions in my TW
something broken !
button no acting , nor I can see logâŠ
I was warned before modifying $:/ tiddler.
Is it because TW apply plugin version consistency check ?
I am on the road at the moment and cannot debug properly.
Try this replacing the entire contents of $:/plugins/sq/ExternalContent/loadWikiActions with the following:
\procedure startupConfigTitle() $:/config/sq/ExternalContent/load-on-startup
\procedure loadWikiActions(wikiURL,contentFilter,deserializer)
<!-- actions invoked after fetching the remote data -->
\procedure getWikiCallback()
<!-- actions to turn the remote data into tiddlers -->
\procedure importTiddlers()
<$let
passthroughFilter="[all[]]"
defaultDeserializer="text/html"
contentFilter={{{ [<contentFilter>!is[blank]else<passthroughFilter>] }}}
deserializer={{{ [<deserializer>!is[blank]else<defaultDeserializer>] }}}
tiddlerJSON={{{[<data>deserialize<deserializer>jsonfiltertiddlers<contentFilter>]}}}
tiddlers={{{ [<tiddlerJSON>jsonindexes[]] :map[<tiddlerJSON>jsonget<currentTiddler>,[title]] :and[format:titlelist[]join[ ]] }}}
>
<$action-log $$filter="wikiURL contentFilter deserializer tiddlers tiddlerJSON"/>
<!-- iterate over each position in the array -->
<$list filter="[<tiddlerJSON>jsonindexes[]]" variable="index">
<!-- get the tiddler at that position in the array from the JSON -->
<!-- get the tiddler at that position in the array from the JSON -->
<$let
tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
title={{{ [<tiddler>jsonget[title]] }}}
newtags={{{ [<tiddler>jsonget[tags]] [{$:/config/NewTiddler/Tags}] :and[join[ ]] }}}
>
<$action-log newtags=<<newtags>> />
<!-- make sure we have a title for the tiddler and exclude system tiddlers -->
<$list filter="[<title>!is[blank]]">
<$action-setmultiplefields
$fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp =tags"
$values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] =[<newtags>]"
$timestamp="no"
/>
</$list>
</$let>
</$list>
</$let>
\end importTiddlers
<!-- actions invoked if there is an error fetching the data -->
\procedure failureHandler()
<$action-log status="error fetching the wiki"/>
<$action-setfield $tiddler={{{ [[$:/temp/http/error/]addsuffix<now [UTC]YYYY0MM0DD0hh0mm0ssXXX]>] }}} text={{{ [[There was an error fetching the wiki ]addsuffix<wikiURL>addsuffix<error>] }}} tags="$:/tags/Alert"/>
\end failureHandler
<$list filter="[<status>match[200]]" variable="null" emptyValue=<<failureHandler>> >
<$action-log data=<<data>> status="succcess" />
<<importTiddlers>>
</$list>
\end getWikiCallback
<!-- fetch the remote data source-->
\procedure getWikiActions()
<$action-sendmessage
$message="tm-http-request"
method="GET"
bind-status={{{ [[$:/temp/http/load-content/]addsuffix<wikiURL>] }}}
oncompletion=<<getWikiCallback>>
url=<<wikiURL>>
var-wikiURL=<<wikiURL>>
var-contentFilter=<<contentFilter>>
var-deserializer=<<deserializer>>
>
\end getWikiActions
<!-- check if its a tiddlyhost URL and if so map it to the json file /tiddlers.json, also trim any trailing slashes -->
<$let isTiddlyHost={{{ [<wikiURL>regexp[(?i)^https:\/\/\S+tiddlyhost.com]then[yes]else[no]] }}}
wikiURL={{{ [<isTiddlyHost>match[yes]] :then[<wikiURL>!suffix[tiddlers.json]trim:suffix[/]addsuffix[/tiddlers.json]] :else[<wikiURL>] }}}
deserializer={{{ [<isTiddlyHost>match[yes]then[application/json]else<deserializer>] }}}
>
<$action-log $$filter="isTiddlyHost wikiURL deserializer"/>
<<getWikiActions>>
</$let>
\end loadWikiActions
<!-- ARE IMAGES IMPORTING CORRECTLY? -->
If that does not work for you, you will need to wait until I have a chance to publish an updated version of the plugin that allows specifying optional additional actions to invoke on every incoming tiddler, and that likely wont happen for another 6 weeks or so.
It does work ! 
Great, i can inter-link TWs with âcross keysâ loading.
⊠Have a nice trip on the roadâŠ
i would need and appreciate a âtw programmingâ lesson
meanwhile, i discovered that clue you dropped
⊠trying to adapt method to âis_volatileâ field
⊠TW structure understanding evolves: TW $:/ looks like old Windows C:/ 
Yes. In case anyone wants to see such a css solution in action:
Thank you for this great demo place !
You made me found all i needed for âlocal / abroadâ tiddler status control.
After having imported
css applyed immediately !
did i get enough tids to get âis_volatileâ removed when saving ?
I wonder âwhat is forâ, what is inside import.bundle ?
TW tweaking capabilities are really incredible
@Springer
your change in publishFilter makes is_volatile=yes tiddlers not recorded.
In my user case, i would like to give an easy way to âkeep it localâ,
Actual action is to remove is_volatile field and save Tid.
Any guidance on how I could add a âtid control buttonâ executing that âprogramâ ?
To have your wiki keep a local copy of the imported tiddlers, just remove that $:/publishFilter
I prefer to keep this (no save) behavior,
so imported tiddlers are not recorded by default.
I would like to make a Tiddler âupper buttonâ to run a program removing âis_volatileâ field.
I try to âreverse engineerâ the extra buttons on your TW
To understand how it works and reproduceâŠ
Check out a quick version of this kind of solution here:
This is a button that will appear ONLY on tiddlers that have the is_volatile field, and clicking the button should toggle that field to ânoâ.
Iâm calling the button âAdoptExternalâ (hoping thatâs fairly clear as a metaphor), and Iâve given it the paper-clip icon, for now (since itâs a bit like attaching). Feel free to tweak as needed!
Iâm not removing the field itself (just setting it to ânoâ), in order to maximize your ability to troubleshoot, or toggle the field back to yes, as needed. (Ideally, the view toolbar button for this toggle would display differently depending on which way the toggle is currently set. But if youâre also using a css solution to make the external content obvious in the story river, then this omission isnât such a big deal.)
This button does not do anything to the externalTiddler field â again because you might have other reasons to continue tracking which tiddlers have come in from elsewhere.
@saqimtiaz â Eventually, it might be nice to have the load process populate the externalTiddler field not just with yes, but with the url of the source wiki. That way, if we load from multiple sources (and âadoptâ some of them over time), it will be easier to follow up, if we need to double-check the source for any reason.
Perfect !
I was missing AdoptExternal âpaper-clipâ button.
Now UX is simple, and easy
I need to address some more issues :
- When âLoadExternalâ action happens, it writes over local Tiddlers, and set back âis_volatileâ=âyesâ to already âclippedâ tiddlersâŠ
- And, how to make tiddlers with âUPPERCASEâ titles not fetched. (I am using this convention for user to create ânon replicateâ Tiddlers)
detect such condition,
if so, add field bypass=âyesâ ,
make import check !hasField[bypass]
\procedure importTiddlers()
<$let
passthroughFilter="[all[]]"
defaultDeserializer="text/html"
contentFilter={{{ [<contentFilter>!is[blank]else<passthroughFilter>] }}}
deserializer={{{ [<deserializer>!is[blank]else<defaultDeserializer>] }}}
tiddlerJSON={{{[<data>deserialize<deserializer>jsonfiltertiddlers<contentFilter>]}}}
tiddlers={{{ [<tiddlerJSON>jsonindexes[]] :map[<tiddlerJSON>jsonget<currentTiddler>,[title]] :and[format:titlelist[]join[ ]] }}}
>
<$action-log $$filter="wikiURL contentFilter deserializer tiddlers tiddlerJSON"/>
<!-- iterate over each position in the array -->
<$list filter="[<tiddlerJSON>jsonindexes[]]" variable="index">
<!-- get the tiddler at that position in the array from the JSON -->
<$let
tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
title={{{ [<tiddler>jsonget[title]] }}}
newtags={{{ [<tiddler>jsonget[tags]] [{$:/config/NewTiddler/Tags}] :and[join[ ]] }}}
>
<!-- Check if the title is in UPPERCASE -->
<$list filter="[<title>regexp[\p{Lu}]]">
<!-- If UPPERCASE, bypass importing -->
<$action-setfield $tiddler=[<tiddler>jsonindexes[]] $field="bypass" $value="yes"/>
<$action-log BYPASS <<title>> />
</$list>
<!-- Check if a tiddler with the same title already exists -->
<$list filter="[<title>get[<title>compare:title]is[0]]">
<!-- If it exists, bypass importing -->
<$action-setfield $tiddler=[<tiddler>jsonindexes[]] $field="bypass" $value="yes"/>
<$action-log Bypass <<title>> />
</$list>
<!-- make sure we have a title for the tiddler and exclude system tiddlers -->
<$list filter="[<title>!is[blank]!hasField[bypass]]">
<$action-log title=<<title>> />
<$action-setmultiplefields
$fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp =tags"
$values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] =[<newtags>]"
$timestamp="no"
/>
</$list>
</$let>
</$list>
</$let>
\end importTiddlers
But this code is not working ;(
Iâm sorry that my response here will not address your actual request.
Though what youâre envisioning is not hard to do, it makes more sense to problem-solve around why you feel the need to juggle capitalized and lowercase versions of titles (rather than to help you invest more effort in that very klugey direction)
Your experience (and mine, prompted by testing your kind of workflow) raises an important implementation question for me, which I think we need to flag @saqimtiaz on (even knowing saq probably canât put significant attention toward this anytime soon):
OBSERVATION: SAQâs CURRENT DEFAULT NEATLY HYBRIDIZES EXTERNAL & LOCAL FIELDS
If a tiddler is âadoptedâ (by toggling away from âyesâ in the is_volatile field), then a copy of it gets saved (as expected). But the next time external tiddlers are loaded, the solution as saq has set it up so far constructs a HYBRID result for any already-âadoptedâ tiddler:
- the incoming tiddler has a new
includeTimestamp - âyesâ is (re)written into the
is_volatilefield - external content overwrites any local field-values.
- HOWEVER⊠any field-value pairs that are unique to the local (pre-import) tiddler remain!
** At least, they remain for the current load sessionâŠ
Is this intentional? I think @saqimtiaz could have very good reasons for doing this: you could want to add local tags and comments / notes about a book (say), and still benefit from importing new information and/or corrections within the authoritative central record.
Then again, perhaps this hybrid behavior could just be an artifact of some json data remaining in un-erased limbo. I wonder this because if saq had wanted this behavior, it would have made sense to protect all local data not just for the duration of this browser session, but beyond (through subsequent saves)âŠ
RISK OF DATA LOSS
ALAS, the benefit of this hybrid load behavior, as currently implemented, is easily outweighed if one must go through and manually re-toggle (for âadoptionâ / saving) all the tiddlers that started the session with local field contents that of course you donât want to lose. But going through and doing this toggling is especially difficult, because we donât even seem to have any clear way within the local wiki of tracking â after the re-import â whether an imported tiddler has fields whose data is only from the local side, and which fields those are!
(A naive user will see, perhaps with satisfaction, that their locally generated field-data from a prior session is âstill thereâ after (re)importing a âconflictingâ tiddler. They may be unlikely to realize that this hybrid tiddler, with its locally-generated content intermixed with remote content, has reset itself to disappear, taking the distinctive local content â which cannot be restored from the source wiki â with it.)
EASY SOLUTION?
If the load process is already doing some savvy âdovetailâ work with local and incoming field values, then surely itâs possible to tweak the process so that whatever data that a wiki had, when its browser session started, all of that could survive the process of importing (âaddingâ) external content (with the possible exception of incoming field values overwriting local ones). That seems like an important norm â something users are entitled to expect, once this solution moves from experimental to mature. (To be clear, @saqimtiaz is doing some great pioneering work here, and of course itâs always true that details can fall into place only after the proof of concept!)
A simple solution would adjust the load-external-content process to check whether the local copy has the is_volatile field, and then to preserve that field value â even if other locally existing field-value pairs are overwritten. (So, the includeTimestamp is updated, confirming the most recent integration of content from the source/authority wiki, and all other fields behave as they now do.)
Optionally, too, perhaps @papiche (and others) would like the option to configure the import/load process so that it simply bypasses any existing local tiddlers with the same name.
(That is not my preference â I love the dovetail solution â perhaps enhanced by letting the local copy protect some specified set of field-value data, including the is_volatie field. Also, I think there will be great uses for having local shadows, and even other system tiddlers, overridden by incoming tiddlers with the same name. This sounds like a great way to sandbox complex and powerful stuff, knowing that those external tiddlers, by default, wonât save.)
MORE COMPLEX VISION:
Ideally, some additional fields could also be user-configured to be SAFE from overwriting; certain local field values (say, tags, or âlending statusâ for a biblio record) could be important to retain, even when the âmasterâ record has a conflicting version of that field.
Most ambitious (and probably not easy): figure out how to employ a âTWO-SIDED FILTERâ (!) Right now the filter for external content is entirely directed at the remote source, and is evaluated only as a filter within that remote wiki: âGo look at your tiddlers and send me all your tiddlers that meet this filter condition.â
But we could also imagine an filter defined/parsed on the LOCAL side, so that the INTERSECTION of the two filters determines the import. For example, "Look at these potential incoming tiddlers (in the filter weâve already been using), and ignore all EXCEPT those that also meet this condition: Say, we want to load those that fit the [is[missing]] filter condition here, or all those remote tiddlers whose titles are listed in the vocabulary field of a local tiddler, etc.
(A rather cumbersome version of that intersection could already be achieved by starting with a bluntly enumerative local filter: I use a local filter like [is[missing]!is[system]format:titlelist] to produce a titlelist, and use that list of titles to start my import filter, going on to add further general constraints that can be parsed on the remote end. But this is clearly a very brittle way of getting at the intersection of local and remote filter criteria!)
LAST THOUGHT:
Another configuration detail that could be helpful ⊠and I can probably figure this one out by myself soon⊠is to make the publishFilter check whether the external tiddler has been modified SINCE its importTimestamp. (Or perhaps any edit to the tiddler actually toggles the is_volatile field to ânoâ so that our css will reflect its new statusâŠ) That way, naive users get the benefit of keeping the local wiki âleanâ (without too much redundancy), while avoiding the risk of losing whatever fresh content they are adding to these remote-origin tiddlers. Maybe this reduces the demand for something like a toolbar button to âadoptâ tiddlers â which probably would be used most frequently on tiddlers that need have a future life with local edits.
@Springer I wont have any time to actively work on this before the end of April at the earliest but will be happy to follow the conversation until then as time allows.
A few quick thoughts that might prove helpful:
- The intent behind creating this plugin was actually just to provide an example of how tiddlers can be fetched and imported from a remote data source using just wikitext. It was meant to serve as an example others could adapt to their own usecases.
- The merging of fields from the local and remote tiddlers that is taking place is entirely an oversight, the original intent was that the remote tiddler should overwrite the local if a corresponding configuration option has been set.
- I am open to working on the plugin further to make it more configurable for common usage scenarios. However, highly specific needs that involve merging fields from remote and local tiddlers are likely going to be best served by emulating the code in the plugin to write a custom solution. Alternatively, one could imagine factoring out the actions that filter the tiddlers and save them as a separate tiddler that could be overwritten and customized by users to meet their specific criteria.
Thanks for the speedy reply, @saqimtiaz !
I do think itâs incumbent on folks (like me and papiche) who benefit from this work to go as far as we can in adapting it and learning from it.
Although my learning curve is steep here, Iâm just far-enough along to have a sense of the structure of a solution that may work for a wide range of users (and a cringey feeling about some workarounds that I think will lead to more trouble).
Much gratitude to you for this proof-of-concept, and for being at least potentially within earshot â even while youâre juggling other tasks. 
Some more info that may help in adapting the code in the plugin:
This the part of the code that extracts each individual tiddler from the incoming data and then saves it: https://github.com/saqimtiaz/tw5-plugins-sandbox/blob/e9ebad0708d6c15eb0b8974a3d9328e3b0c0e9af/plugins/sq/ExternalContent/load-wiki-actions.tid#L20-L35
It is the usage of $action-setmultiplefields here that led to the inadvertent merging of local and remote fields, for an overwrite it should be preceded here by deleting the local tiddler for an overwrite. Note also that rather than manipulating the $action-setmultiplefields in that code which can get hairy, it can also be followed up with $action-setfield to further add/manipulate fields
Thank you for your detailed thoughts.
It is true, that managing field update procedure is tricky.
This is where UPlanet Geokeys and âemailâ tags signatures and crypto coins are useful.
the more a Tiddler gets âemail tagsâ, the more it is considered trustful.
Each TW (associated to an email) benefits from a relative âWeb Of Trustâ.
And is included in a SECTOR that is get all Tiddlers with > 3 signatures.
it receives âExternalsâ from âSectorTWâ
A Tiddler can be shared â1 by 1â in different sectors.
When a 4th signature is collected, corresponding SECTOR will get it, so this Tid is spread to all âin area TWsâ. Then all signers get paid by the last one (Zen is the crypto coin).
In this scenario,
- an external Tiddler overwrites local (with all fields)
only if âexternal_signatures_nbâ > "local_signatures_nb
and if TW wallet have enough to pay "external_signatures_nb âZenâ(0.1 Ä1)
For now, adding [days[-1]] to filters can do the trick, so UX works in a âday by dayâ usage.
It will have more complex fusion algorithm later.
But using âCoinâ and âWoTâ will ease things
About UPPERCASE title, as TW is engaged in such an automatic âblockchain processâ.
This was introduced to give user a way to avoid treatments. It is also matching ânet ethicâ as it is considered as SHOUTING I consider it not so klugey.
I am glad I received already so useful help, I want to thank you.
I am sad my âTW programming skillsâ are still badâŠ
So I opened the side project to make assistant collecting TiddlyWiki PDF documentation to create a âprivateGPTâ for TW. This research and service is also part of âUPlanet REGIONSâ data handlers (more than 144 signatures would be difficult to deal with TW alone).
@Springer I am not yet trying to follow this topic in detail however reading your account of the difficulties you are trying to address, they seem to be related to research I have done for tiddler import etcâŠ
It seems to me if you can have local tiddlers, that are about to be overwritten by any external source, be turned into shadow tiddlers, by packing into a plugin first, they remain safe.
- similarly encoding tiddlers into a json tiddler
You are then free to let any import or external tiddlers to replace the wikis tiddlers because you have a reference copy.
- with a reference copy you can apply additional processing and comparisons etcâŠ
Of course we do not necessarily know the incoming tiddler titles in advance, so perhaps a per tiddler method to push a current copy of each tiddler into a âshadow copyâ before being replaced.
- This is definitely possible
- such plugins can be disabled, exported and deleted
- the same tiddler can be in multiple plugins with one copy winning.
- we are used to thinking of plugins as read only, but they are not, they are just tiddlers, if there is no modules or javascript in a plugin, you donât even need to reload the wikis.
In a similar way to shadow tiddlers we can easily create other types of virtual tiddlers related to a specific tiddler under a prefix, in a json or plugin.
- we do not have too many mature tools for this but we can make them.
Visions?
- Perhaps if all imports and external tiddlers existed in a kind of tiddler sand box, one we could push and pull tiddlers or their fields from/to, that we could nevertheless access, some of these issues would not exist. Over here Using tiddlywiki documentation when contained in a plugin - #16 by TW_Tones related conversation.
Enjoy the journey
This first request is easy (given the helpful pointer by @saqimtiaz in above reply), and itâs also something I expect to employ in my own variants (taking advantage of the fortuitous field-merge behavior noted above). Iâve just crudely modified saqâs version so that is_volatile field is set to âyesâ ONLY if the local tiddler does not already have a value in that field.
Find the modified shadow here:
Iâm not confident enough with regexp to wade into the details myself, but even from a distance I think you would need to specify more things:
- Do you mean tiddlers whose titles have ONLY capitals (and spaces), but no non-letters?
- Or, do you mean tiddlers whose titles have NO lowercase letters (but can have whatever other
$â Âź@đĄĐ§âstuff people can throw into a title?
If you need every potential tiddler title to have two variations â one quiet title and one SHOUTING TITLE â then Iâm just not seeing any simple solution that handles titles that arenât based on letter-strings (which happens easily when someone makes a journal tiddler with title format like 2024-03-12 for example).
On top of that, once youâve got multiple languages in play, youâll have some languages where capitalization isnât even a thing at all⊠ïŒć°ăăăç„ăăȘăăȘâïŒ
Iâm sure we have regexp folks here (like @Scott_Sauyet) who can give you a clear way to filter out ALLCAPS titles (with or without allowing for spaces, numerals, etc.), but Iâm still scratching my head about this method. It seems to me (if I understand a fraction of your ambitions) that youâre aiming for a pretty cosmopolitan (hence international-friendly) project, no? Or is your user community made of tech folks who wonât mind being limited to a strict roman-letter titling convention in order to set up this lowercase/UPPERCASE contrast?