I prefer to keep this (no save) behavior,
so imported tiddlers are not recorded by default.
I would like to make a Tiddler “upper button” to run a program removing “is_volatile” field.
I try to “reverse engineer” the extra buttons on your TW
To understand how it works and reproduce…
Check out a quick version of this kind of solution here:
This is a button that will appear ONLY on tiddlers that have the is_volatile field, and clicking the button should toggle that field to “no”.
I’m calling the button “AdoptExternal” (hoping that’s fairly clear as a metaphor), and I’ve given it the paper-clip icon, for now (since it’s a bit like attaching). Feel free to tweak as needed!
I’m not removing the field itself (just setting it to “no”), in order to maximize your ability to troubleshoot, or toggle the field back to yes, as needed. (Ideally, the view toolbar button for this toggle would display differently depending on which way the toggle is currently set. But if you’re also using a css solution to make the external content obvious in the story river, then this omission isn’t such a big deal.)
This button does not do anything to the externalTiddler field — again because you might have other reasons to continue tracking which tiddlers have come in from elsewhere.
@saqimtiaz — Eventually, it might be nice to have the load process populate the externalTiddler field not just with yes, but with the url of the source wiki. That way, if we load from multiple sources (and “adopt” some of them over time), it will be easier to follow up, if we need to double-check the source for any reason.
Perfect !
I was missing AdoptExternal “paper-clip” button.
Now UX is simple, and easy 
I need to address some more issues :
detect such condition,
if so, add field bypass=“yes” ,
make import check !hasField[bypass]
\procedure importTiddlers()
<$let
passthroughFilter="[all[]]"
defaultDeserializer="text/html"
contentFilter={{{ [<contentFilter>!is[blank]else<passthroughFilter>] }}}
deserializer={{{ [<deserializer>!is[blank]else<defaultDeserializer>] }}}
tiddlerJSON={{{[<data>deserialize<deserializer>jsonfiltertiddlers<contentFilter>]}}}
tiddlers={{{ [<tiddlerJSON>jsonindexes[]] :map[<tiddlerJSON>jsonget<currentTiddler>,[title]] :and[format:titlelist[]join[ ]] }}}
>
<$action-log $$filter="wikiURL contentFilter deserializer tiddlers tiddlerJSON"/>
<!-- iterate over each position in the array -->
<$list filter="[<tiddlerJSON>jsonindexes[]]" variable="index">
<!-- get the tiddler at that position in the array from the JSON -->
<$let
tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
title={{{ [<tiddler>jsonget[title]] }}}
newtags={{{ [<tiddler>jsonget[tags]] [{$:/config/NewTiddler/Tags}] :and[join[ ]] }}}
>
<!-- Check if the title is in UPPERCASE -->
<$list filter="[<title>regexp[\p{Lu}]]">
<!-- If UPPERCASE, bypass importing -->
<$action-setfield $tiddler=[<tiddler>jsonindexes[]] $field="bypass" $value="yes"/>
<$action-log BYPASS <<title>> />
</$list>
<!-- Check if a tiddler with the same title already exists -->
<$list filter="[<title>get[<title>compare:title]is[0]]">
<!-- If it exists, bypass importing -->
<$action-setfield $tiddler=[<tiddler>jsonindexes[]] $field="bypass" $value="yes"/>
<$action-log Bypass <<title>> />
</$list>
<!-- make sure we have a title for the tiddler and exclude system tiddlers -->
<$list filter="[<title>!is[blank]!hasField[bypass]]">
<$action-log title=<<title>> />
<$action-setmultiplefields
$fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp =tags"
$values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] =[<newtags>]"
$timestamp="no"
/>
</$list>
</$let>
</$list>
</$let>
\end importTiddlers
But this code is not working ;(
I’m sorry that my response here will not address your actual request.
Though what you’re envisioning is not hard to do, it makes more sense to problem-solve around why you feel the need to juggle capitalized and lowercase versions of titles (rather than to help you invest more effort in that very klugey direction)
Your experience (and mine, prompted by testing your kind of workflow) raises an important implementation question for me, which I think we need to flag @saqimtiaz on (even knowing saq probably can’t put significant attention toward this anytime soon):
OBSERVATION: SAQ’s CURRENT DEFAULT NEATLY HYBRIDIZES EXTERNAL & LOCAL FIELDS
If a tiddler is “adopted” (by toggling away from “yes” in the is_volatile field), then a copy of it gets saved (as expected). But the next time external tiddlers are loaded, the solution as saq has set it up so far constructs a HYBRID result for any already-“adopted” tiddler:
includeTimestamp
is_volatile fieldIs this intentional? I think @saqimtiaz could have very good reasons for doing this: you could want to add local tags and comments / notes about a book (say), and still benefit from importing new information and/or corrections within the authoritative central record.
Then again, perhaps this hybrid behavior could just be an artifact of some json data remaining in un-erased limbo. I wonder this because if saq had wanted this behavior, it would have made sense to protect all local data not just for the duration of this browser session, but beyond (through subsequent saves)…
RISK OF DATA LOSS
ALAS, the benefit of this hybrid load behavior, as currently implemented, is easily outweighed if one must go through and manually re-toggle (for “adoption” / saving) all the tiddlers that started the session with local field contents that of course you don’t want to lose. But going through and doing this toggling is especially difficult, because we don’t even seem to have any clear way within the local wiki of tracking — after the re-import — whether an imported tiddler has fields whose data is only from the local side, and which fields those are!
(A naive user will see, perhaps with satisfaction, that their locally generated field-data from a prior session is “still there” after (re)importing a “conflicting” tiddler. They may be unlikely to realize that this hybrid tiddler, with its locally-generated content intermixed with remote content, has reset itself to disappear, taking the distinctive local content — which cannot be restored from the source wiki — with it.)
EASY SOLUTION?
If the load process is already doing some savvy “dovetail” work with local and incoming field values, then surely it’s possible to tweak the process so that whatever data that a wiki had, when its browser session started, all of that could survive the process of importing (“adding”) external content (with the possible exception of incoming field values overwriting local ones). That seems like an important norm — something users are entitled to expect, once this solution moves from experimental to mature. (To be clear, @saqimtiaz is doing some great pioneering work here, and of course it’s always true that details can fall into place only after the proof of concept!)
A simple solution would adjust the load-external-content process to check whether the local copy has the is_volatile field, and then to preserve that field value – even if other locally existing field-value pairs are overwritten. (So, the includeTimestamp is updated, confirming the most recent integration of content from the source/authority wiki, and all other fields behave as they now do.)
Optionally, too, perhaps @papiche (and others) would like the option to configure the import/load process so that it simply bypasses any existing local tiddlers with the same name.
(That is not my preference — I love the dovetail solution — perhaps enhanced by letting the local copy protect some specified set of field-value data, including the is_volatie field. Also, I think there will be great uses for having local shadows, and even other system tiddlers, overridden by incoming tiddlers with the same name. This sounds like a great way to sandbox complex and powerful stuff, knowing that those external tiddlers, by default, won’t save.)
MORE COMPLEX VISION:
Ideally, some additional fields could also be user-configured to be SAFE from overwriting; certain local field values (say, tags, or “lending status” for a biblio record) could be important to retain, even when the “master” record has a conflicting version of that field.
Most ambitious (and probably not easy): figure out how to employ a “TWO-SIDED FILTER” (!) Right now the filter for external content is entirely directed at the remote source, and is evaluated only as a filter within that remote wiki: “Go look at your tiddlers and send me all your tiddlers that meet this filter condition.”
But we could also imagine an filter defined/parsed on the LOCAL side, so that the INTERSECTION of the two filters determines the import. For example, "Look at these potential incoming tiddlers (in the filter we’ve already been using), and ignore all EXCEPT those that also meet this condition: Say, we want to load those that fit the [is[missing]] filter condition here, or all those remote tiddlers whose titles are listed in the vocabulary field of a local tiddler, etc.
(A rather cumbersome version of that intersection could already be achieved by starting with a bluntly enumerative local filter: I use a local filter like [is[missing]!is[system]format:titlelist] to produce a titlelist, and use that list of titles to start my import filter, going on to add further general constraints that can be parsed on the remote end. But this is clearly a very brittle way of getting at the intersection of local and remote filter criteria!)
LAST THOUGHT:
Another configuration detail that could be helpful … and I can probably figure this one out by myself soon… is to make the publishFilter check whether the external tiddler has been modified SINCE its importTimestamp. (Or perhaps any edit to the tiddler actually toggles the is_volatile field to “no” so that our css will reflect its new status…) That way, naive users get the benefit of keeping the local wiki “lean” (without too much redundancy), while avoiding the risk of losing whatever fresh content they are adding to these remote-origin tiddlers. Maybe this reduces the demand for something like a toolbar button to “adopt” tiddlers — which probably would be used most frequently on tiddlers that need have a future life with local edits.
@Springer I wont have any time to actively work on this before the end of April at the earliest but will be happy to follow the conversation until then as time allows.
A few quick thoughts that might prove helpful:
Thanks for the speedy reply, @saqimtiaz !
I do think it’s incumbent on folks (like me and papiche) who benefit from this work to go as far as we can in adapting it and learning from it.
Although my learning curve is steep here, I’m just far-enough along to have a sense of the structure of a solution that may work for a wide range of users (and a cringey feeling about some workarounds that I think will lead to more trouble).
Much gratitude to you for this proof-of-concept, and for being at least potentially within earshot — even while you’re juggling other tasks. 
Some more info that may help in adapting the code in the plugin:
This the part of the code that extracts each individual tiddler from the incoming data and then saves it: https://github.com/saqimtiaz/tw5-plugins-sandbox/blob/e9ebad0708d6c15eb0b8974a3d9328e3b0c0e9af/plugins/sq/ExternalContent/load-wiki-actions.tid#L20-L35
It is the usage of $action-setmultiplefields here that led to the inadvertent merging of local and remote fields, for an overwrite it should be preceded here by deleting the local tiddler for an overwrite. Note also that rather than manipulating the $action-setmultiplefields in that code which can get hairy, it can also be followed up with $action-setfield to further add/manipulate fields
Thank you for your detailed thoughts.
It is true, that managing field update procedure is tricky.
This is where UPlanet Geokeys and “email” tags signatures and crypto coins are useful.
the more a Tiddler gets “email tags”, the more it is considered trustful.
Each TW (associated to an email) benefits from a relative “Web Of Trust”.
And is included in a SECTOR that is get all Tiddlers with > 3 signatures.
A Tiddler can be shared “1 by 1” in different sectors.
When a 4th signature is collected, corresponding SECTOR will get it, so this Tid is spread to all “in area TWs”. Then all signers get paid by the last one (Zen is the crypto coin).
In this scenario,
For now, adding [days[-1]] to filters can do the trick, so UX works in a “day by day” usage.
It will have more complex fusion algorithm later.
But using “Coin” and “WoT” will ease things
About UPPERCASE title, as TW is engaged in such an automatic “blockchain process”.
This was introduced to give user a way to avoid treatments. It is also matching “net ethic” as it is considered as SHOUTING I consider it not so klugey.
I am glad I received already so useful help, I want to thank you.
I am sad my “TW programming skills” are still bad…
So I opened the side project to make assistant collecting TiddlyWiki PDF documentation to create a “privateGPT” for TW. This research and service is also part of “UPlanet REGIONS” data handlers (more than 144 signatures would be difficult to deal with TW alone).
@Springer I am not yet trying to follow this topic in detail however reading your account of the difficulties you are trying to address, they seem to be related to research I have done for tiddler import etc…
It seems to me if you can have local tiddlers, that are about to be overwritten by any external source, be turned into shadow tiddlers, by packing into a plugin first, they remain safe.
You are then free to let any import or external tiddlers to replace the wikis tiddlers because you have a reference copy.
Of course we do not necessarily know the incoming tiddler titles in advance, so perhaps a per tiddler method to push a current copy of each tiddler into a “shadow copy” before being replaced.
In a similar way to shadow tiddlers we can easily create other types of virtual tiddlers related to a specific tiddler under a prefix, in a json or plugin.
Visions?
Enjoy the journey
This first request is easy (given the helpful pointer by @saqimtiaz in above reply), and it’s also something I expect to employ in my own variants (taking advantage of the fortuitous field-merge behavior noted above). I’ve just crudely modified saq’s version so that is_volatile field is set to “yes” ONLY if the local tiddler does not already have a value in that field.
Find the modified shadow here:
I’m not confident enough with regexp to wade into the details myself, but even from a distance I think you would need to specify more things:
$†®@𐡀Ч∈ stuff people can throw into a title?If you need every potential tiddler title to have two variations — one quiet title and one SHOUTING TITLE — then I’m just not seeing any simple solution that handles titles that aren’t based on letter-strings (which happens easily when someone makes a journal tiddler with title format like 2024-03-12 for example).
On top of that, once you’ve got multiple languages in play, you’ll have some languages where capitalization isn’t even a thing at all… （困るかも知らないな—）
I’m sure we have regexp folks here (like @Scott_Sauyet) who can give you a clear way to filter out ALLCAPS titles (with or without allowing for spaces, numerals, etc.), but I’m still scratching my head about this method. It seems to me (if I understand a fraction of your ambitions) that you’re aiming for a pretty cosmopolitan (hence international-friendly) project, no? Or is your user community made of tech folks who won’t mind being limited to a strict roman-letter titling convention in order to set up this lowercase/UPPERCASE contrast?
I haven’t been following this thread (and don’t have time to catch up right now.) I saw this and laughed out loud. I’m no regex expert: an advanced beginner at best. But then, I guess I can do this.
The pattern ^[^a-z]*$ will capture all titles that contain no lower-case letters:
So this:
<$let no-lowercase="^[^a-z]*$">
<<list-links """
[[HAPPY BIRTHDAY MOM! (2024-03-12)]]
[[Happy Birthday Mom! (2024-03-12)]]
[[2024-03-12]]
[[FOO AND A little BAR]]
[[FOO AND A LITTLE BAR]]
[[e.e. cummings]]
[[LBJ TOOK THE IRT DOWN TO 4TH ST, USA]]
[[When he got there, what did he see?]]
[[THE YOUTH OF AMERICA ON LSD]]
[[12345!]]
+[regexp:title<no-lowercase>]
""">>
</$let>
Will select these values:
HAPPY BIRTHDAY MOM! (2024-03-12)
2024-03-12
FOO AND A LITTLE BAR
LBJ TOOK THE IRT DOWN TO 4TH ST, USA
THE YOUTH OF AMERICA ON LSD
12345!
(And yes, it’s Mom’s 79th birthday today.)
If we wanted to also ensure that the title contained at least one uppercase letters a regex wizard could probably come up with some self-contained pattern, but I would just use another pass, like this:
<$let no-lowercase="^[^a-z]*$" some-uppercase="[A-Z]+">
<<list-links """
[[HAPPY BIRTHDAY MOM! (2024-03-12)]]
[[Happy Birthday Mom! (2024-03-12)]]
[[2024-03-12]]
[[FOO AND A little BAR]]
[[FOO AND A LITTLE BAR]]
[[e.e. cummings]]
[[LBJ TOOK THE IRT DOWN TO 4TH ST, USA]]
[[When he got there, what did he see?]]
[[THE YOUTH OF AMERICA ON LSD]]
[[12345!]]
+[regexp:title<no-lowercase>]
+[regexp:title<some-uppercase>]
""">>
</$let>
yielding
HAPPY BIRTHDAY MOM! (2024-03-12)
FOO AND A LITTLE BAR
LBJ TOOK THE IRT DOWN TO 4TH ST, USA
THE YOUTH OF AMERICA ON LSD
(note that we now skip 2024-03-12 and 12345!
Wow, that’s fantastic! It may not mean anything to her, but a friendly wave from central CT!
I’m not ignoring your suggestion, but I admit that orienting to plugin-packaging operations feels like a learning-curve I’m not ready for!
But it’s cool to know that there are ways to play with shadow tiddlers that would allow for a safer workflow when importing. I know I’ll want to come back to this, and I appreciate your sketching out this kind of solution.
Nce work @Scott_Sauyet
However;
I expect we can use functions and existing case “filter operators” to compare the input with the modified case. I will explore this and return if I have some nice examples.
I have no idea how much effort is required for true mastery, but it’s not too difficult to gain basic competence.
A small additional instruction how they relate to tiddlywiki would may also help.
Hello everyone
This is a beautiful day this morning.
I am getting all your messages and feel happy about having a new “wikitext” lesson. Btw, is it the name of it? So, it seems all “data twists” are possible but difficult to apply…
I made a “code fusion” adding back newtags={{{ [<tiddler>jsonget[tags]] [{$:/config/NewTiddler/Tags}] :and[join[ ]] }}}" (
<$let
tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
title={{{ [<tiddler>jsonget[title]] }}}
newtags={{{ [<tiddler>jsonget[tags]] [{$:/config/NewTiddler/Tags}] :and[join[ ]] }}}
>
<$action-log title=<<title>> />
<!-- make sure we have a title for the tiddler and exclude system tiddlers -->
<$list filter="[<title>!is[blank]]">
<$action-setmultiplefields
$fields="[<tiddler>jsonindexes[]] -is_volatile externalTiddler includeTimestamp =tags"
$values="[<tiddler>jsonindexes[]] -is_volatile :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] =[<newtags>]"
$timestamp="no"
/>
</$list>
<!-- The following list section amends shadow behavior, to handle the is_volatile field separately (which I've removed from the above set-multiple sequence). I've left the section below as a "set multiple" action so that more complex solutions can be developed. -->
<$list filter="[<title>!is[blank]!has[is_volatile]]">
<$action-setmultiplefields
$fields="is_volatile"
$values="=yes"
$timestamp="no"
/>
</$list>
</$let>
In my understanding,
it applies 2 consecutive <$list filter="...">.... </$list>
and make <$action-setmultiplefields
we should also include !is[system] as default filter
-is_volatile (does it means keep this field unchanged?),!has[is_volatile]] and set "is_volatile"="yes"
But computation happens over the list of “external tiddlers”.
I can’t see where a comparison with “inside tiddler” can be done, because “is_volatile” should be set to “no” if a same title match happens between “external” and “inside” tiddlers.
You’re right, i didn’t really be aware of “capitalisation” is not a worldwide thing.
We’ll have to find another “no_sharing please” expression. It can happen when “asymmetric crypto” (like libsodium or PGP) will be included in TW.
Waiting this layer coming,…
It is why sign is done by copying an “email style” in tiddlers tags.
UPlanet “zero” is highly hackable and players can break it
But as it spread “confidence rings” around each TW, it should not be a problem, just a way to show how “crypto” is acting and Web3 is a wonderful place if we settle in by ourself instead of waiting to be “put it” by force or treachery…
It gives a “UniqID” to anything digital can be linked to.
So the “rules of the digital world” coming will soon be glued to any.
Our collective is wishing to open an “Open World Ledger” where anyone can register its property as “common goods” for the group registered in a “Proof of Humanity” chain (already existing in PGP and G1 community). All observing MadeInZion chart and taking care of the “people, stuff and places” being part of that ledger.
Great (and happy birthdays moms !)
The next question is how this regexp can join the filtering…
<$list filter="[<title>!is[blank][^[^a-z]*$]]"> ??
I am trying to debug through “my browser console”, but any mistake in code just make a shut up. I need to find an “better way” to code and debug wikitext…
proposal is
<$list filter="[<title>!is[blank]has^[^a-z]*$]">
Finally, debugging wikitext can be challenging. Does it exists a dedicated text editor with syntax highlighting for TiddlyWiki or browser extensions that enhance the debugging experience?
$:/tags/load-external-content
I realize that $:/tags/load-external-content condition does not load well in “ipfs condition”
At TW start browser is loading plugin this way.
https://ipfs.copylaradio.com/procedure%20loadMyWiki()/import%20[[$:/plugins/sq/ExternalContent/loadWikiActions]]%3C$transclude%20%20%20%20$variable=%22loadWikiActions%22%20%20%20%20wikiURL=%7B%7B!!wikiURL%7D%7D%20%20%20%20contentFilter=%7B%7B!!filter%7D%7D%20%20%20%20deserializer=%22text/html%22%20%20%20%20/%3E/end%3Cdetails%20open=%22yes%22%3E%20%3Csummary%3ESUPPORT@QO-OP.COM%20TW:%3C/summary%3E%20%20%3Cdiv%3E%3C$edit%20size=60%20field=%22wikiURL%22/%3E%20%E2%AC%85%EF%B8%8E%20%20TW%20%3Cbr%3E%3C$edit%20size=50%20field=%22filter%22/%3E%20%E2%AC%85%EF%B8%8E%20Filtration%3C%3Chspace%3E%3E%3C$button%20actions=%3C%3CloadMyWiki%3E%3E%20%3E%20SUPPORT@QO-OP.COM%20TW%20%3C/$button%3E%20%20%3C/div%3E%20%3C/details%3E[[EXTERNAL|EXTERNAL]]
a “relative/absolute” linking problem happens somewhere
Just a quick reply:
ChatGPT does not know how to say “I’m not sure” or “this is confusing to me” or “I can’t yet grok TiddlyWiki” — and has been rewarded its whole life for such cocksureness, all of which reflects quite badly on its parents.
Documentation for regexp is here: https://tiddlywiki.com/#regexp%20Operator
Since the pattern @Scott_Sauyet mentions has brackets in it, you’ll need another variable (beyond title which is already defined) to hold the regexp, so the relevant section of code (to exclude allcaps titles) would go something like
<$let
tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
title={{{ [<tiddler>jsonget[title]] }}}
allcaps="^[^a-z]*$"
>
<!-- make sure we have a title for the tiddler and exclude system tiddlers -->
<$list filter="[<title>!is[blank]] -[<title>regexp<allcaps>]">
Wooo.
Thanks for all that brainstorming !!
concentration and 3 coffees later … I just finished including all that magic into that pull request !!
NB: There is still the error on “automatic load-external-content” when TW loads from an IPFS gateway ? But feeling great, and going for a little walk, need to move my feet
Maybe this deserves a different thread? I know absolutely nothing about IPFS gateways, and some other folks may have tuned out of this thread, assuming it has just been refining the basic solution that @saqimtiaz shared.