Yes. In case anyone wants to see such a css solution in action:
Thank you for this great demo place !
You made me found all i needed for “local / abroad” tiddler status control.
After having imported
css applyed immediately !
did i get enough tids to get “is_volatile” removed when saving ?
I wonder “what is for”, what is inside import.bundle ?
TW tweaking capabilities are really incredible
1 Like
@Springer
your change in publishFilter makes is_volatile=yes tiddlers not recorded.
In my user case, i would like to give an easy way to “keep it local”,
Actual action is to remove is_volatile field and save Tid.
Any guidance on how I could add a “tid control button” executing that “program” ?
To have your wiki keep a local copy of the imported tiddlers, just remove that $:/publishFilter
I prefer to keep this (no save) behavior,
so imported tiddlers are not recorded by default.
I would like to make a Tiddler “upper button” to run a program removing “is_volatile” field.
I try to “reverse engineer” the extra buttons on your TW
To understand how it works and reproduce…
Check out a quick version of this kind of solution here:
This is a button that will appear ONLY on tiddlers that have the is_volatile field, and clicking the button should toggle that field to “no”.
I’m calling the button “AdoptExternal” (hoping that’s fairly clear as a metaphor), and I’ve given it the paper-clip icon, for now (since it’s a bit like attaching). Feel free to tweak as needed!
I’m not removing the field itself (just setting it to “no”), in order to maximize your ability to troubleshoot, or toggle the field back to yes, as needed. (Ideally, the view toolbar button for this toggle would display differently depending on which way the toggle is currently set. But if you’re also using a css solution to make the external content obvious in the story river, then this omission isn’t such a big deal.)
This button does not do anything to the externalTiddler field — again because you might have other reasons to continue tracking which tiddlers have come in from elsewhere.
@saqimtiaz — Eventually, it might be nice to have the load process populate the externalTiddler field not just with yes, but with the url of the source wiki. That way, if we load from multiple sources (and “adopt” some of them over time), it will be easier to follow up, if we need to double-check the source for any reason.
1 Like
Perfect !
I was missing AdoptExternal “paper-clip” button.
Now UX is simple, and easy 
I need to address some more issues :
- When “LoadExternal” action happens, it writes over local Tiddlers, and set back “is_volatile”=“yes” to already “clipped” tiddlers…
- And, how to make tiddlers with “UPPERCASE” titles not fetched. (I am using this convention for user to create “non replicate” Tiddlers)
detect such condition,
if so, add field bypass=“yes” ,
make import check !hasField[bypass]
\procedure importTiddlers()
<$let
passthroughFilter="[all[]]"
defaultDeserializer="text/html"
contentFilter={{{ [<contentFilter>!is[blank]else<passthroughFilter>] }}}
deserializer={{{ [<deserializer>!is[blank]else<defaultDeserializer>] }}}
tiddlerJSON={{{[<data>deserialize<deserializer>jsonfiltertiddlers<contentFilter>]}}}
tiddlers={{{ [<tiddlerJSON>jsonindexes[]] :map[<tiddlerJSON>jsonget<currentTiddler>,[title]] :and[format:titlelist[]join[ ]] }}}
>
<$action-log $$filter="wikiURL contentFilter deserializer tiddlers tiddlerJSON"/>
<!-- iterate over each position in the array -->
<$list filter="[<tiddlerJSON>jsonindexes[]]" variable="index">
<!-- get the tiddler at that position in the array from the JSON -->
<$let
tiddler={{{ [<tiddlerJSON>jsonextract<index>] }}}
title={{{ [<tiddler>jsonget[title]] }}}
newtags={{{ [<tiddler>jsonget[tags]] [{$:/config/NewTiddler/Tags}] :and[join[ ]] }}}
>
<!-- Check if the title is in UPPERCASE -->
<$list filter="[<title>regexp[\p{Lu}]]">
<!-- If UPPERCASE, bypass importing -->
<$action-setfield $tiddler=[<tiddler>jsonindexes[]] $field="bypass" $value="yes"/>
<$action-log BYPASS <<title>> />
</$list>
<!-- Check if a tiddler with the same title already exists -->
<$list filter="[<title>get[<title>compare:title]is[0]]">
<!-- If it exists, bypass importing -->
<$action-setfield $tiddler=[<tiddler>jsonindexes[]] $field="bypass" $value="yes"/>
<$action-log Bypass <<title>> />
</$list>
<!-- make sure we have a title for the tiddler and exclude system tiddlers -->
<$list filter="[<title>!is[blank]!hasField[bypass]]">
<$action-log title=<<title>> />
<$action-setmultiplefields
$fields="[<tiddler>jsonindexes[]] externalTiddler is_volatile includeTimestamp =tags"
$values="[<tiddler>jsonindexes[]] :map[<tiddler>jsonget<currentTiddler>!is[blank]else[]] =yes =yes [<now [UTC]YYYY0MM0DD0hh0mm0ssXXX>] =[<newtags>]"
$timestamp="no"
/>
</$list>
</$let>
</$list>
</$let>
\end importTiddlers
But this code is not working ;(
I’m sorry that my response here will not address your actual request.
Though what you’re envisioning is not hard to do, it makes more sense to problem-solve around why you feel the need to juggle capitalized and lowercase versions of titles (rather than to help you invest more effort in that very klugey direction)
Your experience (and mine, prompted by testing your kind of workflow) raises an important implementation question for me, which I think we need to flag @saqimtiaz on (even knowing saq probably can’t put significant attention toward this anytime soon):
OBSERVATION: SAQ’s CURRENT DEFAULT NEATLY HYBRIDIZES EXTERNAL & LOCAL FIELDS
If a tiddler is “adopted” (by toggling away from “yes” in the is_volatile field), then a copy of it gets saved (as expected). But the next time external tiddlers are loaded, the solution as saq has set it up so far constructs a HYBRID result for any already-“adopted” tiddler:
- the incoming tiddler has a new
includeTimestamp - “yes” is (re)written into the
is_volatilefield - external content overwrites any local field-values.
- HOWEVER… any field-value pairs that are unique to the local (pre-import) tiddler remain!
** At least, they remain for the current load session…
Is this intentional? I think @saqimtiaz could have very good reasons for doing this: you could want to add local tags and comments / notes about a book (say), and still benefit from importing new information and/or corrections within the authoritative central record.
Then again, perhaps this hybrid behavior could just be an artifact of some json data remaining in un-erased limbo. I wonder this because if saq had wanted this behavior, it would have made sense to protect all local data not just for the duration of this browser session, but beyond (through subsequent saves)…
RISK OF DATA LOSS
ALAS, the benefit of this hybrid load behavior, as currently implemented, is easily outweighed if one must go through and manually re-toggle (for “adoption” / saving) all the tiddlers that started the session with local field contents that of course you don’t want to lose. But going through and doing this toggling is especially difficult, because we don’t even seem to have any clear way within the local wiki of tracking — after the re-import — whether an imported tiddler has fields whose data is only from the local side, and which fields those are!
(A naive user will see, perhaps with satisfaction, that their locally generated field-data from a prior session is “still there” after (re)importing a “conflicting” tiddler. They may be unlikely to realize that this hybrid tiddler, with its locally-generated content intermixed with remote content, has reset itself to disappear, taking the distinctive local content — which cannot be restored from the source wiki — with it.)
EASY SOLUTION?
If the load process is already doing some savvy “dovetail” work with local and incoming field values, then surely it’s possible to tweak the process so that whatever data that a wiki had, when its browser session started, all of that could survive the process of importing (“adding”) external content (with the possible exception of incoming field values overwriting local ones). That seems like an important norm — something users are entitled to expect, once this solution moves from experimental to mature. (To be clear, @saqimtiaz is doing some great pioneering work here, and of course it’s always true that details can fall into place only after the proof of concept!)
A simple solution would adjust the load-external-content process to check whether the local copy has the is_volatile field, and then to preserve that field value – even if other locally existing field-value pairs are overwritten. (So, the includeTimestamp is updated, confirming the most recent integration of content from the source/authority wiki, and all other fields behave as they now do.)
Optionally, too, perhaps @papiche (and others) would like the option to configure the import/load process so that it simply bypasses any existing local tiddlers with the same name.
(That is not my preference — I love the dovetail solution — perhaps enhanced by letting the local copy protect some specified set of field-value data, including the is_volatie field. Also, I think there will be great uses for having local shadows, and even other system tiddlers, overridden by incoming tiddlers with the same name. This sounds like a great way to sandbox complex and powerful stuff, knowing that those external tiddlers, by default, won’t save.)
MORE COMPLEX VISION:
Ideally, some additional fields could also be user-configured to be SAFE from overwriting; certain local field values (say, tags, or “lending status” for a biblio record) could be important to retain, even when the “master” record has a conflicting version of that field.
Most ambitious (and probably not easy): figure out how to employ a “TWO-SIDED FILTER” (!) Right now the filter for external content is entirely directed at the remote source, and is evaluated only as a filter within that remote wiki: “Go look at your tiddlers and send me all your tiddlers that meet this filter condition.”
But we could also imagine an filter defined/parsed on the LOCAL side, so that the INTERSECTION of the two filters determines the import. For example, "Look at these potential incoming tiddlers (in the filter we’ve already been using), and ignore all EXCEPT those that also meet this condition: Say, we want to load those that fit the [is[missing]] filter condition here, or all those remote tiddlers whose titles are listed in the vocabulary field of a local tiddler, etc.
(A rather cumbersome version of that intersection could already be achieved by starting with a bluntly enumerative local filter: I use a local filter like [is[missing]!is[system]format:titlelist] to produce a titlelist, and use that list of titles to start my import filter, going on to add further general constraints that can be parsed on the remote end. But this is clearly a very brittle way of getting at the intersection of local and remote filter criteria!)
LAST THOUGHT:
Another configuration detail that could be helpful … and I can probably figure this one out by myself soon… is to make the publishFilter check whether the external tiddler has been modified SINCE its importTimestamp. (Or perhaps any edit to the tiddler actually toggles the is_volatile field to “no” so that our css will reflect its new status…) That way, naive users get the benefit of keeping the local wiki “lean” (without too much redundancy), while avoiding the risk of losing whatever fresh content they are adding to these remote-origin tiddlers. Maybe this reduces the demand for something like a toolbar button to “adopt” tiddlers — which probably would be used most frequently on tiddlers that need have a future life with local edits.
@Springer I wont have any time to actively work on this before the end of April at the earliest but will be happy to follow the conversation until then as time allows.
A few quick thoughts that might prove helpful:
- The intent behind creating this plugin was actually just to provide an example of how tiddlers can be fetched and imported from a remote data source using just wikitext. It was meant to serve as an example others could adapt to their own usecases.
- The merging of fields from the local and remote tiddlers that is taking place is entirely an oversight, the original intent was that the remote tiddler should overwrite the local if a corresponding configuration option has been set.
- I am open to working on the plugin further to make it more configurable for common usage scenarios. However, highly specific needs that involve merging fields from remote and local tiddlers are likely going to be best served by emulating the code in the plugin to write a custom solution. Alternatively, one could imagine factoring out the actions that filter the tiddlers and save them as a separate tiddler that could be overwritten and customized by users to meet their specific criteria.
2 Likes
Thanks for the speedy reply, @saqimtiaz !
I do think it’s incumbent on folks (like me and papiche) who benefit from this work to go as far as we can in adapting it and learning from it.
Although my learning curve is steep here, I’m just far-enough along to have a sense of the structure of a solution that may work for a wide range of users (and a cringey feeling about some workarounds that I think will lead to more trouble).
Much gratitude to you for this proof-of-concept, and for being at least potentially within earshot — even while you’re juggling other tasks. 
1 Like
Some more info that may help in adapting the code in the plugin:
This the part of the code that extracts each individual tiddler from the incoming data and then saves it: https://github.com/saqimtiaz/tw5-plugins-sandbox/blob/e9ebad0708d6c15eb0b8974a3d9328e3b0c0e9af/plugins/sq/ExternalContent/load-wiki-actions.tid#L20-L35
It is the usage of $action-setmultiplefields here that led to the inadvertent merging of local and remote fields, for an overwrite it should be preceded here by deleting the local tiddler for an overwrite. Note also that rather than manipulating the $action-setmultiplefields in that code which can get hairy, it can also be followed up with $action-setfield to further add/manipulate fields
2 Likes
Thank you for your detailed thoughts.
It is true, that managing field update procedure is tricky.
This is where UPlanet Geokeys and “email” tags signatures and crypto coins are useful.
the more a Tiddler gets “email tags”, the more it is considered trustful.
Each TW (associated to an email) benefits from a relative “Web Of Trust”.
And is included in a SECTOR that is get all Tiddlers with > 3 signatures.
it receives “Externals” from “SectorTW”
A Tiddler can be shared “1 by 1” in different sectors.
When a 4th signature is collected, corresponding SECTOR will get it, so this Tid is spread to all “in area TWs”. Then all signers get paid by the last one (Zen is the crypto coin).
In this scenario,
- an external Tiddler overwrites local (with all fields)
only if “external_signatures_nb” > "local_signatures_nb
and if TW wallet have enough to pay "external_signatures_nb “Zen”(0.1 Ğ1)
For now, adding [days[-1]] to filters can do the trick, so UX works in a “day by day” usage.
It will have more complex fusion algorithm later.
But using “Coin” and “WoT” will ease things
About UPPERCASE title, as TW is engaged in such an automatic “blockchain process”.
This was introduced to give user a way to avoid treatments. It is also matching “net ethic” as it is considered as SHOUTING I consider it not so klugey.
I am glad I received already so useful help, I want to thank you.
I am sad my “TW programming skills” are still bad…
So I opened the side project to make assistant collecting TiddlyWiki PDF documentation to create a “privateGPT” for TW. This research and service is also part of “UPlanet REGIONS” data handlers (more than 144 signatures would be difficult to deal with TW alone).
@Springer I am not yet trying to follow this topic in detail however reading your account of the difficulties you are trying to address, they seem to be related to research I have done for tiddler import etc…
It seems to me if you can have local tiddlers, that are about to be overwritten by any external source, be turned into shadow tiddlers, by packing into a plugin first, they remain safe.
- similarly encoding tiddlers into a json tiddler
You are then free to let any import or external tiddlers to replace the wikis tiddlers because you have a reference copy.
- with a reference copy you can apply additional processing and comparisons etc…
Of course we do not necessarily know the incoming tiddler titles in advance, so perhaps a per tiddler method to push a current copy of each tiddler into a “shadow copy” before being replaced.
- This is definitely possible
- such plugins can be disabled, exported and deleted
- the same tiddler can be in multiple plugins with one copy winning.
- we are used to thinking of plugins as read only, but they are not, they are just tiddlers, if there is no modules or javascript in a plugin, you don’t even need to reload the wikis.
In a similar way to shadow tiddlers we can easily create other types of virtual tiddlers related to a specific tiddler under a prefix, in a json or plugin.
- we do not have too many mature tools for this but we can make them.
Visions?
- Perhaps if all imports and external tiddlers existed in a kind of tiddler sand box, one we could push and pull tiddlers or their fields from/to, that we could nevertheless access, some of these issues would not exist. Over here Using tiddlywiki documentation when contained in a plugin - #16 by TW_Tones related conversation.
Enjoy the journey
This first request is easy (given the helpful pointer by @saqimtiaz in above reply), and it’s also something I expect to employ in my own variants (taking advantage of the fortuitous field-merge behavior noted above). I’ve just crudely modified saq’s version so that is_volatile field is set to “yes” ONLY if the local tiddler does not already have a value in that field.
Find the modified shadow here:
I’m not confident enough with regexp to wade into the details myself, but even from a distance I think you would need to specify more things:
- Do you mean tiddlers whose titles have ONLY capitals (and spaces), but no non-letters?
- Or, do you mean tiddlers whose titles have NO lowercase letters (but can have whatever other
$†®@𐡀Ч∈stuff people can throw into a title?
If you need every potential tiddler title to have two variations — one quiet title and one SHOUTING TITLE — then I’m just not seeing any simple solution that handles titles that aren’t based on letter-strings (which happens easily when someone makes a journal tiddler with title format like 2024-03-12 for example).
On top of that, once you’ve got multiple languages in play, you’ll have some languages where capitalization isn’t even a thing at all… （困るかも知らないな—）
I’m sure we have regexp folks here (like @Scott_Sauyet) who can give you a clear way to filter out ALLCAPS titles (with or without allowing for spaces, numerals, etc.), but I’m still scratching my head about this method. It seems to me (if I understand a fraction of your ambitions) that you’re aiming for a pretty cosmopolitan (hence international-friendly) project, no? Or is your user community made of tech folks who won’t mind being limited to a strict roman-letter titling convention in order to set up this lowercase/UPPERCASE contrast?
I haven’t been following this thread (and don’t have time to catch up right now.) I saw this and laughed out loud. I’m no regex expert: an advanced beginner at best. But then, I guess I can do this.
The pattern ^[^a-z]*$ will capture all titles that contain no lower-case letters:
So this:
<$let no-lowercase="^[^a-z]*$">
<<list-links """
[[HAPPY BIRTHDAY MOM! (2024-03-12)]]
[[Happy Birthday Mom! (2024-03-12)]]
[[2024-03-12]]
[[FOO AND A little BAR]]
[[FOO AND A LITTLE BAR]]
[[e.e. cummings]]
[[LBJ TOOK THE IRT DOWN TO 4TH ST, USA]]
[[When he got there, what did he see?]]
[[THE YOUTH OF AMERICA ON LSD]]
[[12345!]]
+[regexp:title<no-lowercase>]
""">>
</$let>
Will select these values:
HAPPY BIRTHDAY MOM! (2024-03-12)
2024-03-12
FOO AND A LITTLE BAR
LBJ TOOK THE IRT DOWN TO 4TH ST, USA
THE YOUTH OF AMERICA ON LSD
12345!
(And yes, it’s Mom’s 79th birthday today.)
If we wanted to also ensure that the title contained at least one uppercase letters a regex wizard could probably come up with some self-contained pattern, but I would just use another pass, like this:
<$let no-lowercase="^[^a-z]*$" some-uppercase="[A-Z]+">
<<list-links """
[[HAPPY BIRTHDAY MOM! (2024-03-12)]]
[[Happy Birthday Mom! (2024-03-12)]]
[[2024-03-12]]
[[FOO AND A little BAR]]
[[FOO AND A LITTLE BAR]]
[[e.e. cummings]]
[[LBJ TOOK THE IRT DOWN TO 4TH ST, USA]]
[[When he got there, what did he see?]]
[[THE YOUTH OF AMERICA ON LSD]]
[[12345!]]
+[regexp:title<no-lowercase>]
+[regexp:title<some-uppercase>]
""">>
</$let>
yielding
HAPPY BIRTHDAY MOM! (2024-03-12)
FOO AND A LITTLE BAR
LBJ TOOK THE IRT DOWN TO 4TH ST, USA
THE YOUTH OF AMERICA ON LSD
(note that we now skip 2024-03-12 and 12345!
Wow, that’s fantastic! It may not mean anything to her, but a friendly wave from central CT!
1 Like
I’m not ignoring your suggestion, but I admit that orienting to plugin-packaging operations feels like a learning-curve I’m not ready for!
But it’s cool to know that there are ways to play with shadow tiddlers that would allow for a safer workflow when importing. I know I’ll want to come back to this, and I appreciate your sketching out this kind of solution.
- I feel if we know something is a plausible solution it should be raised, but need not be used right now.
- I would only suggest its easier than you think.
Nce work @Scott_Sauyet
- I either need a library of regular expressions, Mohamad’s comes close, or bite the bullet and learn Regular expressions.
However;
I expect we can use functions and existing case “filter operators” to compare the input with the modified case. I will explore this and return if I have some nice examples.
- If I can’t do this easily it may point to a need for a small enhancement to the operators or filters.
I have no idea how much effort is required for true mastery, but it’s not too difficult to gain basic competence.
A small additional instruction how they relate to tiddlywiki would may also help.