It may be useful if we have a plugin did diff and merging like git, choose which part from updated shadow tiddler we want to merge into our tiddler, visually.
TLDR;
Conclusion … I think having “diffs” that show the differences between a tiddler and its shadow is a nice to have. … Most of the times I would only be interested in the diffs. …
But … maintaining patch-sets with TW will probably be a maintenance nightmare. There are whole systems that are designed to mange the problems discussed below. eg: Git.
The problem here is that we would need to keep track of the “patch-sets”, because they are version dependent. … There is a really good example at: Diff, Match and Patch: Demo of Patch
→ Play with it … It’s fun.
Demo of Patch
- Two texts can be diffed against each other, generating a list of patches.
- These patches can then be applied against a third text.
- If the third text has edits of its own, this version of patch will apply its changes on a best-effort basis, reporting which patches succeeded and which failed.
- In this scenario Shakespeare wrote Hamlet in Early Modern English, the source document.
- Then two derivative works were created. One is Hamlet updated to Modern English.
- The other is a Star Trek parody in Early Modern English.
- This demonstrantion creates a list of patches between the source and the Modern English version. Then it applies those patches onto the Star Trek parody, thus creating a Star Trek parody in Modern English.
Applying the concept of patches to TW will show the problems we would have to deal with.
- TW v5.2.4 is like the very first source document from Shakespeare
- The Modern English text may be TW v5.2.5 … Some little changes
- The Star Trek version may be TW v5.3.0 … Big changes.
As you can see, while the patches may all have been passing. The output is extremely far away from v5.2.4 and v5.3.0 …
The patch-set that was designed to update v5.2.4 to v5.2.5 … and is tested and will work.
The chances are very high, that it will completely mess up v5.3.0 if applied to it. … That’s called edit conflict which usually needs “user interaction”. … Without a really detailed documentation a patch-set most of the time is useless for wikitext code
It may work for prose text, where it does not really matter, if you get the Treky version of Hamlet. … may be 
1 Like
Mario, You understand this subject well, provided elegant documentation in the related topics. Thanks very much.
- I truly understand the issues you raise, and we have discussed in the past.
- Here I carefully spell it out to maintain simplicity of my argument.
I would like to add a little additional nuance and discuss a very simple dif and merge, one which users/designers may make use of and may one day be applicable in the upgrade process.
- If more than one shadow/plugin contains the same tiddler, all bets are off and it is something we should avoid.
- I am only interested in the difference between a shadow and its edited tiddler. These are the two tiddlers, you have documented this nicely here
- I am not talking about patch management, distributing patches or anything else, just the ability to identify and capture differences when a shadow has being modified, at an all fields level.
As you know when we edit a shadow tiddler, typically the modified and modifier and/or created creator fields most likely change. If that is all that changes most likely one could just delete the tiddler and restore that shadow.
- A simple programmatic way, eg a macro or better a filter operator, to detect an edited tiddler with no “effective difference” would be helpful. This would help list and delete tiddlers that are unnecessarily edited tiddlers, This is an all fields check, excluding nominated fields, at present it is only the text field using manual preview diffs.
There are however a number of legitimate reasons to edit a shadow tiddler, changing the text or any other field. But there is an issue when an upgrade to the underlying core or plugin occurs, in so much as the edited tiddler remains in force while the underlying tiddler has changed.
- it is important to acknowledge this status quo is not perfect.
- these changes are in fact trivial 90% of the time like;
- “show” instead of “hide”,
- or insertion of a key/value pair in a create tiddler widget etc…
- or adding / removing a tag/field or value etc…
My thought is, what if we could run a process to save the differences between shadows and tiddler to a data tiddler (or otherwise) and allow us to restore the shadow, we would then have a record of the edits. This would then allow a batch or manual reapplication of the saved changes later?
- We may retain both the original shadow and the edited shadow in this process, ensuring no information is lost.
- After a successful upgrade we can inform the wiki owner the following shadow edits were stored and they should reapply them only if necessary.
Thus the upgrade process could just save the differences (with a warning perhaps) clear the over written tiddlers and upgrade the wiki and or plugins. Then let the user or plugin selectively reapply the difference if need be.
- With such a process any “modified shadows” will become obvious, as they will have a differences record.
- The upgrade will be applied, reliably and all new features present
- It would be easy to “restore customisations” if needed/wanted.
Most importantly such a set of tools could be used on demand for troubleshooting or at upgrade time, They have nothing to do with the upgrade process except perhaps making it more reliable by letting all upgrades apply, and is only an issue at upgrade time. It is merely about being able to manage edited shadows at a moment in time.
- These tools would be useful even if not associated with the upgrade mechanism.
Finally if the same method could be used against other tiddler pairs like a new tiddler and its tiddler-source, previous versions etc… additional features could be build on top of tiddlywiki for change management and other solutions.
1 Like
I wonder if there is any plausible way to determine this case?
That’s already possible.
- Create a tiddler with some text
- Save it
- Edit it
- In the preview you have “differenced from current”
Which imo is what you describe here
No, You can only do this visually, The difference engine only counts the changes in a single field, text by default, you can’t get a change count for a tiddler to then use to control other code. You can’t write the tools I am suggesting.
- I believe additional TiddlyWiki access to the JS in the librry is there.
That’s exactly what I wanted to describe in my very first response. … If we are OK, that we get the Klingon version of a prose text tiddler, that would probably work, because it does not matter.
For code-shadows an automatic process may work for some trivial changes … until it does not work any more. …
If such changes are done by the core update process and it creates something that is broken, it is perceived as a core bug. Users will call for help because the update process is broken.
This will add additional maintenance pressure on the dev team.
I think it would be nice, if the upgrade process can show if shadow tiddlers have been overwritten without a real change. – So it will be OK to delete them.
I think that would be a very welcome improvement. Since unchanged but overwritten shadows also cause extra maintenance requests. … If we can avoid that. → good
I think only the first point it viable. … It will be obvious that there have been changes. → Yes I would support that.
It will not be reliable in the sense that it is guaranteed that the changes will work.
It will reliably overwrite the working shadow tiddler with something that may ore may not work.
It will look like if it would be easy for trivial cases and it would be easy to create examples where it works. … BUT the problems are caused by real world cases where it does not work.
2 Likes
I understand the concern that non trivial cases may not work if the difference is applied automaticaly but having the information about past changes is just more information than we currently have.
I am suggesting we have the tools to address a feature gap and designers can make their own design decisions.
That’s all cool stuff and it’s possible to do so, with the libraries we do have in the core. As the examples show but somebody would have to implement it.
I think several community members can implement the wikitext elements that are possible already, as I showed at HowTo Compare the Content of a Shadow Tiddler With the Content of an Overwritten Shadow … The merge options would need some js-code, but to make the differences visible is wikitext only.
The core devs have to deal with about 1.1k open issues, where probably half of them are still valid and about 160 open pull requests with probably about 30% are still thinkable.
So until we can clone several devs none of that will happen in the near future if the community does not step up more and actually start to try to contribute to the core using GitHub
1 Like
I hope to be able to do this, but it will likely be several months before I feel confident enough as a TW user to even consider becoming a TW developer.
I’ve just published a draft PR for three additional filter operators that expose more of the functionality of the diff-match-patch library that is already integrated into TW5:
- Compute the Levenshtein distance between two strings (the Levenshtein distance is defined as the number of characters that must be inserted, deleted or substituted to turn one string into another); it is a useful way to measure the similarity of two strings
- Create a set of patches to represent the difference between two strings
- Apply a previously created patch to a string
I think these primitives are sufficient for exploring the three way merging idea.
5 Likes
Nice. I did create some examples to play with. Texts are from the original examples. Order to click the buttons from top to bottom. If you import the StoryList the tiddlers are sorted in the right way
Import the JSON to the preview wiki: https://tiddlywiki5-git-diff-match-patch-operators-jermolene.vercel.app/
diff-match-patch-examples.json (2.8 KB)
have fun!
mario
1 Like
Please see here [RFC] Running a difference check on whole of tiddler including fields - #3 by TW_Tones a desire to get a little more from that Library, specifically to return a count so we can then use this in a condition test, even to sum the number of changes.
Note the count is currently returned inside a message and not usable.
- Even just returning zero, one or more would help.
- Perhaps the levenshtein operator can be used for this but perhaps a simpler operator is different or has differences …
But;
This means the count is something you can return, once for each comparison, it gets more complex trying to sum the count of multiple comparisons. Or multiple widget calls.
- However I will revisit it to check
diff-text - @jeremyruston’s Levenshtein (distance between two strings) operator is a reasonable proxy because for a set of comparisons, as I want to know if there are 0, 1 or more changes between two tiddlers fields.
- However I think we can make full tiddler comparisons easier.
IMO as soon as there is a number different to 0 it will be needed to show the real differences. Instead of count you can use the levenshtein distance. IMO it’s equally useful, since the only number that does not require any user action is 0
From Wikipedia, the free encyclopedia
In information theory, linguistics, and computer science, the Levenshtein distance is a string metric for measuring the difference between two sequences. Informally, the Levenshtein distance between two words is the minimum number of single-character edits (insertions, deletions or substitutions) required to change one word into the other.
Part of the reason I worded this the way I did is there is also a difference between an empty existing field and no field at all.
Whether we use a javascript a tiddlywiki macro or a widget that calls the existing library we are building a lot of detail into the difference check so it approaches a patch system, all good, however we have not yet built a simple difference test, which allows whole of tiddler or field by field comparisons (with the necessary field exceptions).
- I can build a wikitext macro on top of the Levenshtein operator
- However I would seem we should be able to use the existing library
- perhaps a diff-tiddler widget to complement the DiffText widget.
- Identifies all fields in two tiddlers and compares them
- Allows except fieldnames
- option count new ot deleted empty fields option
- returns a true/false or count of fields with a difference not changes within differences.
There is a new preview wiki wiki, which contains this PR. make import listing more configurable
This PR allows us to implement user defined extensions to the “overwrite” message area.
If you import this JSON into the preview wiki you’ll get an overview similar to the following screenshot
import-listing-verbose.json (1.5 KB)
This code is far from being finished, since IMO at the moment it adds more visual clutter than useful info. …
But it shows what’s possible.
If the title would adjust to the width of the longest title it would be easier to read.
- Ideally if the title consumed more than 50% of the width the status would appear on the next row.
If in the area that displays the Status could be populated using a tag such as $:/tags/import-status the existing and incoming information could be placed in a tiddler with the $:/tags/import-status and it would be easy, without core modification for me to include new tools like;
- Full count of differences by field
- If suffix “readme” or “instructions” display rendered text from that tiddler.
Should it be possible to use these to create an in-wiki differential tiddler backup / snapshot feature?
I’ve always shied away from in-wiki backups as they would not only explode its tiddler count, but also its size. Using patches at least the size concern would be diminished. File-backups do not increase the tiddler count but can blow up your required hard/cloud drive space. This could be a middle-ground sweet spot, at least for single-file wikis.
1 Like