I have a ton of GEDCOM files as well. Some are quite small and some are quite large. I’m sure I could give it a good go.
Put me down for testing imports into MemoryKeeper! Would sure save a LOT of typing, especially if sources or at least citations would come in as well. More concerned about linking to local copies of files (photos, scans, etc.) as a source rather than online data or books.
1 Like
@dixonge thank you. I’ll post back here with any progress, which has been very little so far. I said it’ll be slow, and even slower if the weather is nice. Last weekend was gorgeous and this weekend is even better. I did get some work done last week, but more importantly I elected to created a “Change Management” system (in TW) to capture requirements, design notes, tasks, etc. It is also a knowledge base to capture tips, solutions etc. My very first TW was a change management system for work. So I had a shell to work with. I have since created another instance for tracking Memory Keeper.
2 Likes
Looking for some feedback. I have been moving forward with a GEDCOM plugin. However, some feedback might prevent a nasty re-write later. To provide feedback do not feel you need to know or understand GEDCOM or its structure. You simply need to know it is a text-based, human-readable, file format that represents a genealogical record exported from a genealogical tool that we what to import into TW.
So a GEDCOM plugin will do the following:
- import GEDCOM data source into TW
- export GEDCOM data source from TW
Anything else?
To support various TW implementations these two methods will follow a configuration. Therefore, the plugin will also include a UI to manage GEDCOM import/export configurations. The plugin will come with a few system-supplied configurations, such as one for Memory Keeper. New configurations can be created from scratch or cloned from an existing one. Configurations are defined in tiddlers tagged with: $:/tags/cls/mk/gedcom/configuration
An example configuration might include something like… a GEDCOM INDI record (a person record) generates a dedicated tiddler tagged with “person”. A user could change the tag. How else could one represent a person in TW? GEDCOM SEX M is mapped to tag: “male” and SEX F is mapped to tag: “female”. With a configurable approach, the user could change this so GEDCOM SEX M is mapped to a field: “gender” with a value of M.
Using the examples above what other configurable options could we define in TW? What I mean is what other approaches could be defined in a configuration beyond tags and fields? There are obviously more complex data structures than a person’s gender that needs to be addressed, so I suspect I’ll want more configurable options. I’ve seen other TW solutions use tags to represent parent/child relationships. I’m not a fan of this approach, but I should support it.
When developing MK I took my own past experience and TW knowledge to define how I wanted to represent genealogical information, but this plugin should support what others might want to do.
Once the user selects a configuration and selects the GEDCOM file to import, I have defined the import process with three major stages:
- import GEDCOM into a JS object model
- apply necessary changes to the data in the model to accommodate the import configuration
– E.g. address potential duplicate tiddler names (likely another configurable option) - generate tiddlers from the JS objects and based on the selected configuration
Step 1 (above) does not need to know anything about the configuration. It is written–and it appears the object model is populated as expected. I think I have about 80%, the most popular, of the specification data structures and elements implemented. I feel the good ole 80-20 rule applying here. The last 20% will take 80% of the time. 
I have not started the next two steps or written anything for the configuration.
Other considerations an import should/could/must address are:
- multiple GEDCOM files are being imported.
- related to the point above, should the plugin offer duplicate tiddlers resolution tools?
– This refers to duplicate individuals, or duplicate source records (not duplicate tiddler names)
– A tool would enable the user to merge data. I’m thinking this is a complex process and should be managed elsewhere, not in this plugin.
Anything else?
Appreciate any comments, feedback, things to watch out for, and better ideas!!
Sorry for the late response. I’ve been under the weather the last few days.
I like the configuration idea. A good thing to have under configuration options might be the capability of adding custom tags. I have a large file in Rootsmagic right now and that program using tags specific to it. I like the idea of this being configurable since many different programs treat GEDCOM slightly differently.
You may have thought of this already, but, at least with GEDCOM import, an error reporting mechanism would be extremely useful. This could be written to a log file or tiddler that would log what caused each of the errors such that the user could then go back and correct the records.
Those are the things that immediately come to mind. If I think of anything else (now that I’m a bit clearer headed), I’ll let you know.
Yes. This is a must. I have not finalized a target location yet. I do want a log, not just errors. Currently I’m logging to the console, but I’m experiencing that it has limits. I’ll experiment with a Tiddler, perhaps tagged as $:/temp so it does not persist. Not sure! Thank you for the feedback.
To all… please read (an odd TW observation you might be interested in—and hopefully you will comment on). To @HistoryBuff and @dixonge do you remain willing to test a GEDCOM import? I have not written a generic GEDCOM plugin. I still want to. I have implemented a GEDCOM import specifically for Memory Keeper. This built-in import mechanism dramatically exceeds the performance behaviour of the PowerShell script I wrote.
My goal is to complete an import for build 9— a GEDCOM export will come later. Build 9 is not complete, but I think the GEDCOM import is ready for testing–though there are a few more things I want to do.
A pre-release of MK build 9 can be exported from my “Churchill” demo project.
Using the link above navigate to the advanced search tiddler. Then on the filter tab select from the dropdown:
Memory Keeper plugin tiddlers
This will list all the MK tiddlers. Export the tiddlers to a json file. Then import that json file into your MK project file.
Be very careful. This is a pre-release. You will also find a number of other new enhancements and bug fixes. My favourite enhancements are the new views. There is a new photograph view to show them in a grid (see demo). To take advantage of this view photocaption and caption fields need to be populated. Other new items include a new source type: DNA results; new statistical charts; custom datasets; and a configuration to enable the cross-referencing of tiddlers between separate MK project files.
I want to mention one issue I had with the GEDCOM import process, so you can watch out for it, and perhaps others here can comment on the TW behaviour.
There were a number of bugs and performance issues I needed to address and I will not be surprised if more are found. One issue I ran into (which I hope I have fixed) is that a specific set of tiddlers could not be edited in TW after the import. After the import when I opened a specific tiddler into edit mode and made changes to it, it would save the changes into a new tiddler and the existing tiddler remained intact. What I found was my import process, on a specific set of tiddlers, was wrongly adding a trailing space character to the title of the tiddler. What I also discovered is TW trims these trailing space characters when you save your tiddler edits. I think I understand why, but shouldn’t the addTiddler method, which I am using in my import, remove trailing spaces from the title field too, to prevent this ill behaviour?
Example call the import process performs to add a tiddler:
this.wiki.addTiddler(new $tw.Tiddler(this.wiki.getCreationFields(),this.wiki.getModificationFields(),fields));
Aside: Having hundreds of tiddlers with this ill trailing space character also caused a performance refresh issue. Therefore, correcting the trailing space not only corrected the edit behaviour but also dramatically helped the performance.
The GEDCOM import process can be found using the menu (Contents tab) Research–>Research Tools–>GEDCOM Import
Thank you to all. I think most of the enhancement ideas I implement come from here, not to mention many of the bug fixes. Thank you in advance to @HistoryBuff and @dixonge
Issues can be logged here:
https://github.com/clsturgeon/MemoryKeeper/issues
Craig
1 Like
I’m interested, although it might be a bit until I have time to test.
1 Like
Craig,
I was finally able to get to experimenting with this today, but am having trouble with the import process. Is the GEDCOM import button supposed to open a file chooser? How do I tell it which gedcom file to import? The button (nor the review button) seem to do anything. I followed the above instructions. What am I missing? Thanks.
Never mind. It seems to have something to do with the downloading of the empty 0.08 file. The button works in your Churchill demo. When I used that file to import a small gedcom of 29 individuals, it skipped all 29. From here on out, I’ll report the issues via github. I just wanted to make sure I started out on the right foot before I did that.
Thank you for this. It has been fixed and the demo has been updated to build 09 - prerelease 2.
I’ll try the update.
Aside from that issue my first try seemed to be smooth. A more detailed look coming soon.
1 Like
Finally getting to this. Exported all the tiddler plugins etc. to json, drug that to my existing copy of MK - was this correct, or should I have started w/ something else? The demo?
Ran into the same problem @HistoryBuff did - import button doesn’t do anything. As a matter of fact, the Review and Search buttons don’t do anything either…
According to info in the Control Panel I’m running at:
|TiddlyWiki version|5.2.3-prerelease|
|Memory Keeper version|0.0.09 prerelease 02|
So I’m stuck…
[UPDATE: I downloaded the demo version and the buttons work, but no individuals or events are imported. I added details to the GitHub issues…]
This typically means that the project file was not saved and reloaded after you imported the json file.
Clicking on any of the buttons on the GEDCOM tiddler should provide a prompt to load a GEDCOM file.
I have made two minor changes to the GEDCOM process., 1. when you do a REVEW it will now report how many lines were read. I’m wondering if the import was reading your file as a single line. To that end the other change I made was to support various end-of-line encodings. This is included 09 pre-release-03. You can download the json file here:
Buttons all work now, so probably not properly reloaded…
Updated to 03, tried Review and Import - same results, but Review did not report how many lines were read, FYI.
1 Like