tiddlywiki and genealogy
is it possible to write a routine that will transform a file in gedcom format into a tiddlywiki with one tiddler per person in the family tree?
but maybe it has already been done.
3 Likes
I’ve wondered the same thing and have even considered attempting it myself. Too lazy so far.
@pierre_deparieux I have put together a TW family history solution called “The Memory Keeper” (MK). This solution remains very much experimental, but it is maturing.
An example/demo can be viewed here:
http://craigsturgeon.com/mk/example/Churchill-Example.html
The example is using a pre-release of build 8; where build 7 is available for download. I want to release build 8 soon.
Regarding your question, I’m sure a TW javascript plugin could be written to import GEDCOM files. This would be a complex macro. There are three main issues that come to mind. 1. Being able to support all the aspects and complexities of GEDCOM, 2. How does a generic GEDCOM plugin support cross-product TW solutions? 3. Performance.
When I started MK I manually re-entered my data. However, when I generated my wife’s TW file I did not want to go to that effort. She did not want to do it either. Back in the '90s, I had written a genealogical software product with a GEDCOM import/export feature. I still had that code – I grab two methods from that: 1. relationship calculator (which I did convert to a TW macro and implemented it into MK); 2. GEDCOM import (which I converted to a PowerShell script).
This PowerShell script (GEDCOM2JSON.ps1) attempts to convert a GEDCOM file to a json file that you import into an MK solution. Remember my 3 main issues (above). This PS script is challenged with all three. 1. It will NOT cover all aspects of GEDCOM, but its does support primary GEDCOM tags. 2. It is not generic, it only addresses imports into MK. And, 3. It is SLOW.
I tested the PS with a 30K of people and 80K+ number of events and it took hours (double digits). The processor, memory, disk access, and size of the GEDCOM file will all determine how long it will take to run. Importing a huge resulting json file can be time-consuming too.
In the end, this PS script should not be used for large data collections. In my wife’s case, her GEDCOM file had hundreds of names—not thousands, and TW/MK worked well with her data set.
This PS script can be found here:
https://github.com/clsturgeon/MemoryKeeper
Note: PS will only be useful to Windows users. To use the script you may need to review your execution policy of PS. There is also a readme file associated with the PS script. Read that over… it provides the information to set the execution policy of PS.
As for a GEDCOM export feature, a macro could be written to support that too. The same issues apply. I have not looked into this yet.
Craig
I’m pretty sure somebody with big TiddlyWiki skills (widgets and, in particular search-replace with regular expressions) could do this with just TiddlyWiki. Not by dragging the file into TiddlyWiki, but rather by dragging the file contents (i.e .the text) into TiddlyWiki via dropzone widget and triggering custom import.
Or have some application/script (written in whatever programming language) to take your genealogy file and pump out “TID” files that could be dragged into TiddlyWiki for the native (i.e. au-natural) import mechanism.
Either way, you need a person with the skills. The second approach: you might have access to a broader pool of people to do the job, whether it be javascript, C, Python, … any programming language, somebody who can figure out how to get the data out of your genealogy database, but then only needs an understanding of the TID file format on the TiddlyWiki side of things.
Hi and Welcome!
Can you post a link to the gedcom format, so we can have a closer look?
Thanks @clsturgeon and @Charlie_Veniot for eloquently explaining why it’s not a simple task. Hence, my laziness in tackling it. Besides, I highly doubt I have the necessary skills, but could learn. Just need the time. Sigh.
1 Like
Welcome @pierre_deparieux to talk.tiddlywiki.org
Just having a brief look at the GEDCOM Specification once you take account of the different data objects within a file it may be possible to import each as a tiddler with its values in fields. You would then have the data in your wiki. But then making use of such data is where the devil is in the detail. But then if you have a specific reason to import this data you may not need to do much to make use of the data.
I note there are a number of tools to convert 5 Best Free GEDCOM to CSV Converter Software for Windows (a random link) and CSV is easy to import as tiddlers using JSON Mangler plugin.
Once you have the data in tiddlywiki it would be about how much you want to access the data and how far along the path from view/query to fully operational Application, you want to travel.
Rega5rds
Tony
I’ll take a first crack at this. I am very familiar with GEDCOM (v5. Less familiar with v7). I have working knowledge of JS. When I wrote the PowerShell script I had no experience writing TW JS macros. I am gaining working knowledge on these.
My first version will be dedicated to Memory Keeper. However, I plan to make it configurable. For example the user can configure how a person tiddler is defined. How is gender identified (tags, field)? Where are vital events placed? Dedicated Tiddlers or as a field in person Tiddler?
Last night I started. It is currently loading the file and makes a first pass through the file. This will take time to complete and as long as the weather is nice it will take even longer—which it will be this weekend. 
Any suggestions or requests please let me know.
1 Like
I guess the first question I would ask is if you expect to be able to edit or add to your family tree in TW and then expect to be able to re-export to a GEDCOM file at some time? In other words, do you expect loss-less data import?
1 Like
Just wanted to link to a resource. Tamura Jones has been writing about genealogy software for a long time. There are some good articles on GEDCOM versions, names, dates, gender, etc.
GEDCOM is deceptively simple format - it is fairly easy to parse data, less so to interpret the information.
1 Like
Great question. I’ll keep that mine. The last project I worked on (over 20 years ago), what I did was I would import raw GEDCOM the host application did not support. That data would get merged back in during an export.
I have a Ruby program to generate a TW JSON file from my gedcom. It’s too customized to be generally useful but can clean up if there’s interest.
One strategy I thought of using was to generate a data tiddler for each gedcom INDI record and FAM record. That way, the gedcom data is preserved 100%, and exporting would be a matter of concatenating the data tiddlers. Then use TW as a gedcom editor, in essence. I’m not sure how generally useful it would be to the average person as you’d need to be comfortable with manually editing the gedcom record. Then you could build tiddlers that are essentially views into the raw gedcom data tiddlers. I never experimented with this idea so I can’t say if it’s a viable workflow.
Well said. I see there is support for media. A separate challenge.
Another interesting idea. I’ll make note of this too. There maybe ways of reducing the GEDCOM look in TW. Thank you.
Craig - thanks for taking this on. I would love to help test your GEDCOM import once it’s ready for testing. I have a copy of your Memory Keeper that I’ve started playing with (which I love by the way).
1 Like
Are you thinking the raw GEDCOM (for an INDI record for example) is imported directly to a text field? I was wondering if we could, optionally, take advantage of Tiddler fields somehow. It would be nice for TW to still be used to draw charts, perform relationship calculations, etc, with the data still in this raw GEDCOM format. I’ll want an easy way to cross reference tiddlers.
I’d also want to write some sort of data integrity routines—users could easily corrupt GEDCOM syntax.
Thank you. This will be large under taking and need plenty of testing. I have a good collection of GEDCOM files, but I’ll for sure miss things.
I know Memory Keeper is not for everyone, but a GEDCOM plugin might attract a greater number of genealogist to TW.
I recommend this version:
https://www.tamurajones.net/DownloadGEDCOM551AnnotatedEdition.xhtml
The annotations are varied in nature. There are corrections of errors, commentary about differences with GEDCOM version 5.5, observations on the structure of GEDCOM files, notes meaning of some sections, clarifications of confusing parts and resolutions of contradictions. There is commentary on bad examples within the specification and on some real-world issues the specification never mentions. Several annotation provide advice, guidelines and best practices, often with links to relevant articles.
Obsolete and deprecated sections have been clearly marked as such, but have not been otherwise ignored; those sections do still contain corrections and annotations.
@clsturgeon Once I removed the Windows specific FileBrowser code, your script worked fine on powershell on Linux, specifically Debian. Was still kinda slow.
2 Likes