I’m going to describe parts of how I built a wiki for my local political party. I think the analogies should be obvious. But if not, please ask.
Input Data
Among other things I want in my wiki is a list of voters, so we can query various things about them without going out to an external tool. In fact only a few of the intended users of this wiki have access to that tool, but I was able to use it to get an extract, which looks like this:
RawData/VAN_Extract.txt
Voter File VANID|LastName|FirstName|MiddleName|Suffix|Sex|DOB|Age|Party|mAddress|mCity|mState|mZip5|mZip4|mAddressID|Address|City|State|Zip5|Zip4|AddressID|Preferred Phone
123|Bear|Yogi|Da||M|05/01/1958|66|U |123 Jellystone Park |Hanna Barbera|ST|12345|6789|294191234|123 Jellystone Park |Hanna Barbera|ST|12345|6789|294191234|8005551234
234|Rubble|Betty|Marie||F|11/01/1960|64|U |347 Cave Stone Rd |Hanna Barbera|ST|12345|6789|294191491|347 Cave Stone Rd |Hanna Barbera|ST|12345|6789|294191491|8005552345
345|Doo|Scooby|Joseph||M|05/01/1960|63|D |5 Mystery Machine Ave |Hanna Barbera|ST|12345|6789|294191899|5 Mystery Machine Ave |Hanna Barbera|ST|12345|6789|294191899|
456|Flintstone|Fred|C||M|03/01/1960|64|R |345 Cave Stone Rd |Hanna Barbera|ST|12345|6789|294195505|345 Cave Stone Rd |Hanna Barbera|ST|12345|6789|294195505|8005553456
567|Flintstone|Wilma|J||F|03/01/1960|64|D |345 Cave Stone Rd |Hanna Barbera |ST|12345|6789|294190525|345 Cave Stone Rd |Hanna Barbera|ST|12345|6789|294195505|800055553456
(One row for each of the approximated 2500 voters in my town.)
Okay, perhaps I changed the details to protect the innocent guilty.
I extracted all the addresses from there and used an online service to get latitude and longitude information for each address, and stored it alongside the above, in this format:
RawData/LatLong.json
{
"Hanna Barbera": {"latitude": -14.599400, "longitude": -28.673100},
"123 Jellystone Park": {"latitude": -14.592123, "longitude": -28.654321},
"345 Cave Stone Rd": {"latitude": -14.598888, "longitude": -28.684666},
"347 Cave Stone Rd": {"latitude": -14.597777, "longitude": -28.684567},
"5 Mystery Machine Ave": {"latitude": -14.600600, "longitude": -28.678901}
}
(and the lat/long’s have been moved to the middle of the Atlantic to protect them 
)
Conversion Process
I wrote some Node.js code to parse these two files and convert to the tiddler .tid format, one for each Voter, one for each Address. The specific code is probably not very helpful, but it’s available if you want to see it:
Code
scripts/buildContent.js
const {writeFile, mkdir, rm, readFile: rf} = require ('fs/promises')
const tap = (fn) => (x) => ((fn (x)), x)
const map = (fn) => (xs) => xs .map (x => fn (x))
const call = (fn, ...args) => fn (...args)
const display = msg => tap (() => console .log (msg))
const allPromises = (ps) => Promise .all (ps)
const readFile = (filename) => () => rf(filename, 'utf8')
const main = (fileName, latLong) =>
deleteOutputDirs() // ensure there's no detritus from previous runs
.then (createOutputDirs)
.then (readFile(fileName))
.then (delay(500))
.then (display ('Built directories'))
.then (psv2arr)
.then (handleVoters)
.then (handleAddresses(latLong))
.then (() => console .log ('Completed!'))
.catch (console .warn)
const deleteOutputDirs = () =>
rm ('./tiddlers/HannaBarbera/Voters', {force: true, recursive: true})
.then (() => rm ('./tiddlers/HannaBarbera/Addresses', {force: true, recursive: true}))
const createOutputDirs = () =>
mkdir ('./tiddlers/HannaBarbera/Addresses', {recursive: true})
.then (() => mkdir ('./tiddlers/HannaBarbera/Voters', {recursive: true}))
const delay = (t) => (v) => new Promise (r => setTimeout(() => r(v), t))
const psv2arr = (
psv, [headers, ...rows] = psv.split('\n').filter(Boolean).map((r => r.split('|')))
) => rows.map((r) => Object.fromEntries(r.map((c, i) => [headers[i], c.trim()])))
const handleVoters = (rs) => Promise.resolve(rs)
.then (map(getOverview))
.then (map(writeTiddler))
.then (allPromises)
.then (tap (ps => console .log (`Wrote ${ps.length} Voter tiddlers`)))
.then (() => rs)
const getOverview = (r) => [
`./tiddlers/HannaBarbera/Voters/van-${r['Voter File VANID']}-${r.FirstName}_${r.LastName}.tid`,
convertPerson(r)
]
const convertPerson = r => `title: Voters/${r['Voter File VANID']}
tags: Voter
caption: Voters/${r.FirstName + ' ' + r.LastName + (r.Suffix ? (' ' + r.Suffix) : '')}
first-name: ${r.FirstName}
last-name: ${r.LastName}
middle-name: ${r.MiddleName}
suffix: ${r.suffix || ''}
full-name: ${r.FirstName + ' ' + r.LastName + (r.Suffix ? (' ' + r.Suffix) : '')}
gender: ${r.Sex}
age: ${r.Age}
party: ${getParty(r.Party)}
phone: ${makePhone(r['Preferred Phone'])}
address: ${r.Address}
`
const makePhone = (p) => p
? `${p.slice(0, 3)}-${p.slice(3, 6)}-${p.slice(6, 10)}`
: ''
const getParty = (p) => ({
'U': '', 'D': 'Democratic', 'R': 'Republican',
'I': 'Independent', 'G': 'Green', 'L': 'Libertairan',
}) [p]
const handleAddresses = (latLong) => (rs) =>
Promise.resolve(rs)
.then (convertAddresses(latLong))
.then (map(writeTiddler))
.then (allPromises)
.then (tap (ps => console .log (`Wrote ${ps.length} Address tiddlers`)))
.then (() => rs)
const convertAddresses = (latLong) => (rs, loc) => Object .entries (Object .fromEntries (rs.map (r => [ // `entries` dance for uniqueness
`./tiddlers/HannaBarbera/Addresses/${r.Address.replace(/\s/g, '_')}.tid`,
`title: Address/${r.Address}
tags: Address
caption: Address/${r.Address}
address: ${r.Address}
street-number: ${r.Address.split(' ')[0]}
street: ${r.Address.split(' ').slice(1).join(' ').replace(/ (?:Apt.?|#).*$/i, '')}
${addApt(r.Address)
}city: ${r.City}
state: ${r.State}
zip5: ${r.Zip5}
zip4: ${r.Zip4}
${(
loc = latLong[r.Address.replace(/ (?:Apt.?|#).*$/i, '')] || latLong['Andover'],
`lat: ${loc.latitude}
long: ${loc.longitude}
alt: 0`
)}`
])))
const addApt = (a, m = a.match(/ (?:Apt|#) (.*)$/)) => m ? `apt: ${m[1]}
` : ''
const writeTiddler = ([fileName, content]) => writeFile (fileName, content, 'utf8')
main ('./RawData/VAN_Extract.txt', require('../RawData/LatLong.json'))
And I run this code with node scripts/buildContent
Tiddler format
This reasonably simple script creates files that look like this:
tiddlers/HannaBarbera/Voters/van-123-Yogi_Bear.tid
title: Voters/123
tags: Voter
caption: Voters/Yogi Bear
first-name: Yogi
last-name: Bear
middle-name: Da
suffix:
full-name: Yogi Bear
gender: M
age: 66
party:
phone: 800-555-1234
address: 123 Jellystone Park
and like this:
tiddlers/HannaBarbera/Addresses/123_Jellystone_Park.tid
title: Address/5 Mystery Machine Ave
tags: Address
caption: Address/5 Mystery Machine Ave
address: 5 Mystery Machine Ave
street-number: 5
street: Mystery Machine Ave
city: Hanna Barbera
state: ST
zip5: 12345
zip4: 6789
lat: -14.6006
long: -28.678901
alt: 0
This is the .tid format. Fields are given in lines like field-name: Field value. title is a required TW field. tags and caption are very common ones. The others in these samples are specific to my wiki. Note that here I don’t include a text field in these tiddlers, but if you want one, it simply appears last, and without any key, separated from the other fields by a blank line:
title: My Tiddler
tags: Demo [[This is temporary]]
And here we begin the multi-line
`text` field, with //whatever// wikitext
we choose.
Folder Structure
Note that these files get dropped into subfolders inside the tiddlers folder. If you’re running in Node, all tiddlers live inside tiddlers, and the internal folder structure is simply a convenience. They all end up in TW’s flat namespace – based on internal title and not the file name. But if you’re not running in Node, these files can still be dragged and imported into any wiki.
(In actual practice, I don’t drop these in the tiddlers folder, but in the plugins one instead. That treats them essentially as data-only tiddlers. But it’s not important here.)
Now by adding various templates and cascade options, I can view and search this data in an interconnected manner. You can see it in action at http://scott.sauyet.com/Tiddlywiki/Demo/HannaBarbera.html.
So that’s my practice. I’ve done similar things on a number of wikis. I don’t know if you can get your data into good enough shape for such an automated convers. I have my doubts, seeing the text generated by pdf2go, but there’s much I don’t know. In any case, I find this a useful way to initiate data-heavy wikis, based on an external source.
