How should I store complex data

,
#1

Okay, So a brief overview. What am I even asking?!?!

So I have a tiddlywiki project similar to the TWBible however for this project I intend to break the books out into dictionary tiddlers, each book is a dictionary, each verse is a line. This means that a line can be cited via using something like so {{Book 1##01}},

Q: Why Not Just keep it in the tiddlers vs keeping it in a separate dictionary?
A: For language swapping, version swapping and citation.

Q: Have you tried using fields?
A: yes, it’s very clunky for my needs and takes forever to add Verse by verse, aswell as would require new wikis for new languages/translations

So what am I asking? In essence there will be ~31,000 dictionary references and give or take a few hundred structural tiddlers, would having this many dictionary tiddler entries render TW Slow ir unusable? Assuming that for each verse there is Both a dictionary entry and field value, 1,189 dictionary tiddlers. Would the complexity of this method cause any issues? I know by default performance will be hit but I don’t wanna convert the entire Bible just to discover tw5 goes full snail mode without even having the added functionality planned. Any tips? Or am I on the right course?

#2

If you read 99% of the time dictionaries are OK.

Do I read it right. You have 1189 dictionary tiddlers and every one of them contains 31,000 references?

That would be about 37 mio lines of text. So if every line has 100 characters that would be a 4GB wiki. I would never want to open that one from the web

#3

No 31,000 across all dictionaries not 31,000 in each that would be an impressive amount of entries but is not the total amount, at most the wiki will have about 32K entries across all dictionary tiddlers, and 32K entries as footnotes in tiddler fields. Im sorry I specified poorly theres not going to be millions of dictionary references, only tens of thousands. I expect that a full wiki should be less than 32 megabytes. I just was more so making sure that dictionary tiddlers was the right way to handle it given I can pack them as plugin tiddlers and make them swapable like language is. I hope that better clarifies :open_hands:

Estimated specs:
~02,500 total Tiddlers
~31,000 total Dictionary Entries
~31,000 max field entries

#4

If your users can download them as plugins, it would be possible to install the core Browser Storage plugin. That would allow you that plugis can be installed and then loaded from the local browser storage, which will probably reduce the load time quite a bit.

There will be several mechanisms to improve load times

#5

The core of my project I tend to have setup as multiple “plugins” so like Base & version, Base handles displaying the pages and version contains the book(s) [aka dictionary tiddlers] I know a biblewiki exists [TheBibleTW] but it’s not capable of language or version swapping and note taking requires adding the comments plugin, you could use the text slicer plugin to cite but that would end up being clunky and hard to use on such a massive scale vs {{Book 01##01}} which is a lot cleaner and would change with versions aswell. It’s good to know about browser storage I will 100% implement that. Thank you for the tip!

#6

No, it won’t. There should be no significant slowdown, based on my own experience.

You might be interested in my recent approach, King James Version (discussion in King James Bible Wiki). Here the basic tiddler is a verse:

title: 1 John 4:7
tags: Verse [[1 John 4]]
book: 1 John
chapter: 4
verse: 7
seq: 31836
para: 3

Beloved, let us love one another: for love is of God; and every one that loveth
is born of God, and knoweth God.

Chapters are built out of verses using a template. Books are built out of chapters using another. (There is special handling for Psalms in both of these.) I didn’t attempt a whole-bible tiddler, because I think that would be too slow, and it didn’t seem very important. seq is a sequence number which runs across all books, chapters and verses, making for consistent sorting.

This was a playground for me, meant to be extremely minimal so that others could build atop it. If you have the biblical text you want in a reasonably structured format, it should be trivial to convert it to this verse format. But, unlike the chapter-oriented format in the bible wiki you cite, there is some overhead to this. My minimal wiki is 11.8 MB where that one is 6.8MB.

The upside is that every verse is addressable (e.g. 1 John 4:7) as is every chapter and every book. If we wanted to add, say, note-taking on the content, it should be very easy.

There is a slight delay in loading, based on your connection speed, but once loaded, there is no obvious slowdown in rendering anything. There should be little difference if you do this with dictionary tiddlers by chapter. You may have to figure out a separate mechanism, though, if you want to handle the verses’ combination into paragraphs. My raw data keeps the list of paragraph data in a separate metadata section, per chapter, for instance. It gets converted on tiddler generation into the para field on the verse tiddlers. Again, there is significant extra processing for Psalms, but that is the only exception area.

There are two different “current” versions of that work, based on two different ways of connecting the verse numbers with the text, v5 and v7. (There’s nothing wrong with v6, but it was only a way-station.) What I really like would be something of a combination of them; I have a mock-up, but I don’t know how to implement it.

I have thought a bit about how to internationalize this. After converting the text, I believe it would chiefly be a matter of abstracting the tag names “Book”, “Chapter”, and “Verse” into language-specific tiddlers, and rewriting the parts of the templates that depend upon them. But I haven’t looked in detail.

If by some chance building atop this seems useful to you, let me know, and I’ll help as best I can. This work really is meant for that role, not to be anyone’s actual bible wiki.

#7

This is a curious question, Personally I would try and make use of tiddlers, no matter how many, before moving content into dictionary tiddlers.

  • I am not well versed in the Bible but belive I can perceive its data structure.

However given you know the fairly large size of the real content and how it is addressed you can take a minimalist approach and buid on that.

ChatGPT Says

Plain Text: A plain text version of the Bible is relatively small. For instance, the King James Version (KJV) is about 4-5 megabytes (MB) in plain text format, which is roughly 4,000,000 to 5,000,000 bytes.

  • This seems realistic

So my initial thoughts would be what if you have a tiddler for each book, and within that the content within. A key question being “do you need to hard code the references or can they be derived from the content?”, if the references can not be derived from the content, then you may need to use a data dictionary to get a reference for each piece of content.

  • Keep in mind the book is known, all you need is the verses within it.

You would then provide an interface to interrogate the content, read, present, list, search etc…

Then, if the solution is designed to be annotated, then there is a way one could create tiddlers with a title like 1 John 4:7 who’s content is automatically provided by splitting the title and extracting the verse from the book and displaying it.

  • You may also be able to split the title through testing if the book exists, then within that the verse.
  • These tiddlers could actually not exist, just their title, a missing tiddler, in TiddlyWiki terminology.
    • If you wish to annotate the verse, you just create the tiddler and use the text field.
  • The Model would not require separate tags or fields to document the book and verse because these are already available in the data.
  • This model could be extended to allow prose to include references which then become links or a way to transclude the named verse in the prose. Perhaps with the help of a function, procedure or parametised transclusions.
#8

Belayed FROM: Lurker 43 … "Discussions have gotten, recently, both tripleth more informative, exciting, and v. obscure (if you are not in the loop) regarding The TW. "

Somethin’ 'appening on reach?
Most interesting.

TT

1 Like
#9

Hi @TiddlyTweeter ,

I ended up using Dictionaries (and a plugin: SearchableDictionary Demo Edition — a demo of the plugin) for my Bible Project - since I could not get Tags to be “fast” - well, for me at least.

See:

You can see that there are a few Dictionaries:

image
image736Ă—444 23.9 KB

  • Each is large - really large - BUT - that’s the only way I could make them work - for my case - - your’s may be different

The “light bulb” moment for me - was help from various folks:

It’s “how” you return the search of your results - that slow down the Tiddlywiki file:

  • Here, I “limit” the return results to 1000 entries - keeping the site “snappy”
    • This is for open-ended searches
  • If you have a drop-down -or- restricted way of looking up items - then you can omit the “limit[1000]” (below) in your returns
    • I have two that are unrestricted - and two with limits to 2500 - - still snappy :innocent: 
<$set name="Word" value={{Main!!z_Bible/Verses/Popup}}>
<$list filter='[tag[/kjv_d/Bible_Plain]keyvalues:value::usevalue<Word>sortan[]limit[1000]]' variable=SearchValue>
<<SearchValue>><br>
</$list>
</$set>

I (personally) prefer to use Dictionaries - for all of my “Read Only” data - - that is - data that never changes – and will work extra hard to convert Read Only items to Dictionaries - as I discovered they are (for my use) fast.

This is not to say there are Tags and other ways to do this - I am not knowledgeable in Tags or other ways - sadly.

Hope this helps.

TwN00b

P.S. My home file is about 450 MB in size and I have one dictionary that’s nearly 200MB - - snappy, zippy and fast - - due to the search limits I place on the results.

(edit - added) P.P.S. The site I link to is a work in progress - I am still figuring out how to add stuff to it - ,…

1 Like
#10

This pretty much confirms my query so I’m marking it as a solution. Dictionary tiddlers will work fine for my use case but it seems others are working on a similar structure so I may retool my method to work for existing projects vs reinventing the wheel so-to-speak. Thanks again!

1 Like