How big can a TiddlyWiki get?

[etardiff]

@medley

What language did you study and what level did you reach? Was it Japanese? I see there are a lot of automatic Anki card creation resources for Japanese.

It was, yes. I was very spoiled for choice of tools in that regard, and I imagine if you’re learning a less “popular” language, it may be significantly harder to find comparable options with an English-language base. (And for anyone who is an English speaker learning Japanese as a foreign language, I’d actually recommend Renshuu (available at renshuu.org or as an Android/iPhone app) over nearly any other option for vocab + sentences!)

I’m not very familiar with the CEFR scale, but I passed the JLPT N1 in ~2013, and I’d estimate I knew about 2500 kanji—enough for day-to-day life or to pick up and read a novel. I wouldn’t say I ever reached native-like fluency; I could read an academic paper in my field, but I certainly couldn’t have written one. And unfortunately I’m considerably rustier now, having been out of the country for nearly a decade.

@medley

If you are comfortable answering, what is each wiki about and how has TiddlyWiki specifically helped you in that domain?

I only have a few wikis I use regularly, mostly for creative purposes—fictional worldbuilding, D&D campaigns, that sort of thing. TW was revolutionary as a writing aid for me, though I admit my prior system (dozens of semi-sorted Google Docs and the occasional spreadsheet) was not winning any awards! But I do love how easy TW makes it to search, connect, and interleave information.

• The filter system is complex but really rewards you for even partial mastery; I use it as much (if not more!) to collect and display related content on a tiddler as I do to find things with Advanced Search.
• I appreciate that TW encourages me to revisit and reuse preexisting content, and thus decrease redundancy and avoid duplicating work.
• The template system lets me create a visually consistent layout for tiddlers of the same type, which makes it easier to find what I need at a glance and to recognize sections that need more work. It also lets me build my own fillable forms to automate field creation and make it easier to “chunk” things into reusable bits.

I do have to admit that “tweaking my TWs” probably constitutes a secondary hobby at this point—and some days, you could be forgiven for thinking that it was my primary one. I’ve joked before that my favorite thing about TW is also my least favorite: You can build a tool that’s tailored perfectly to your needs, organized in whatever way makes most sense to you! But… you do have to build it. The upside here is that, should you decide it’s not perfect, the modular way in which TW stores information means that you can generally restructure it without too much difficulty. (Though, again, I’m sure you can see the danger inherent here…)

I’ve also helped set up a handful of wikis for my partner, who appreciates the TiddlyWiki ethos but not the coding. Off the top of my head, she has…

• a general PKMS-type wiki, probably still competing with Notion and Google Calendar
• a wiki for digitizing family recipes
• a personal Sims 4 wiki, replacing the previous ecosystem of spreadsheets

@medley

Are there any plugins and features that you found extremely helpful?

I see you’ve already got Relink, Trashbin, Favorites, and CodeMirror which are all some of my personal favorites! I also use TiddlyMap (powerful but more fiddly than I’d like) and the draw.io plugin (no problems with this one, I just haven’t used it much).

I installed FontAwesome when I was building my first wiki, but I eventually decided it was bloat I didn’t really need. Instead, I drag the Icon Gallery tiddler into any wiki I make and import new SVG icons from the TW Icons collection as needed.

I think I also had 3click2edit (or something similar) installed at one point, but removed it because I kept triggering it by accident.

Here are some other plugins I’d include on my personal can’t-live-without list:

• Commander - easy batch-processing!
• commandpalette - keyboard-first search; this has all but replaced the Search tab in my day-to-day usage
• Link to Tabs - for easy access when you want to edit a tab or just see how it works
• SideEditor - for real-time editing, or editing while referring to a different tiddler
• Streams - a great configurable outliner, which lets you take “notes” on any tiddler. If I were adding sample sentences for vocabulary terms, this is probably where I’d start!
• Tinka - a simple plugin maker; also a good way to package large, versioned batches of tiddlers to share between your own wikis

And some additional quality-of-life improvements:

• Auto Complete
• Section Editor
• Uni-Link
• Todolist

I’ve also taken a great deal of inspiration (and some piece-meal tools) from @Eskha’s DelphesNotes and @Mohammad’s Mehregan editions, as well as @EricShulman’s TiddlyTools, @saqimtiaz’s Sandbox, @pmario’s Wikilabs, Mohammad’s Shiraz and Utility plugins, and countless other talented developers here.

And in case you’d like to peruse the variety of options out there, the CPL aims to collect plugins from all over the web in a single library.

@medley

Do you know of any public TiddlyWiki I can view for inspiration?

• For an Anki-like SRS, TidMe is the first that comes to mind… though I believe it aims to replace Anki rather than work in conjunction with it as TiddlyRemember does.
• I haven’t seen many public language-learning wikis, but @Charlie_Veniot’s lexicon of Acadian French might give you some ideas!

I’m not sure what else would be helpful, but if you’ve got other examples of things you’d like to see, I’m happy to wrack my brain.

@medley

Especially since I want to reduce the size of my TiddlyWiki as much as possible by not using images and audio files.

This is only really a concern if you’re importing media directly into your wiki, where it will contribute to the total size of the file. If you’re using external media, your wiki will retrieve the image from its local path when needed, and it shouldn’t impact your general performance.

You can also use an $image (or an HTML element that uses the src attribute) to embed an external resource with a URL or a local file path.

[sobjornstad]

I have heard of TiddlyRemember and do use it occasionally but I assume it isn’t the best tool for this purpose. Especially since I want to reduce the size of my TiddlyWiki as much as possible by not using images and audio files.

You say you are going to link each sentence to the note ID of an Anki note. With TiddlyRemember, you could instead link each sentence to a flashcard tiddler, which would contain such audio and images and would automatically sync from there to Anki. I guess if you didn’t want to have this information within your TiddlyWiki at all, this would be pointless, but to me it seems easier than manually copying Anki note IDs.

I know this idea is probably overkill to learn a language but I have been experimenting with different ways of learning.

But I also think this is probably true

As others have said, you would definitely want to use externalized images here. In recent versions, TR will sync these directly into Anki’s media database, which will then sync with other devices, so you don’t need to make them web-accessible or anything.

[linonetwo]

If you are using NodeJS wiki already, you can try TidGi-Desktop , which can open nodejs wiki folder, and is optimized for huge wiki.

If you use TidGi desktop to creat a new wiki, it pre-install the TidMe plugin that is a Tiddlywiki-Native supermemo or Anki. Works best with in-wiki backlinks, and use FSRS algorithm that is better than Anki’s SM2 altgorithm.

By the way, my wiki is 3GB and have many images, and has nearly 30k tiddlers, I use TidGi desktop to open it.

Hope you can join the The 5,000 Tiddler Club – People with the Need for Speed one day.

[medley]

Alright, so I looked a little more into TiddlyRemember. And I think I remembered why I thought it wouldn’t work with my use case. In TiddlyRemember you can’t create your own note types, right? You have to choose between either basic, basic and reversed, or cloze. If I’m mistaken, you don’t need to explain how to do it. I’ll give your demo video a closer watch and look more deeper into it.

Also, if I was to use my method of manually copying note ids over. Do you think I would run into any issues where two notes somehow end up with the same note id? I researched this a little and it seems like note ids aren’t necessarily unique identifiers and that you need to use the unique ID to assure that your notes aren’t overwritten. However, the issue for me was that I can’t search by unique identifier (maybe you can, but from looking at the Anki manual it seems like you can’t). I was planning on having a default field in every tiddler called ‘anki’ and I would copy over the note id of each Anki note that was created for a tiddler delimited by a comma. So that if I wanted to see all the flashcards I created for a tiddler, all I would need to search in anki is something like: nid:170887993025,170887874.

[sobjornstad]

In TiddlyRemember you can’t create your own note types, right? You have to choose between either basic, basic and reversed, or cloze.

Currently true, although the main reason this is the case is that nobody has ever asked for anything else. Also, you can do basically any templating that you would do in Anki within TiddlyWiki prior to pushing the notes to Anki, so it’s more a question of whether you need certain kinds of organization within Anki. What kind of layout are you hoping to accomplish here?

I researched this a little and it seems like note ids aren’t necessarily unique identifiers and that you need to use the unique ID to assure that your notes aren’t overwritten.

They’re the millisecond timestamp of the creation time. So collisions aren’t theoretically impossible, but they should be extremely unlikely. I think Anki also does some remapping when it imports, not 100% clear on that.

[medley]

I still haven’t thought this out fully and don’t even know if this is feasible, but here’s my initial idea:

Phase 1:

• Create scaffolding to make learning new words easier by memorizing 1000-2000 or so words:
  • The types of words I would memorize:
  • simple nouns that aren't reducibe (e.g. boy, girl, bird, cat, sad, hunger)
  • common verbs and some of their conjugations (e.g. sleep, eat, drink, laugh)
  • adjectives (e.g. hot, cold, fast, slow)
  • I don't really care about the classifications of the words (since that changes based on what grammar you follow or what language you are studying). I am more focused on picking words that don't depend on other words. For example, I wouldn't want to learn the word 'security' without having learned words such as 'safe', 'pain', 'good', or 'danger'.

Phase 2:

• In this phase I plan on creating Anki cards that assist in building my passive vocab. The goal of these cards is to be reminded of the vocab and grammar rules that I come across. I plan on putting sentences that I have a shallow but complete understanding of. That is, I have a general sense of the purpose of every word in a sentence.
• Let's say I come across a sentence or phrase. I, first, internally evaluate whether I understand it to some degree or not. Are there any words that I have never encountered? Is the way this word is being used not something I am familiar with? What idea are these words trying to get across?
• This is where TiddlyWiki and Anki come in. I will put every interesting sentence/phrase/saying that I come across in my target language into TiddlyWiki. I consider a sentence/phrase/saying interesting if it fits one of the following criteria:
  • Something that I don't fully understand even in a shallow sense. That is, it includes some word, grammar rule or concept, that I am not ready for at the current moment, but would like to come back to at a later date.
  • A new grammar rule, vocab word, or concept that is novel to me and that I have a shallow understanding of.
• If, for example, I have already come across and put the following sentences into TiddlyWiki/Anki:
• This is a red car.
• The red ball was bouncing.
• I stopped at the red light.
• If I came across the sentence "I want the red boat", I wouldn't put it in the wiki. But if I came across something like "Her lips are red" or "She has red hair" I would. Because in the sentence "Her lips are red" red is being used in a different kinda way here in the sentence (I don't even know what grammar rule this would be called in English). And in the sentence "She has red hair", even though it's being used in the same way as the previous sentences, it's conveying a different idea : someone having ginger hair, not literal red hair.
  • I have been doing something similar to this lately when I am reading. The other day I had to look up examples of how to use 'much less' in a sentence because it isn't apart of my vocabulary. I'm thinking learning a new language is something like this but just on an insane level.
• In an ideal world, every sentence would be its own tiddler and would be tagged with each word in it, and maybe even punctuation or something. (I struggle with what ways punctuation is appropriate to use even though I am a native English speaker because I hated writing and reading for fun growing up.)
• I would only put sentences that I have a shallow but complete understanding of into Anki. This might mean that I will have to make some 'artificial' sentences using words that I know with the external resources. For example if I came across the sentence : 'twenty guests can be accommodated in the hotel's annex' and I didn't know what the word accommodated and annex meant. I would use the external resources like a dictionary and grammar books to create sentences that made sense to me. Like "We have a really nice hotel with an all-you-can-eat buffet, nice rooms, and a pool. You will be well accommodated."If I can't use words that I already know to make a sentence that clearly describes how the word is used, I will consider it beyond my current-skill level and still put it into TiddlyWiki but not Anki. I will have a central tiddler that keeps tracks of words that have been used as tags at least 20 or so times (I am not sure what number is appropriate to use or how to really use filters to do this but I am pretty sure it's possible). This will prompt me to possibly consider trying again to create Anki cards for it.

Phase 3:

If I can create passive cards for the word. I will consider creating active cards that will not test me on understanding but on producing a word or a phrase in context. For example, I could create a card that has a picture of a red boat and have a prompt "This is a _ boat" (maybe I will include a hint like this first letter to avoid accidently using another word that makes sense like 'maroon' or something). This will change as I learn more words, so I will refactor to add more context to make the sentences have a consistent answer.

My goal is to take a top-down and bottom-up approach to learning languages. I will read books on grammar, but I still haven’t figured out how to use them effectively. I don’t think I will memorize or Anki-ify the grammar rules I learn. I will acquire them by creating many Anki cards that illustrate the rule in context and use the grammar books to help form them.

There are obviously other details that I didn’t include or haven’t considered, but this is where I am starting at. I’ve ignored talking about how I will deal with learning the sound system and orthography of these languages, but it would probably be in a somewhat similar fashion.

The main reason, I think this idea is helpful is because it kind of reduces the meta work you have to do. You don’t need to worry about learning this many words, or this rule, or that. You just do what you can and trust TiddlyWiki/Anki to prompt you at a later time when you get better. It might reduce the feeling of ‘you aren’t making any progress’, although it is definitely more intense and tedious.

[Scott_Sauyet]

Winner of the Prize for the Most Interesting Thread by a New User

This has been a fantastic thread to read!

@medley

I was wondering how well TiddlyWiki scales using Node.js instead of a single html file?

I haven’t built large enough wikis to run into any issues with scaling. But I’m guessing for the most part Node won’t make much of a difference with this. Although there is some Lazy Loading capability; I think that will help mostly with tiddlers that have large bodies, especially images. Other than that, Node wikis still load the entire content in one go. Their benefits have to do with resilience, more discrete chunks, easier saving, and other things. But I don’t think they offer much more in the way of per-wiki capacity.

@medley

In an ideal world, every sentence would be its own tiddler and would be tagged with each word in it, and maybe even punctuation or something. (I struggle with what ways punctuation is appropriate to use even though I am a native English speaker because I hated writing and reading for fun growing up.)

An alternative might be to work with virtual tiddlers. I can imagine a central list of sentences, and a startup (or build-time) process that splits those sentences into words. Those words are then either actual tiddlers (if you’ve created them) or virtual tiddlers, that when linked to can list all the sentences which include them.

So the virtual sentence tiddler, s14861 might have the content “She sold the cauliflower to him at the market”, might (after removing the words from some stop list) link to “she”, “sold”, “cauliflower”, “him”, and “market”. And although you already have tiddlers for “she”, “him”, and “market”, the words “sold” and “cauliflower” are for the moment virtual tiddlers. “Sold” might link to four or five different sentences, and “cauliflower” only to this one.

This style might let you include a fair bit more content than a tiddler-per-sentence design. Searching will likely be slow when this grows very large, but plain navigation might be ok.

Note well: I’ve never tried anything like this! I may be crazy here.

[medley]

I looked a little into virtual tiddlers. Are they basically the same as missing tiddlers? Also, in what way would using virtual tiddlers allow me to include more content than a ‘tiddler-per-sentence’ design and by how much do you estimate? I’m currently looking into if I can do this using Anki and Excel? Maybe I can export all my Anki cards of a language deck periodically into an Excel sheet and then use Excel formulas in some way to figure out if a ‘tag’ appears multiple times? I have no idea how I would do this. I have zero experience using Excel.

[XYZ]

This is an example, I know anki, but recently the focus has been on tiddlywiki, where there are more possible ways for you to memorize new words, such as setting tags, editing new notes that you don’t know how to do nothing (see your weak points in the recent list). Using a journal to keep track of the words you get stuck on each day (ask the forum for the descending rank of references in an unordered list), and by building knowledge links (each tiddlers is a sentence you can understand, and building links on each word, gradually forming a knowledge graph), I think that in addition to computer systems, you must know. Using more software to complete a seemingly simple task may take more time, you can do is to stay on the sidelines when using any software, and choose the software that supports the import and export of common data formats as your main learning tool, perhaps you can look at this plug-ins,It’s also a great memory aid
$__plugins_linonetwo_intention-tower-knowledge-graph.tid (63.8 KB)

This is based on the presentation process of wps, your question does not involve excel compound function, but simply by clicking on the page ui icon to achieve, you may use the English version, the icon and the Chinese version is not much different

Alternatively, you can use python to quickly filter duplicate tags in json data exported by tiddlywiki

[Scott_Sauyet]

@medley

I looked a little into virtual tiddlers. Are they basically the same as missing tiddlers?

They are a subset of the missing tiddlers, ones that you specifically handle with a templates that can infer from the title enough information to present important content.

Imagine a data tiddler that looks in part like this:

...
s14860: He never believed that nonsense.
s14861: She sold the cauliflower to him at the market.
s14862: There is simply never enough time to do all the things you would like.
...

When you try to load the tiddler titled s14861 a template notes that this is in the right format for a sentence key, and presents the sentence, with the words linked to word tiddlers. Some of those may already exist, but others may still have to be processed. For the latter, you will get back a list of links to sentences. The former will show this list of sentences after all the word content you’ve created.

@medley

Also, in what way would using virtual tiddlers allow me to include more content than a ‘tiddler-per-sentence’ design and by how much do you estimate?

Using the above, this line:

s14861: She sold the cauliflower to him at the market.

is 55 bytes.

A tiddler in a reasonably compact format might look like this:

title: s14861
tags: word/she word/him word/sold word/cauliflower word/market
created: 20240417165113078
modified: 20240417170039340

She sold the cauliflower to him at the market.

That’s 179 bytes, and might grow slightly if rendered in JSON… You can reduce this if you have a more compact form of tag-names for words, but not by much. It’s also likely to grow with additional built-in fields, such as revision and bag.

But we’re talking approximately a 70% reduction.

Again, I don’t know if this is feasible for a large lexicon; I’ve never tried to scale up that far. But this might offer you the ability to scale further (3+ times as many words!) than in a tiddler-per-sentence design.

@medley

I’m currently looking into if I can do this using Anki and Excel?

I know nothing about Anki, so I can’t help there. But others have been offering advice about that.

[john.edw_gmail.com]

I use the node.js version to host 8 wikis currently. My largest wiki currently has 24,093 tiddlers and the backup file on github is 130mb in zip format.

I create and add 1 long format tiddler (1mb-1.7mb) and 72 smaller echart data tiddlers 4 times a week.

I am configured with lazy loading tiddlywiki --listen root-tiddler=$:/core/save/lazy-all which results in 6.58mb of data transfer on initial load.

[Springer]

@medley

Create scaffolding to make learning new words easier by memorizing 1000-2000 or so words:

For what it’s worth, I think there’s a language-learning methodology called n+1 that develops exactly this kind of idea. (Probably you already know this.)

For Japanese (Waving to @etardiff here, with big respect for the N1 success!), I used JALUP, which is a marvelous sequence of carefully-crafted sentences that depends on a base of exactly 1000 words treated as “primitives” (for which translation/English explanation was provided). After that list is mastered, the program leads the learner through a sequence of levels, each including a thousand sentences, where each sentence introduces exactly one new word, or new sense of a word, or new pronunciation-compound, and then (This is the cool n+1 part!) explains that unfamiliar word/usage only with the use of the already-mastered words, and all completely in Japanese. So, the first 1000 words get you into the pool, and then you’re just swimming in the target language. And — unlike with a kludgey translation-based system like duolingo, which is especially disastrous for Japanese anyway — an intermediate learner beginning from any language could learn through Jalup with equal success, once that first set of 1000 is familiar.

Of course, some of the J-J definitions are roundabout. For example, the word for finger 指 or monkey 猿 is not so much “defined” in robust dictionary-sense as gestured-toward: hand has five 指. 猿 is small cute animal, looks like human, lives in trees (or some such) — all leveraging Japanese that learner has already learned, all in a careful SRS playspace. What I discovered was that the initial encounter with a word put it in one kind of understanding-space, but when that word was leveraged in the definition of another word — that’s when the real synaptic connections started to happen.

It was so much more sophisticated than home-grown Anki (because of the carefully-crafted path of example sentences, but also because it included a recording of native speaker reading each sentence with standard pitch accent). It also requires so much less effort to set up, and to sit down and just “follow the yellow brick road” rather than generate new cards at the end of a long day. So I did drop Anki when I found Jalup. But I’m excited to see tools like this being developed in TiddlyWiki!

[caderoux]

Our largest TW looks like it’s 45MB (17730 tiddlers). I have many others that are 15MB each (3100 tiddlers in one). To put it in perspective, those are TWs produced by our testing process which had to be broken up just because the single file of all the test results got a little unwieldy to quickly load and look around at results. Some of the tiddlers are large tables of tests, others are tables transcluding other tiddlers where each tiddler is a test result. It all varies by need. Some of the test results are human-verified, so a person goes in and checks off reviewing certain cases, and then that file is saved and goes back through the process. Others are only pass/fail where either the code or the test has to be modified in the testing system until they pass.

In addition we use it to generate our user manual/data dictionary, which is about 18MB (1000 tiddlers).

[medley]

What do you mean by ‘our’ largest TW? Isn’t there an example of a TW that contains 70,000 tiddlers?

[bluepenguindeveloper]

@Springer

I used JALUP

Looks like Jalup is semi-defunct while the Anki decks are still pricey. Too bad, I was looking for something that introduced vocab and grammar in careful order like that (and wasn’t overly pricey).

Context: I am looking to learn Japanese, and because I know Korean already, I expect to be able to learn the grammar and the Sino-Japanese vocab fairly naturally and quickly (without most of the difficulties westerners typically face with understanding Japanese grammar). (Unfortunately, I don’t know many kanji yet, so I will still have to learn those a little more slowly like everyone else - though a system like WaniKani might help speed things up a little bit.) I haven’t found a suitable resource yet, but I would like to be able to use some resource that presents grammar and vocab in a logical order, but in a way that I could largely speed through (by making connections to Korean to learn quickly), so that, as soon as possible, I could start learning by going into the wild (internet “immersion”, the closest I can get to immersion at the moment), while building kanji knowledge in parallel and constantly reinforcing my kanji knowledge as I come across the kanji in the wild (which might make an SRS mostly unnecessary even for kanji).

And yes, I too was looking to use TiddlyWiki to take notes on Japanese, though I’ll save that discussion for another post sometime.

[LoryMoney]

@medley

I am not even sure how well Anki itself scales, but I am assuming it scales well since medical students and other language learners use it.

I highly recommend you check this video out, same goal as yours with Anki based approach:

learning 10,000 German Words in 2 Years with self-study

[caderoux]

“Our”, meaning our largest within our release process (by size) for our cardiovascular reporting data warehouse product, which has dozens of TW for testing results as well as a data dictionary/user manual. Some of those were previously combined in one, so could easily have been 100MB or more before I split them in the build process. Given how the TW are produced, I believe there are at least 250k tiddlers being generated, split across all those files, and it’s about to get a lot larger as I start to fan the validation testing across more versions. Honestly, it’s such a big process, I probably need to get better control of the size of the files being generated, since I really often have no idea how quickly that grows as new data is curated into the data mart, because they do stick around on every build and consume space.

I do think once they start to get large the performance of just loading those files from our TeamCity build server is actually the main bottleneck, not the browser actually processing the HTML. And while they’ve been slow to load and view, it is not as common for it to be slow to pull up individual tiddlers since I split up the files.

[TW_Tones]

I have not read this whole thread, yet. But one thought I had for language learning to leverage the features of modern browsers, especially with translation, dictation and reading. Use tiddlywiki as the interface/repository.

• Read some text out loud (speech to text) and see if it is recorded as it was written.
• Type some text and have the browser read it out.

You can do this both in your native language and the one(s) you wish to learn. Sure there is a lot more to learning a language but this does I believe fill a gap in the process. Material to read and spoke, spoken and read, listened to etc… can be found in your preferred language all over the internet.

On “How big can a TiddlyWiki get?”

As long as a piece of string!

Just design your solutions so that if one day you can’t optimise your tiddlywiki enough you can export your content for use elsewhere.

[medley]

What things do you think I need to take into consideration when designing my TiddlyWiki, so that it can be exported?

[TW_Tones]

Well if you break your data into tiddlers, with their content in the text field, or even spilt into fields, and can tell the difference between them, like this is a contact, this is a company… then there is almost an unlimited number of ways you can export your content to get it out of tiddlywiki,

• The rule is that your data remains logically structured with no missing information in TiddlyWIki then the ways you can export the data are effectively unlimited.
• The same rules apply to media images/audio/Video but their size may add complexity.

Unless you know what you want to export it to already, you can leave it until needed.