Simple Table Handling

(I split a few posts from Issue with $text widget, and calculations from its results, inside a procedure, as they might be worth discussing on their own.)


I wonder if it would make more sense to store the table in a tiddler and write a quick procedure to extract values from that table based on row- and column-headers. The idea is to have a tiddler, say skills, with content like this:

# Skill-cost table

*   E   A   H   VH
-3  -   -   -   1
-2  -   -   1   2
-1  -   1   2   4
+0  1   2   4   8
+1  2   4   8   12
+2  4   8   12  16
+3  8   12  16  20
+4  12  16  20  24
+5  16  20  24  28

and a macro/procedure that works like this:

Wikitext Result
<<table-lookup skills +2 H>> 12
<$macrocall $name="table-lookup" tiddler="skills" row="+4" col="VH"/> 24
<$transclude $variable="table-lookup" tiddler="skills" row="-1" col="A"/> 1

You wouldn’t have to pass plain text; any variable or field references should also work for the latter two syntaxes.

I created a tiddler to do this. It’s JS, though, not wikitext. I’m sure we could redo it in wikitext, but JS is still much quicker for me. The important code looks like this:

exports.run = function(tiddler, row, col) {
  const text = ($tw.wiki.getTiddler(tiddler) || {fields: {text: ''}}).fields.text;
  const lines = text.split(/\r?\n/).filter(Boolean).filter(r => !r.startsWith('#'))
  const headers = (lines.length ? lines : [''])[0].split(/\s+/)
  const records = lines.slice(1).map(s => s.split(/\s+/))
  const x = headers.indexOf(col)
  const y = records.findIndex(r => r[0] == row)
  
  return (x > -1 && y > -1) ? records[y][x] : null    
};

This does not handle the Extra row of your table – I wasn’t really sure if that was part of the main lookup table or something additional.

I feel like this technique would make many such lookups easier to deal with, both quicker to write and easier to maintain.

This is mostly a proof-of-concept. There are a number of weaknesses:

  • There’s no way for any of the keys to include spaces. That could definitely be fixed, using TW’s double-square brackets, but it would add some implementation complexity.
  • It requires that initial * as a placeholder on the first row.
  • It does not signal a missing value or even a missing tiddler, only returning a blank.
  • It would not allow you to use # as a row header (as it’s used to indicate a comment.)

But before spending any time on cleaning up such things, it’s necessary to decide whether this is a useful idea at all. I’d love to hear feedback from the OP and others.

You can try this by downloading this link, dragging it to any wiki, saving and reloading (because it’s a JS module) and then opening the Skills Demo tiddler.

table-lookup.json (2.3 KB)

1 Like

Interesting :slight_smile:

In my use case, the extra of the original table is important to handle. Also, the important part of the table from which one would startup a lookup would be the difficulty, not the result on the left. Then, you proceed to look the left column to determine what end score you want, and that gives you the amount of character points you need to spend to get that skill at the level you want; but in my TW implementation, I only deal with the character points, the rest is supposed to be computed. So the only value manipulated by users is the amount of character points.

Also, this particular table will be used on only two entity types that work almost exactly the same: skills and spells (spells are considered skills and work the same way, but all spells have a difficulty of either Hard or Very Hard and rely on the IQ attribute, whereas skills can have any of the four difficulties, and rely on any of the base stats, ST, DX, IQ or HT). So I basically wrote the logic only for skills, and it’s called a few times per character tiddler rendered, probably not that big of a deal.

I also have never written a single line of JS code :stuck_out_tongue_winking_eye: (did some C, C++ and C# decades ago, but that’s about it). I’m actually trying to fulfill my scope without ever having to rely on JS, for s%$# and giggles mostly and see if I ever hit a wall with just pure TW (which I still find confusing, but increasingly awesome). Thanks for the approach suggestion though!

Yes, I did entirely misread what you were doing. (I’m afraid I know nothing about role-playing games or their setup.) But an extremely similar technique would work for your actual case as well. The point was whether this would be something easier to maintain with visual table and a standardized handler rather than in custom code. It may well not be. You might well prefer the code. I simply think this alternative is worth exploring. Here is a similar table-scan function

Wikitext Result
<<table-scan skills VH 17>> +2
<$macrocall $name="table-scan" tiddler="skills" col="H" val="2"/> -1
<$transclude $variable="table-scan" tiddler="skills" col="A" val="25"/> +5

with the important code:

exports.run = function(tiddler, col, val) {
  const text = ($tw.wiki.getTiddler(tiddler) || {fields: {text: ''}}).fields.text;
  const lines = text.split(/\r?\n/).filter(Boolean).filter(r => !r.startsWith('#'))
  const headers = (lines.length ? lines : [''])[0].split(/\s+/)
  const records = lines.slice(1).map(s => s.split(/\s+/)).reverse()
  const x = headers.indexOf(col)
  const y = records.findIndex((r, i, a) => Number(r[x]) <= Number(val))
  
  return (y > -1) ? records[y][0] : null
};

and the same skills tiddler as before:

# Skill-cost table

*   E   A   H   VH
-3  -   -   -   1
-2  -   -   1   2
-1  -   1   2   4
+0  1   2   4   8
+1  2   4   8   12
+2  4   8   12  16
+3  8   12  16  20
+4  12  16  20  24
+5  16  20  24  28

I think of the JS in this case as a proof of concept. I’m quite sure that it wouldn’t be too hard to rewrite this in wikitext… But before I bother, I’d want to know if this is a generally useful technique to others. We have alternatives to this plain text table in JSON. That would be more flexible. But this might be easier to use for a number of cases.

This table-scan is a little less general than the earlier table-lookup, because instead of searching for an exact match, it’s looking for the last value no larger than our target. We can probably abstract that as well, but a generic solution might get uglier and uglier.

I didn’t do anything with it, as I don’t see how it’s being used in your code sample.

table-scan.json (3.9 KB)

Hi @Scott_Sauyet, this is possibly better split off as it pertains to your question of “store in a table”. I don’t want to read more into it than was intended, but I’d love to have table as a native format that you can have filter operators work on. We have data-tiddlers (1-d) that have their own filter operators (though lacking without @pmario 's key-values plugin) and now we at least have JSON handling with their own filter operators.

My background is in data-science and am regularly copy-pasting in 2-d data (tables/dataframes) from R or Python or Excel and always have to pick between a few bad options. Sometimes it’s worth the effort to convert to JSON, often I end up doing a data-tiddler format and have the values be some delimited format that I parse in second stages. I’ve written R functions to “export” data in this way, but it’s clunky and slow processing in TW for obvious reasons.

I’ve followed TW for many years now and have seen glimpses of this being discussed around CSV handling. There’s a CSV tiddler type according to https://tiddlywiki.com/#ContentType though I don’t actually see it anymore on the drop-down. There’s a macro around it https://tiddlywiki.com/#csvtiddlers%20Macro to convert to CSV format, and ways to export to CSV.

In some of the github commentary @jeremyruston has mentioned possibly expanding things

Back in 2014 (link)

And in 2022 (link)

I imagine the tiddler storage that’s used against all of the normal filter operators is essentially a 2-d format, in my dream scenario, we could trick the filter operators to use a single “csv tiddler” as the data source and use all the regular filter operators against it.

Anyways, if anyone has solved this otherwise via plugin or something and I missed it, please let me know.

1 Like

While this is definitely a step in that direction, it’s a relatively small step. The tables we could support with this approach are fairly limited. If we wanted to allow spaces in headers or values, it would add some significant complexity.

This technique is limited to finding the value at the intersection of a column header and a row header. That’s an important case, but we probably also want to support extracting a row or column by header-value or by index, to iterate over rows or columns. and maybe some more generic version of the table-scan above.

So I wouldn’t aim for having this included in the core.

Still, I wholeheartedly agree that it would be useful to have built-in handling of data tables. I way too often end up converting a perfectly reasonable format of its own into a less comfortable TW format. I think that CSV is likely that format. While JSON can handle grids, that’s not its strength. Clearly dictionaries are not robust enough. One other nice thing about CSVs is that we could easily choose to format them to look similar to my samples above, just adding commas. Sure it takes extra whitespace, but it does make for a readable format:

*,   E,   A,   H,   VH
-3,  -,   -,   -,   1
-2,  -,   -,   1,   2
-1,  -,   1,   2,   4
+0,  1,   2,   4,   8
+1,  2,   4,   8,   12
+2,  4,   8,   12,  16
+3,  8,   12,  16,  20
+4,  12,  16,  20,  24
+5,  16,  20,  24,  28

or

*  ,  E  ,  A  , H  ,  VH
-3 ,  -  ,  -  , -  ,  1
-2 ,  -  ,  -  , 1  ,  2
-1 ,  -  ,  1  , 2  ,  4
+0 ,  1  ,  2  , 4  ,  8
+1 ,  2  ,  4  , 8  ,  12
+2 ,  4  ,  8  , 12 ,  16
+3 ,  8  ,  12 , 16 ,  20
+4 ,  12 ,  16 , 20 ,  24
+5 ,  16 ,  20 , 24 ,  28

So if we were to consider this, presumably it would start as a plug-in with an eye to adding the functionality to the core. What sort of API would we want for the plug-in? Would we want to support spreadsheet style references (such as C12 or AH248), or only row-column ((12, 3) or (248, 34)). Would we want the sort of row-header/column header intersection lookup I mentioned above <<lookup skills +3 VH>> -> 20? All of the above? What procedures/macros/functions/operators would be essential? Which others would be nice to have? Would we want a specific edit mode for our tables? What else is crucial? What else would be on the wish list?

TW already understands type: text/csv. So there is a parser in the core, that can handle CSV and also TSV type content.

The default output of type: text/csv is a table. So the values are “stored” in the internal parse-tree. There is no conversion to a fast internal store.

For tiddlers with that type the same arguments are valid as for TW data-tiddlers of type application/x-tiddler-dictionary. Jeremy considers data-tiddlers as a conceptual mistake.

text/csv tiddlers are data-tiddlers.

On the other hand JSON filter operators have been implemented lately. IMO basically to be able to use content fetched with tm-http-request using filter operators.

Using CSV table-formatting, there are 2 usecases that will immediately be requested by users.

  • Sorting by column
  • Edit in view mode

And a bit later:

The main problem here is, that TW is not optimized using data-tiddlers in a performant way. Except, if they are 99.99% read only.

Yes, @stobot noted that this works. But it’s a very different use-case. Displaying CSV data as an HTML table certainly has its uses, but I don’t think that will help us use it as data.

Can you expand that? I’m not following.

Any more info on this, or a reference to a related discussion?

Do you think this is a fundamental issue or something that could be improved with a reasonable amount of work?

In TW a tiddler is a key / value store, which internally is a JavaScript object. In TW we call the keys → fields. So if we transclude {{tiddler!!title}} we request the value of the title-key.

The advantage of tiddlers is, that there are several caching and indexing mechanisms in place, that make standard filter operators as fast as possible.

That is not the case for data-tiddlers. The data-tiddler is cached, but the containing data is text only, and needs to be evaluated, every time it is used. That’s slow.


You mentioned my keyvalues-plugin. I did create it, to be able to easily output the key and the value within the same list-widget, which is not possible with the core getindex operator.

I am not sure, if the plugin uses the latest filter caching optimizations. (Will have to have a look)

See the whole discussion about the keyvalues-plugin at Github Implement a data tiddler keyvalues operator by pmario · Pull Request #3971 · TiddlyWiki/TiddlyWiki5 · GitHub

There are several other, similar comments, that I can not find at the moment. You’d probably need to search GitHub PRs or issues.


Jeremy wrote:

Hi @pmario this is interesting, and certainly addresses one of the many limitations of working with data tiddlers

As you may recall, my view on data tiddlers has evolved over the years, and I broadly no longer think they were a good idea. The present implementation is limited, and would take a lot of code and complexity to make it universal. The implementation of data tiddlers is spread throughout the core code, and hard to work with. I think that the whole idea was misguided: it was an attempt to avoid the proliferation of tiddlers, but I now think that that is a non-goal.

Against that background, I think of data tiddlers as a candidate for deprecation in v5.2.x, and so I’m reluctant to expand support at this point. It looks like this operator could be delivered as a plugin, which I think is what I would advise at this point, although I’m happy to discuss.

I think the whole conversation there is interesting. But as I wrote that was way before we got the JSON operators.

@jeremyruston – Any new views about this one?

That’s a systemic problem. Data tiddler handling is not as optimized as tiddler handling. So tiddlers should be preferred.

The main problem is, that optimizing data-tiddlers will increase code complexity in the core quite a bit. Mainly by duplicating tiddler functionality, which is a no-go … So IMO if we can convert data tiddler content into a tiddler-store, we could use the existing optimized mechanisms.

Thank you. That does clarify things. It doesn’t stop my desire for such a system, but I will think about how to do something similar with plain tiddlers. I certainly can do it with the techniques we’ve discussed on the SQL threads. But I was looking for something simpler to use.

QQ: am I missing an operator that is similar to either indexOf or findIndex from JS’s Array.prototype? That would definitely make things easier.

We use $tw.utils.each(object,callback) which can iterate over arrays and object keys executing the callback function.

That’s very handy.

There also is a $tw.utils.LinkedList() which implements a .pushTop() which is faster than array the old TW array .pushTop() function. LinkedListss are used by some filter operators.

Sorry, I wasn’t specific enough. I was looking for a wikitext operator. I was thinking about how the code I used above could be converted to wikitext. I actually started writing it in wikitext, but didn’t find that operator. I’m sure I could write a recursive version of it in wikitext, but that seemed like I was getting afield, so I switched to a JS module.

The code works on matching the column by finding the index of header that matches our input column, then finding the matching row and looking up that index. But I didn’t find a quick and easy way to do so. Am I missing it?