JSON Filters of Data in an Array

#9

If you can share the CSV I can give you back a plugin to get you started?

In the mean time I want to see how hard it would be to query the csv directly.

#10

I’ve added it to my original wiki linked above in the tidder MtGQuery.csv

https://mtg-sketchbook.tiddlyhost.com/#MtGQuery.csv

#11

This definitely seems like a “right tool for the job situation”! I tried this but got impatient, stopped it, and wrote a function in my Excel workbook to wrap the names in double square brackets instead, copied the cell range to Notepad (Tiddlywiki wanted to treat the paste as an image instead of text) and copied the text to a new tiddler named MtgCardNames. The rest worked great and the autocomplete is plenty fast enough for my needs. Thanks!

Next up: getting the rest of the JSON or CSV data into tiddlers that I can use to store the rest of the information I want about the cards. I think once I understand the best way to create the tiddlers in bulk with the data stored in fields, I can create a default view that will show the information I care about and reference other tiddlers that reference that card.

#12

As far as I can see that is not CSV

#13

@technome … Do you control how the JSON tiddler with the cards is created?

If yes, you could try to create a JSON file, that can be imported as single tiddlers, per card, to run tests.

Using json-operators is relatively complex, even if the structure is relatively simple. TW is optimised to be used with tiddler. So it would be worth some tests.

If "name": will be changed to "title:", it should be possible to import it into TW.

-m

1 Like
#14

It attached it as a binary file (not sure why, maybe because my system has CSV associated with Excel?) I have deleted and re-added a new tiddler with the same name and copied the contents in.

I’ve also created a rough sketch of what I would like each card to look like if I can create all 29,667 tiddlers. Ideally this would use a default template for tiddlers with the “Card” tag.

1 Like
#15

I do have control over it, as I had to filter out a lot of data I didn’t want to import.

Is the JSON format for import documented somewhere? I know I’ve seen examples of what it looks like but I am struggling to find an example now. I would want to set the title and multiple fields along with a tag of “Card”

#16
  • I have imported it into a wiki with JSON Mangler and another required plugin.
  • I changed the type to application/csv known in the wiki with JSONMangler installed
  • Using the import tools that appear on the CSV because of JSON Managler I imported it with no setting changes
    • Note there is a display issue but its working.
  • Exported it as a JSON file, The result is this 84kb file with 250 items? Data_MtGQuery.json (83.4 KB)

This only took a minute and no timeouts. I did it on a single file wiki at file://

Due to load time on mtg-sketchbook.tiddlyhost.com I recomend a splash screen

I could improve the import based on what you want the tiddler title to be. eg MTGcard-name-number

1 Like
#17

See: How do TWs JSON Formats Look Like – It basically is the first format, that is described.

1 Like
#18

in addition to that
size wize

i imagine using
a key/tiddler to link field name text to number

title:obj-key
1: image_uris.normal
2: legalities.commander
3: mana_cost
4: set_name
5: type_line

put the data in
numbered field’s (if that is possible?) should lessen size of repeated key name’s reducing final size

1 Like
#19

Update: I swear that when I answered there were no other answers! I must have been blind. Sorry, this post doesn’t add much to the discussion.

This wiki has over 32K tiddlers, with no noticeable slowdown. I would suggest that it’s simply worth trying. You could just import the above if you converted slightly:

  {
    "title": "A Realm Reborn",
    "tags": "MtgCard",
    "image_uris.normal": "https://cards.scryfall.io/normal/front/d/1/d1af74e4-38d5-44b5-85e1-4d13f6970453.jpg?1748706495",
    "mana_cost": "{4}{G}{G}",
    "type_line": "Enchantment",
    "legalities.commander": "legal",
    "set_name": "Final Fantasy"
  }

That is, change name to title and add a tags parameter to identify the sort of data involved. The tag makes it much easier to query, but is not required. title is required; it’s the unique key. If any of these names are likely to conflict with other tiddler titles you want to have, then you can add a common prefix, such as Card/A Realm Reborn, in which case, you might also want caption: A Realm Reborn to have the familiar title in lists.

(I would also suggest normalizing the field names to using a single separator. But only for cleanliness; it’s not essential, and it won’t hurt anything to have image_uris.normal, but it can get difficult to remember. Most peoples settle on kebab case or snake case, but anything will do, so long as there are no spaces.)

If this doesn’t slow things down, and I don’t think it will, then you should find that querying your data is much easier than it would be in JSON. You can see in a current thread that there are discussions about how to make it easier to query JSON tiddlers, and you’ll see me arguing that it might not be worth doing. Right now JSON data is definitely a second-class citizen in TW, and there’s a very good chance that this will long remain so.

1 Like
Why is JSON data a second-class citizen in TW?
#20

Agreed. This was a quick and dirty PowerQuery of the JSON data. I’ll come up with a more robust ingestion workflow that cleans this up and standardizes.

#21

I have a test wiki setup here: https://mtg-test.tiddlyhost.com/

All cards imported and loads in about 4 seconds on my machine, perfectly reasonable. I’ll consider this a solved problem. Thank you!

In an effort to reduce the size, I am generating the Scryfall URI from the card ID. However, I discovered that the URI is prefixed with the first two characters of the ID, presumably because they store the images in static folders. For example:

https://cards.scryfall.io/normal/front/a/4/a471b306-4941-4e46-a0cb-d92895c16f8a.jpg

  • The ID for this card is a471b306-4941-4e46-a0cb-d92895c16f8a
  • The first part of the URI https://cards.scryfall.io/normal/front/ is static
  • After that it is followed by id[0], a slash, and id[1]

I could add a field and pre-process this while avoiding storing the whole URI, but I’m struggling to find a string function to give me the first and second characters of the string. I’ve attempted to create a macro ($:scryfalluri in the new wiki) but I’m getting no output or error messages. Hopefully there’s a built-in macro or function to do a substring operation that I’m missing.

#22

I disagree, showing me an example that was performant was very useful. Thank you!

1 Like
#23

Since those characters are separated by “/”, you might try using a split[/] filter operator.

Let’s assume you have the complete URI in a variable named <<uri>>…

<$let path={{{ [<uri>trim[https://cards.scryfall.io/normal/front/]] }}}>
<$let  firstchar={{{ [<path>split[/]nth[1]] }}}>
<$let secondchar={{{ [<path>split[/]nth[2]] }}}>
<$let   filename={{{ [<path>split[/]nth[3]] }}}>

For convenience, the 1st line removes the static part of the URI so the rest of the filters only need to deal with the remaining part of the URI.

-e

1 Like
#24

This doesn’t do what I’m looking for. I need the first character from the ID:

<$let cid="fa7a883f-e180-48ce-b7d4-cded2f74c323">
<$let firstchar={{{ ... }}}> // Should look at 'cid' and return 'f'
<$let secondchar={{{ ... }}}> // Should look at 'cid' and return 'a'

* "<<cid>>"
* "<<firstchar>>"
* "<<secondchar>>"
#25

Try the following (and note that you can combine multiple variable definitions into a single $let widget, where subsequent definitions have access to any variable that has been previously defined):

<$let
	cid="fa7a883f-e180-48ce-b7d4-cded2f74c323"
	firstchar={{{ [<cid>split[]nth[1]] }}}
	secondchar={{{ [<cid>split[]nth[2]] }}}
>
2 Likes
#26

That was the last piece I needed. I now have the following in my default view:

\define scryfalllink(scryfalluri) [[Scryfall|$scryfalluri$]]

<$let
    baseuri="https://cards.scryfall.io/normal/front"
	cid={{!!cid}}
	firstchar={{{ [<cid>split[]nth[1]] }}}
	secondchar={{{ [<cid>split[]nth[2]] }}}
    imguri=`$(baseuri)$/$(firstchar)$/$(secondchar)$/$(cid)$.jpg`
>

<% if [all[current]tag[MtgCard]] %>

<$image source=<<imguri>> />

* Type: {{!!tl}}
* Mana Cost: {{!!mc}}
* <$macrocall $name="scryfalllink" scryfalluri={{!!sfuri}}/>

<% endif %>

I still need to wrap my head around when I can and can’t pass a variable to wikitext/macros/filter expressions but I’m getting there. Thanks to everyone for your help!

#27

Just keep in mind that using shorthand forms like <<>> {{}} [[]] that it is harder to provide parameter values from other places ie other <<>> {{}} [[]] If you want to call a function/procedure/transclusion using the long form givers you more power eg “<$macrocall, <$transclude, <$link, <$transclude $variable”.

However if you are writing the macro/procedure or function you can choose what those parameters are, and can often hard code items, or accept names eg varname, fieldname then use that to obtain the value eg [<varname>getvariable[]], [get<fieldname>] allowing the reference not the value to be passed into your code.

#28

I did try to build a wiki out of your JSON, and it also seems perfectly performant:

http://scott.sauyet.com/Tiddlywiki/Demo/MtG/v1/.

I used my personal starter edition and converted your JSON to a new collection of Card and Set tiddlers.1 I added very simple templates for Set and Card tiddlers, and that was it.

The resulting file is 8.7MB. It loads for me, over a reasonably fast connection, in about 3 seconds. There are no noticeable performance issues in navigating. I have not tried any complex queries yet, though.

Feel free to ignore it, to take it outright, or anything in between.




1 The (Node.js) code looks like this:

See code 
const cards = JSON.parse(require('./MtgCards.json')[0].text)

console.log(JSON.stringify([
  ... cards.map(card => ({
    title: card.name,
    tags: `Card [[${card.set_name}]]`,
    'image-url': card['image_uris.normal'],
    'mana-cost': card.mana_cost,
    'type-line': card.type_line,
    'legalities-commander': card['legalities.commander'],
    'set-name': card.set_name 
  })),
  ...[... new Set(cards.map(card => card['set_name']))].map(title => ({
    title,
    tags: 'Set'
  })),
  {
    title: 'Set',
    tags: 'TableOfContents',
    caption: 'Sets',
    text: `<div style="column-width: 15em;"><<list-links filter:"[tag[Set]]">></div>`
  }
]))

which I stored as index.js in a folder alongside the result of downloading your MtgCards.json tiddler, and I simply ran it as node index > cards.json and dragged the resulting cards.json file onto my empty starter edition.