Searching JSON records

#1

Does anyone have a good example of providing a UI to search the records within a JSON array? I have over 2000 name/address/latitude/longitude/metadata records that I would like the user to be able to search. The system will use a few fields from the chosen search result to auto-populate certain fields on a new tiddler. While I could make these 2K records into tiddlers, that feels like cluttering up the tiddler space. So I would like to simply store them in a JSON tiddler. But I don’t know how to search the JSON data in wikitext.

Is this a solved problem? Are there examples around?

#2

Pay attention to this KeyValues — advanced data-tiddler functions

#3

That looks like a great building block. I’ll try playing with it tonight.

Thank you!

#4

Or maybe not. I was thinking of full-fledged JSON array,

[
  {
    "id": "123"
    "first-name": "Joe",
    "last-name": "Biden",
    "address": "1600 Pennsylvania Ave, Washington DC, USA"
    "latitude": 38.897957, 
    "longitude": -77.036560,
    "more": "fields here"
  },
  {
    "id": "456"
    "first-name": "Anthony",
    "last-name": "Albanese",
    "address": " 10 Downing St, London SW1A 2AA, United Kingdom"
    "latitude": 51.503368, 
    "longitude":  -0.127721,
    "more": "fields here"
  }
]

Or I could structure it as an object:

{
  "123": {
    "first-name": "Joe",
    "last-name": "Biden",
    "address": "1600 Pennsylvania Ave, Washington DC, USA"
    "latitude": 38.897957, 
    "longitude": -77.036560,
    "more": "fields here"
  },
  "456": {
    "first-name": "Anthony",
    "last-name": "Albanese",
    "address": " 10 Downing St, London SW1A 2AA, United Kingdom"
    "latitude": 51.503368, 
    "longitude":  -0.127721,
    "more": "fields here"
  }
}

But the point is that I want to search the name fields to find a match. I’d like to use the address, first-/last-name, and latitude/longitude in building a new tiddler when the user searches “alban”, finds and selects the record for “Anthony Albanese”.

If needed, I will add a full-name field to simplify the searching.

I’m not sure that plugin will add anything to help with this. But I’ll check it out.

#5

Of course you can search the text field of a JSON to find anything but on reality we tend to want to search data structures according to their inherit structure or the records they define.

To do this we need to reference the content using indexes and operators that help us query individual records inside the data. It gets more complexity if you want to edit records.

Personaly in cases like this I tend to use the “JSON Mangler plugin” to convert such data into a data plugin containing each record as a shadow tiddler and used standard tiddler and fields to query the data.

  • you can edit records if you want but maintain the original data in the shadows.

The truth is a data plugin uses around the same bytes as the JSON tiddler and is in fact almost identical.

#6

That makes sense. As shadows they won’t clutter up the cognitive space as much, but I can still use normal techniques on them. And for my case, I don’t need to use the data mangler, because I can generate tiddlers for this data as easily as I can generate JSON.

Yes, it wasn’t byte-count that I worried about. It was the intellectual weight of so many tiddlers that I would have to deal with. This suggestion should minimize that.

Thank you – once again! – for your thoughtful and sound advice!

#7

In a plugin you can choose if they are system tiddlers or not, and if not (ie no $:/ prefix), you can also prefix them with the data set name eg; $:/contacts/Fred Jones and extract Fed and Jones from the tiddler or remove prefix :/contacts/

  • ie hide them behind the $:/contact namespace.
#8

Yes, I have unique keys from a system source that I will probably keep, so this lets me simply name them using that: $:/voter/id/123 and $:/voter/id/456. It will also make it easier to normalize the addresses, so that $:/voter/id/123 has an address-id field with a value such as $:/address/id/789. This will simplify those places where I want to discuss a household.

I think this will work very well. Thank you!

#9

Hi Scott,
It seems you can control the structure the data is exported from a different system. So imo you should export it as a format, that TW can import as tiddlers.

I did describe the different JSON formats, that TW can handle out of the box at: How do TWs JSON Formats Look Like

IMO you can use any format that contains several tiddlers and is described there.

Even the internal wiki store would make sense.

  • You can use empty.html
  • Create eg: data.store

and then

copy data.store+empty.html new-wiki.html

The new-wiki.html should be a valid TW with single tiddlers now. So you can use the existing possibilities and filters we know.

If you do not want, that the users can see you data tiddlers, you can use the plugin format. So all your data-tiddlers will be shadow tiddlers.

So your filters have to start with [all[shadows+tiddlers]...]

Just some thoughts

#10

Yes, that’s the conclusion I reached in discussion with @TW_Tones. It’s not so much that I can control the format that system generates (usually comma/tab/pipe delimited records), but that as a programmer, I can transform it however I want after the fact. So I will do this as a plugin for now, with the tiddlers behind a system namespace. I do something similar in my Periodic Table work. (I swear one of these days I’ll get back to that!) so the techniques are clear enough to me.

I read that when it came out. It’s a great summary!

I don’t think that’s what I want; a plugin seems cleaner to me. But I will probably play with it nonetheless, since that looks surprising. Wouldn’t you at least have to wrap that JSON in a <script ...> tag to get this to work, and possibly insert that tag before the </body> or </html> tag? If not, I’m curious as to how that would work.

#11

Yes. You are right. I did rename it to data.store, which needs the script tags.

I did a test. The browser moves the script-tag inside to the BODY tag, but TW does not seem to find it.
I did expect, that the position should be fixed after the first save. :confused:

Edit: It seems there is a bug in the core – more testing needed.
Edited again: The following command works.

copy data.store+empty.html new-wiki.html

Reason: The data-store needs to be parsed by the browser before the core itself.

The TW core is part of the <div id="bootKernel"..> which is the last DIV in the BODY. So if the store is concatenated to the front of the file it seems to work fine. :slight_smile:

#12

I don’t have much time to investigate right now, but I would guess that this simply means that TW looks for the very first script with the right class and type, and quits with that. That would be a shame, as this technique sounds like it could be useful, especially if it would work when put after the </html> tag, and the simple concatenation you propose did work.

#13

I did find it out and “edit-edited” the post accordingly :wink:

#14

Oh, that makes sense. This is nice to know, a useful technique, and it allows us to find our additional content easily too in the HTML! Thank you for spending the time!

#15

Yea, but only if you do not save the wiki. Since TW uses it’s default template to save the file, all the tiddlers will be stored within the “initial store” again. It does not track different store areas atm

#16

Oh, ok. Not a big deal; it just sounded like a nice side benefit. I should have tried it first.

Again, thank you for your contributions here. They’ve helped clarify my mind a lot.