Compact filter for JSON tiddler

[RetRoland]

Hello,
I’m still playing around with JSON tiddlers:

Example1-JSON

{
	"0": {
		"Name": "Fountain Pen",
		"Details": "Green, metallic",
		"Price": 10
	},
	"1": {
		"Name": "Pencil",
		"Details": "No.2",
		"Price": 1.50
	},
	"2": {
		"Name": "Ballpoint Pen",
		"Details": "red",
		"Price": 2
	},
	"3": {
		"Name": "Eraser",
		"Details": "Soft, blue",
		"Price": 2
	},
	"4": {
		"Name": "Ruler1",
		"Details": "20cm, transparent",
		"Price": 2.50
	},
	"5": {
		"Name": "Ruler2",
		"Details": "10cm, wooden",
		"Price": 0
	},
	"6": {
		"Name": "Ruler3",
		"Details": "5cm, cheap plastic",
        "Price": 0.50
	}
}

At last I found a way to display all names in one single filter (before I used two nested ListWidgets), but I also like to sort out all items without a price.
With some trial I ended up with this filter, which works the way I like it:

<$set name="JSONdata" value={{Example1-JSON}} >
<$list filter="[<JSONdata>jsonindexes[]] :map[<JSONdata>jsonget<currentTiddler>,[Price]!match[0]then<JSONdata>jsonget<currentTiddler>,[Name]]" >

</$list>

According to the description of :map:
Filter runs used with the :map prefix should return at least the same number of items that they are passed
What does that mean exactly? Is this only a rough guideline or do I have to expect unwanted behaviour when I use it to filter out some items?
If yes: how could I do it better?

I also like to display the details in the output. My first idea was to use another :map-instruction (in the same filter run) to access the JSON-Details, but that doesn’t work because I cannot access the currentTiddler of the first expression (it seems that it only refers to the last filter).
Or is there some sort of a Push/Pop-mechanism (onto a stack) available for the variables in filters?

As always, I’d be grateful for any suggestions!
Thank you in advance.

[tw-FRed]

@RetRoland

According to the description of :map:
Filter runs used with the :map prefix should return at least the same number of items that they are passed
What does that mean exactly? Is this only a rough guideline or do I have to expect unwanted behaviour when I use it to filter out some items?
If yes: how could I do it better?

It’s not only a rough requirement, without suffix :map filter runs must return the same number of items as their input. The :flat suffix allows you to add items, but not to remove items. If you try to remove items by filtering them in a :map filter run, the missing items will be replaced by empty/blank items.

For your use case, I suggest you first filter the items using a :filter filter run, which will allow you to remove items but not to transform them, and then use a :map filter run, where you can extract/transform whichever data you need.

Hope this helps

Fred

[Scott_Sauyet]

@RetRoland

According to the description of :map:
Filter runs used with the :map prefix should return at least the same number of items that they are passed
What does that mean exactly? Is this only a rough guideline or do I have to expect unwanted behaviour when I use it to filter out some items?
If yes: how could I do it better?

Yes, this is important. Even if you have something that seems to be working, I wouldn’t count on it.

Perhaps this version with filter will help:

<$set name="JSONdata" value={{Example1-JSON}} >
<$list filter="[<JSONdata>jsonindexes[]] :filter[<JSONdata>jsonget<currentTiddler>,[Price]!match[0]] :map[<JSONdata>jsonget<currentTiddler>,[Name]]" >

</$list>
</$set>

@RetRoland

I also like to display the details in the output.

How about something like this?:

<$set name="JSONdata" value={{Example1-JSON}} >
<$list filter="[<JSONdata>jsonindexes[]] :filter[<JSONdata>jsonget<currentTiddler>,[Price]!match[0]]" >
<li>
  <$text text={{{ [<JSONdata>jsonget<currentTiddler>,[Name]] }}}/>
  (<$text text={{{ [<JSONdata>jsonget<currentTiddler>,[Details]] }}}/>): 
  €<$text text={{{ [<JSONdata>jsonget<currentTiddler>,[Price]] }}}/>
</li>
</$list>
</$set>

[Scott_Sauyet]

@tw-FRed

The :flat suffix allows you to add items, but not to remove items.

Are you sure? I haven’t tried anything with this, but I would find this quite surprising. I would have assumed it was based on JS’s Array.prototype.flatMap, which flattens one level of output array into a new array.

It makes for a very elegant implementation of filterMap¹:

const filterMap = (f, m) => (xs) => xs.flatMap(x => f(x) ? [m(x)] : [])

When the filter function returns false, the callback to flatMap return an empty array. I’d be surprised to find TW doesn’t allow for this.

---

¹filterMap is used this way:

const isOdd = (n) => n % 2 === 1
const square = (n) => n * n

filterMap(isOdd, square)([1, 2, 3, 4, 5]) //=> [1, 9, 25]

[RetRoland]

Hi Fred,
thank you for your answer.
I didn’t notice that empty fields were in the ouput list, too, but I can understand now why my first attempt is not the right way.
Your suggestion to combine :filter and :map seems to be the right direction, I will give it a try!

[RetRoland]

Hi Scott,
thank you for your answer and providing me some examples, too
I still forget that I can alter the ouput of the list widget easily. Most of the time the solution is much easier than my thoughts

Thank you very much to both of you (@tw-FRed & @Scott_Sauyet) this is my first attempt at “complex” filters. You gave me a good insight how things should work the right way!
Still so much to learn, but as long I can get slightly better with every tiddler I write, this is alright

[tw-FRed]

@Scott_Sauyet

@tw-FRed

The :flat suffix allows you to add items, but not to remove items.

Are you sure? I haven’t tried anything with this, but I would find this quite surprising.

Hi @Scott_Sauyet

I don’t know anything about the :map:flat implementation, but here is some code showing the behavior:

`<<list-links "[tag[HelloThere]]">>`

<<list-links "[tag[HelloThere]]">>

`<<list-links "[tag[HelloThere]] +[count[]]">>` 

<<list-links "[tag[HelloThere]] +[count[]]">>

---

`<<list-links "[tag[HelloThere]] :map:flat[<currentTiddler>!prefix[A]]">>`

<<list-links "[tag[HelloThere]] :map:flat[<currentTiddler>!prefix[A]]">>

`<<list-links "[tag[HelloThere]] :map:flat[<currentTiddler>!prefix[A]] +[count[]]">>`

<<list-links "[tag[HelloThere]] :map:flat[<currentTiddler>!prefix[A]] +[count[]]">>

---

`<<list-links "[tag[HelloThere]] :filter[<currentTiddler>!prefix[A]]">>`

<<list-links "[tag[HelloThere]] :filter[<currentTiddler>!prefix[A]]">>

`<<list-links "[tag[HelloThere]] :filter[<currentTiddler>!prefix[A]] +[count[]]">>`

<<list-links "[tag[HelloThere]] :filter[<currentTiddler>!prefix[A]] +[count[]]">>

And the result:

You can see that if we try to filter out an item in a :map:flat run with !prefix[A] the item is emptied but not removed.
The same run with :filter prefix does what’s expected and returns a count of 7.

[Edit]: the :map filter run prefix code is here if you want to have a look:
https://tiddlywiki.com/#%24%3A%2Fcore%2Fmodules%2Ffilterrunprefixes%2Fmap.js

Fred

[Scott_Sauyet]

@tw-FRed

You can see that if we try to filter out an item in a :map:flat run with !prefix[A] the item is emptied but not removed.
The same run with :filter prefix does what’s expected and returns a count of 7.

I apologize. I shouldn’t have challenged this without testing it myself. I was in a hurry on something else, and just did a drive-by on this, without any proper analysis. Thank you for the demonstration.

This is good to know, but disappointing to see.

[tw-FRed]

@Scott_Sauyet

I apologize.

Please don’t!
I’s a counter-intuitive corner case and maybe the demonstration will be useful for the community. Besides, it was fun to elaborate.

Fred

[TW_Tones]

@RetRoland

I also like to display the details in the output. My first idea was to use another :map-instruction (in the same filter run) to access the JSON-Details, but that doesn’t work because I cannot access the currentTiddler of the first expression (it seems that it only refers to the last filter).
Or is there some sort of a Push/Pop-mechanism (onto a stack) available for the variables in filters?

Be carful here, a lot of people try and cram everything in a single filter, when it is unnecessary,Iis trivial and a similar performance, to nest filters via nesting list widgets.

The difference with using a single filter, is for nested list, each list level can set the name variable name, and this is available within that list in the inner widgets (as long as you don’t reuse the same variable name.

• So in this case the outer list widget would iterate each JSON entry and have the key available to retrieve any detail from the JSON entry using the variable as the key to the JSON.

If you nest list widgets the variable, used in the list widget, is a form of push/pop.

• I can explain this further, but I expect you will learn more if you experiment yourself.
• This is true whether or not you name the variable, but if you do change the variable name it is available within the inner lists.

[RetRoland]

Sorry for my late reply, I didn’t had much time to experiment further.

Yes, I see and I’m afraid that I’m one of those “a lot of people”
I was eager to keep this filter as short as possible, ending up with other limitations.
You are right, in this case nested lists would be less mess and the availability of variables in a wider scope is an advantage.

When I was talking about push/pop mechanism I was thinking of some sort of a virtual stack, where push shoves a value (or a variable) onto it and pop removes it again (and sets a variable to the current value). Of course this only would make sense in a filter run with several filter steps.

But that was just an idea to “solve” my problem, because I didn’t know another way to solve it easier at this point.
Thanks to your input it has become more clear for me.