TT,
It would be nice if I your example patters can be collected and added to the demo!
I was thinking to have search patterns like search Filters.
I think regex needs good documentation here! we can use the docs in TW-regex!
I would be happy to add your example, tips and docs.
1 Like
Right! Maybe it should have it’s own method to save the regex patterns used? Why? They often won’t be compatible with Advanced Search > Filter.
So maybe a way to save the patterns just for the regex component of Advanced Search > Fields would be a good idea???
Just thoughts
TT
1 Like
Please use any of my examples any way that is helpful to your project on this!
Best wishes
TT
Ciao @Mohammad
MINOR CONCEPTUAL ISSUES ON CASE SETTING
Currently the UI looks like this for case setting …
The conceptual issue is the “case sensitive” setting applies to “words” & “literal” but (rightly) not to the “regexp” mode.
In the regexp one sets case sensitivity via a flag in the regex itself. Insensitivity being the prefix (?i).
It is a minor point but in some ways the regexp component maybe needs to be presented as a thing in its own right. For instance …
Just a thought!
TT
There’s no maybe about it - it’s vital. Along with a Go button I allow search strings to be saved (and cleaned up)
If I understand you, you means casesensitivity is not applied to regex!
That is correct! If you select regex, the case sensitive checkbox is disabled (greyed out)
I will give a try on this! In Commander, I have used a mechanism quite similar to main TW Saved Filters!
Right! Just FYI, my comment was about possibility that the user might still think it was activated. I think, well just for me, it struck me having all three modes on one line might be confusing?
But is was just a minor thought! I don’t think it is particularly important.
Best wishes
TT
It is possible to hide case sensitive when regex is used!
I will investigate this more!
Ciao @Mohammad, another comment I hope you find interesting!
What is so very good about the tool is the ability to set “Scope” (i.e. a filter that predefines which Tiddlers the search for “Match” probes) separately from the match pattern itself.
This two step approach of yours is really good I think!
Here is an example …
EXAMPLE: FINDING TIDDLERS WITH EMPTY TAGS
Use case: From a large collection of Tiddlers that are named “Perry Mason ##” find only those that have an empty tag field. The reason for this is they are all supposed to have the tag “#mason”. Having isolated the specific Tiddlers lacking it they are easy to correct.
Note: to get this to work I had to change the hidden setting “$:/config/Search/MinLength” to 2 so the regex “^$”, for the empty string, would work. Note I also use regexp in the scope filter. I transcluded the scope setting into your interface so it’s easier for you to see what is going on.
Very best
TT
2 Likes
Nice combination for intermediate/advanced users!
SearchTools from Eric uses a multistep interface!
1 Like
Ciao @Mohammad, just one more test. This time for performance at scale.
STRESS TEST: Search 14,000 Tiddlers’ Text Field
Use Case: Large number of small Tiddlers. (14,000 Tiddlers of maximum size of 280 characters each; plain text format). They are slices of television screenplays of the Perry Mason tv series. The transcription is poor with many errors that need correcting. Mostly I do this before I import into TW but there are some that have to be handled manually. In the example I find errors where some characters were replaced in scanning by " _" that need to be restored manually. Fortunately there are not too many of them.
Performance Notes: Well workable. JSON import was a little slow but worked fine. Searching through the 14k Tiddlers was fast (& 191 Tiddlers with errors out of 14K is not bad!).
This is what it looks like …
...
Best wishes
TT
1 Like
Right! I remember it. But I can’t find it right now. TBH, I would like to compare it to your tool here. Do you have a link?
TT
Advanced Search in Fields has got update v0.5.0
The major changes here
- native searchbox (looks and behave like other search tabs except navigation keys)
- customizable primary fields, exclude fields
- improved documenation
- predefined regex patterns
- user added predefined regex patterns
Code and demo
- Demo: Advanced Search in Fields — search in fields
- Code: GitHub - kookma/TW-Search: Advanced search in fields using regular expression and tabular output
Do not forget to to have Shiraz installed on your Tiddlywiki.
1 Like
@TiddlyTweeter
You can simply use your own customized UI here with v0.5.0
This is the almost the final UI
I may change the icons only later!
I would be happy to add predefined useful regex pattern!
Best wishes
Mohammad
1 Like
@EricShulman Power Search is here
This is very useful, yet very powerful tool.
https://tiddlytools.com/filtergenerators.html#TiddlyTools%2FFilterGenerators%2FPowerSearch
1 Like
Whoah! It is super good. From that alone you’d understand TW filtering using it.
It is different than your more limited “Search in Fields”.
The 2-step approach you have taken (i.e. “Scope filter” + “match pattern”) is simpler. I suspect your approach will prove pragmatic for much ordinary use. Eric’s take is comprehensive and wide, though likely a bit more complex?
Different approaches. Let’s see!
TT
I think it is much more powerful! while transparent to user! You see what happen behind the scene!
1 Like
Ciao @Mohammad here is an illustration of how use of regex can get very exact … Your “2-step process” is great for this kind of thing as the user can focus on the regex rather than having to also cope with the “scope” at the same time–which is what you’d have to do normally in Filter search–i.e. cope with a very complex long string.
ADVANCED REGEX: Negative Lookahead
Use Case: Find all instances in the text field where the letter “u” does not follow the letter “q”.
Dealing with true negatives in regex (i.e. not just “nots”) you use either “lookahead” or “lookbehind”. In the specific case, using negative lookahead, I searched 14,000 Tiddlers and found only one that broke standard English. That was because it used “Q” unconventionally …
Here is the search pattern/result …
Here is the Tiddler found …
ADVANCED REGEX: Positive Lookahead
By way of contrast, the 1674 normal cases.
Use Case: Find all instances in the text field where the letter “u” follows the letter “q”. Here is the search pattern/result …
The more advanced use of regex I do think needs you to be able to focus hard on what you are doing. This tool definitely helps that by reducing distraction to the minimum.
Best wishes
TT
2 Likes