How to Solve Regex Issues

,
#10

Ciao Mohammad

I looked at it. You know I’m a great fan of regular expressions and where I actually, unlike everything else, have some competence. So it is the regex aspect that I’m most interested in. I think, overall, it is as great idea! But for really useful, safe, implementation, I have a couple of initial comments that I hope will be relevant.

  • Regarding saving the filters. The existing default system is really only useful if you have a limited number of saved filters. IF you thought that, eventually, there might be, say 50, regex heavy filters then, to be usable, one would need to change the filter dropdown so it had a cascading menu system–i.e. with sub-sections for different groups of filters.

UPDATE: I need to rephrase this as the way the tool works at the moment there is no dropdown. Later.

  • Regarding the issue of guarding against “catastrophic backtracking”, as well as for half-completed regex, I very much agree with @CodaCoder. Meaning, for regex heavy filters, it is best if there is a Go! button. The end-user would enter the full regex before applying it. The dynamic nature of TW–with an incomplete regex running–would potentially cause more issues than anything else. I do realise this makes the implementation much more complex but I think without it users, especially those not so familiar with regex, could get in a pickle! :smiley:
#11

Right. But it is still possible to lock the wiki up with a malformed regex. I think the point is to guard against live application before the regex is fully complete?? i.e. a Go! button. Though I’d have to test that to be sure.

Just FYI, as far as I can see, TW is interacting with the JS regex engine and serious error protection is JS based. And I don’t know quite what browsers are doing on that (by the way the JS regex engines do differ slightly between browsers). Personally, to test difficult regex, I use the commercial software RegexBuddy which analyses one’s regex to isolate issues on scope and lookbehind and lookforward.

But, in practice, I think for most use cases in TW it isn’t likely to be much of an issue.

#12

Ciao @Mohammad

This is just a side comment.

I asked an Italian programmer friend to look at TW and give me his first impression of it.

He commented: “It is a very sophisticated self-modifying regex machine with bells and whistles.”

It isn’t quite that, but I thought his comment had quite a lot of truth in it. :smiley:

1 Like
#13

10 posts were split to a new topic: Advanced Search in Fields v0.2.0

#23

I take the blame for this - I’m somehow failing to get the point across.

Forget all your “static HTML” bits and pieces, they are not the issue. Or, at least, they are not the lion’s share of the problem. Browsers can spit out spans and divs and even tables without even breaking a sweat.

Not so events!

Attaching events to elements takes – relatively speaking – quite a long time. When you have many hundreds, or even thousands to wire up, then you will start to see performance issues.

Now… on to TiddlyWiki.

TW is a dynamic presentation environment. ANY change you make, practically any move you make, will cause the TW internals to reconsider and perhaps even do a rebuild of the dom. Your job, as a seasoned TW dev is to aid TW in that task.

If your code requires 10,000 (ten thousand) divs, pfft – easy.

If your code requires 10,000 divs containing event handlers, oh dear… hmmm… uh, perhaps I’ll go get a coffee…

:yawning_face:

I don’t understand what the resistance is, but for any seasoned TW dev, this should be your mantra going forward:

Never write another $list of links/buttons and not consider using $eventcatcher

Actuallly, I do understand the resistance -- you've invested in code that mostly works. Changing it is painful. Suck it up... "it's only code"

:wink:

#24

It’s not clear what you’re referring to, but I hope you’re not comparing a list of event-handler laden links with a single event handler attached to a parent div ($eventcatcher). If so, your definition of “very fast” is very different to mine.

I’m going to stop now.

#25

:sweat_smile:

There is no resistance at all! I really welcome any improvement! The only resistance I have showed by now if to go with pure wikitext instead of JavaScript! The reason is/was to let more people understand your code! and develop/extend the TW ecosystem!

So, please when you read the Search plugin codes, kindly let me know where we can improve the performance! Note that Shiraz is a separate plugin and it can be improved too!

I have developed simpler dynamic tables in other plugins, I can think to use them instead of Shiraz, if there is room for improvement!

In dynamic table there are many buttons/links and I have read the post by @saqimtiaz on improving the performance by using $eventchatcher!

Thank you for time and comments on Search plugin!

1 Like
#26

I’m sorry, but I really don’t have that kind of time right now.

I appreciate that and that’s an admirable stance. I wish I too wrote TW code with an eye on re-use. But that’s my failing. Sorry. the best I can do is “share with myself” through the bundler plugin. Again, I’m time limited so for now, at least, it has to be that way.

Did I read somewhere you have an engineering background? If so, take my input as sharing known facts with you, some of them measured (by others, not me). None of what I gave you above is opinion, just what is known.

Anyway, I MUST get back to “real” work :frowning:

#27

Thank you anyway!

Good luck!

I leave this here and hope others interested in the search tools find useful points!

#28

That, BTW, is pretty much a definition of the human brain. :brain: :wink:

4 Likes
#29

To help prevent that, when regex is selected, I disable live searching and add a Go! button.

I do believe TW users should not be able to crash their wiki/browser when using regexp, I experienced it last week and nearly lost a few hours of work. But in my case, the Go button wouldn’t have helped, for I thought my regexp was legit, and I would have hit the button!

Maybe the solution should be coded in the core, with some kind of “watchdog” monitoring the filters interpreter and “killing” it when a filter processing takes too long ? Just an idea, I don’t even know if this makes sense :sweat_smile:

Fred

#30

That’s very true, of course. But you would get chance to take one last look. And also, your regex wouldn’t be executed before it is finished being written, which was my overriding concern.

1 Like
#31

Rather than a “go” button, which may still not prevent problems, why not have a limit applied? Say 150 tiddlers. Then if a user is sure they want to see all the results, they can select to “See all tiddlers – this may slow your machine down.” . This way users can use the interface normally 90% of the time, be protected from mistakes, verify that a filter works, and still have complete usability when they need it.

#32

It could be 50,000 operations in just the first tiddler (again, this is your splitter in use, asking it to do something really stupid where it matches “everything”). Then add in just 1,000 tiddlers and you’re going to be there a while… like a week :scream:

It’s not ideal, I agree, but something like the Go! button helps slow me down a notch.

And of course, the point in this thread is about stopping regexes running before they’re finished being written.

Anyway… the ship’s already sailing, I think.

#33

Ciao @Mark_S

Ha! Nice one! Thinking about it, despite @CodaCoder’s caveats, I think that might be a good practical solution in many, if not most, use cases?

I guess, finally, it is an empirical issue. Sad to say I haven’t yet created a regex that locks up mother.

Best, TT

#34

I’m a keen user of regexp and I definitely prefer my expression not being used before I’ve completed it! Especially because my machine is slow! And because it would just be a visual nuisance it it would be a quick machine!

But, when it’s written and executed at least once without having been modified afterwards, I would really appreciate it would be live and auto-updated’ in the usual TW way.

#35

@jypre Agreed, it’s a conundrum. The Go! button solves the former problem, but doesn’t help much after that.

1 Like
#36

@CodaCoder Hum… Let’s see: after clicking on go, I could write a copy of the regexp as a live expression that would do the trick. Any modification in the edit box would immediately delete that live expression and that’s for the second part. It would only necessitate a hook on that edit box.

#37

That’s pretty much how I do it - with a button inside the hook.

#38

@CodaCoder You mean you auto-click your button (with js I suspect) in order to get your hook doing its job? Could you post a source for this? It is of tremendous interest!!!