I can tell you that it is really hard to check url’s automatically these days. Most links don’t lead to a simple 404 or “site can’t be reached” (though this one did). In most cases, there is still a page there – it’s just the wrong page. Often, it’s a page attempting to sell the domain name. Grrr.
Oddly, given my general incompetency with tech code, this issue interests me.
If I can help let me know as I hate informational dead-ends.
I did think a PS script that probes an address and retrieves whatever it is THEN does a crude guess (via regex) whether it is an error page, a domain farmer or, hopefully, well-kosher would be possible. With sweat.
Anyway. On this one I’m definitely up for a call.
@TiddlyTitch perhaps you could test for something that exists in an empty html eg I see this in the metadata of the html file <meta name="application-name" content="TiddlyWiki" />
If we view the page source on a bob shared wiki it also contains this.
Or methaps if metadata can be queried <meta name="tiddlywiki-version" content="5.1.23" /> we could show the wiki version.
The question in my mind is whether we should even continue with the community resources – why have two sources of “truth” ? Maybe better just to have the aggregator? Especially since the aggregator gets updated automatically. e.g., I submitted a PR for the Projectify tiddler 24 hours ago.
For link checking, In the end, what I did was to set up a methodology and click through each and every link (for the aggregator – didn’t back-feed into community). In addition to the usual dead-ends, there was also one or two instances where the TW file was there, but the content had gone away.
There’s also the problem of (hopefully) temporary dead-ends. Like I imagine there’s a lot of servers not being maintained right now in Uk raine.
Just FYI (I was inspired by your Polly) to experiment a bit with using PowerShell to try automate a bit with probing resources. @TW_Tones makes some good points about a bespoke way to check fidelity, I think?
Ah! Complexity now where there was simplicity before. That “fouls-the-footpath” (I can’t find a net explanation of this common idiom, but you’ll know what I mean) and makes your work hard?