17 Sep
2025
17 Sep
'25
10:17 a.m.
On Tuesday, September 16th, 2025 at 17:51, Paul Koning via cctalk
<cctalk(a)classiccmp.org> wrote:
If they are honest they will obey robots.txt and you
can use that to stop them. If it doesn't
robots.txt doesn't seem to help. If anything it can hurt in this regard. Analyzing
my site
logs I noticed at least one scraper botnet pulling /robots.txt on its initial connect and
then
immediately pulling down everything listed in it. Which is also its downfall if you seed
the
file with traps like a tarpit or a data compression bomb.
The Doctor [412/724/301/703/415/510]
WWW: https://drwho.virtadpt.net/
Get thee down. Be thou funky.