17 Sep
2025
17 Sep
'25
9:59 a.m.
On Sep 17, 2025, at 10:38 AM, Doug McIntyre via cctalk
<cctalk(a)classiccmp.org> wrote:
On Wed, Sep 17, 2025 at 09:33:25AM -0400, Paul Koning via cctalk wrote:
Yes.
paul
A web crawler that does not obey robots.txt is
not a law abiding outfit. Best would be to block it entirely. If they are that
dismissive of honesty, they are also unlikely to pay attention to such matters as
copyright and intellectual property ownership.
So, you want to block the whole of the Internet, including every AI company that all
ignore robots.txt?