[cctalk] Large language model (LLM) Web Scrapers

For those of you who run vintage computing-related info sites, have you noticed all of the LLM scraper activity? AI services are using the LLM scrapers to populate their knowledge bases. At any given moment 5-10 of them are active on vintagecomputer.net. It’s funny, when I ask an AI about something vintage computing-related, something obscure, I can trick into giving me an answer from my own site. I have actually had to modify the site code to manage the traffic, to improve efficiency. But they’re not going after just my site, these scrapers are absorbing copies of the entire WWW. I wonder how long the WWW will remain open, it would be a bummer if I found copies of my site elsewhere. Bill