The volume of AI-powered web scrapers has grown to a point where even small websites are feeling the server load. One web operator recently documented their HTTPS server being overwhelmed by bots identified as LLM scrapers - the automated programs that crawl websites to collect training data or feed AI-powered search and retrieval systems.
This isn't an isolated case. Site operators across the web have been logging spikes in bot traffic over the past year, with AI crawlers from companies building large language models (AI systems trained on massive amounts of text) accounting for a growing share. Unlike traditional search engine bots, which typically respect crawl rate limits and identify themselves clearly in request headers, many LLM scrapers are more aggressive - hitting the same pages repeatedly, ignoring robots.txt directives (the file that tells bots which pages to avoid), or masquerading as regular browser traffic.
For small websites on modest infrastructure, the math is punishing: a few dozen concurrent AI scrapers can generate the same server load as hundreds of human visitors, without the ad revenue or conversions to offset the bandwidth cost.
The practical defenses are becoming routine - Cloudflare's bot management layer, aggressive rate limiting, blocking known AI crawler user-agent strings, and adding JavaScript challenges that most scrapers can't pass. Some operators have moved to Cloudflare's free tier specifically for bot protection, not because they're expecting human DDoS attacks.
The underlying tension is structural. AI companies need web data to train and update their models. Web operators need servers that can handle their actual audience. Without clearer industry standards around crawl rates, self-identification, or some form of compensation, smaller sites will keep absorbing costs they never agreed to.