{"id":82582,"date":"2026-03-20T04:56:00","date_gmt":"2026-03-20T08:56:00","guid":{"rendered":"https:\/\/www.inmotionhosting.com\/blog\/?p=82582"},"modified":"2026-04-07T12:42:51","modified_gmt":"2026-04-07T16:42:51","slug":"rate-limiting-ai-crawler-bots-modsecurity","status":"publish","type":"post","link":"https:\/\/www.inmotionhosting.com\/blog\/rate-limiting-ai-crawler-bots-modsecurity\/","title":{"rendered":"Rate Limiting AI Crawler Bots with ModSecurity"},"content":{"rendered":"\n
\"Rate<\/figure>\n\n\n\n

AI training bots from OpenAI, Anthropic, Amazon, and a dozen other companies are now hitting production web servers with the same aggression as a DDoS attack, and robots.txt isn’t stopping them. This guide walks through how InMotion’s systems team uses ModSecurity to enforce per-bot rate limiting at the server level, without cutting off your site’s indexing exposure entirely.<\/p>\n\n\n\n

The Problem: AI Bots That Don’t Follow the Rules<\/h2>\n\n\n\n

robots.txt<\/strong> has been the de facto agreement between websites and web crawlers for decades. A directive like Crawl-delay: 10<\/code> tells compliant bots to wait 10 seconds between requests. Google gives you a way to configure crawl rate through Google Search Console<\/a>. Traditional search crawlers have operated within these boundaries long enough that most sysadmins never thought much about them.<\/p>\n\n\n\n

LLM training crawlers are a different story.<\/p>\n\n\n\n

Starting in 2024, InMotion’s systems administration teams began seeing a pattern of unusually heavy traffic across shared and dedicated infrastructure. The source wasn’t a single bot running wild. It was several bots, each operated by a different AI company, simultaneously crawling the same servers with no delay between requests and no respect for Crawl-delay<\/code> directives. None of them coordinated with each other. None of them needed to. The combined load of GPTBot, ClaudeBot, Amazonbot, and their peers hitting the same server concurrently produces resource exhaustion that looks functionally identical to an unintentional distributed denial-of-service attack.<\/p>\n\n\n\n

That surprises a lot of website owners who assume robots.txt<\/strong> is binding. It isn’t. It’s a convention, and these bots aren’t observing it.<\/em><\/p>\n\n\n\n

Two Options, One Clear Tradeoff<\/h2>\n\n\n\n

The blunt instrument is a full block via .htaccess. You can deny access by User-Agent<\/a> and the bots stop hitting your server entirely. Problem solved, except it isn’t: your site also disappears from AI-driven discovery systems. For businesses that want to appear in AI-generated answers or LLM-powered search features, blocking training crawlers entirely carries a real long-term cost.<\/p>\n\n\n\n

Rate limiting is the better path. You slow the bots down to a pace your server can absorb. They still index your content. You still maintain visibility. And when a bot refuses to respect the rate limit you’ve set, you block that specific request rather than the bot permanently.<\/p>\n\n\n\n

How ModSecurity Rate Limiting Works<\/h2>\n\n\n\n

ModSecurity<\/a> is an open-source Web Application Firewall that operates inside Apache or Nginx, inspecting HTTP traffic in real time. It’s the same tool that blocks SQL injection attempts and cross-site scripting attacks on properly hardened servers. What makes it useful here is its ability to track request frequency by User-Agent and deny requests that exceed a defined threshold.<\/p>\n\n\n\n

The approach works in two steps:<\/p>\n\n\n\n