{"id":130930,"date":"2025-10-16T08:55:16","date_gmt":"2025-10-16T12:55:16","guid":{"rendered":"https:\/\/www.inmotionhosting.com\/support\/?p=130930"},"modified":"2025-11-12T17:28:33","modified_gmt":"2025-11-12T22:28:33","slug":"how-to-use-robots-txt-to-block-crawlers","status":"publish","type":"post","link":"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-use-robots-txt-to-block-crawlers\/","title":{"rendered":"How to Use robots.txt Disallow to Block Crawlers and Protect Site Performance"},"content":{"rendered":"
\"How<\/figure>\n\n\n

Crawlers drive search visibility, but they can also overload your server if left unchecked. This guide shows you how to use robots.txt to take control. Learn how to block low-value directories, protect essential assets that Google needs to render your site, manage crawl rates with delays, and defend against bandwidth-heavy AI bots. You\u2019ll get practical examples, real-world use cases, and best practices for combining robots.txt with server rules. Strategic crawler management improves site speed, reduces hosting costs, and ensures search engines focus on your most important content. <\/p><\/div>\n\n\n

Search engines discover your content by sending automated crawlers, often called bots or spiders, that scan your pages and index them. This process is essential for visibility in Google or Bing, but if left uncontrolled, bots can overrun your server and slow down performance.<\/p>\n\n\n\n

That\u2019s where the robots.txt file comes in. It\u2019s one of the simplest yet most powerful tools for directing how search engines interact with your website. With a few lines of text, you can tell compliant bots what to explore and what to leave alone.<\/p>\n\n\n\n

Whether you\u2019re running a high-performance WordPress site<\/a>, managing multiple client websites, or scaling an e-commerce platform, controlling crawler access protects your infrastructure and keeps your site running at peak speed.<\/p>\n\n\n\n

However, not all crawlers play by the rules. A growing wave of AI-driven bots are consuming massive amounts of bandwidth without contributing to your SEO visibility. Knowing how to manage both traditional and modern crawlers is now part of maintaining your site\u2019s speed, stability, and search performance.<\/p>\n\n\n\n

What Robots.txt Really Does (and What It Doesn\u2019t)<\/h2>\n\n\n\n

Before you begin blocking or allowing crawlers, it\u2019s important to understand what this small file actually controls. Many site owners assume robots.txt is a universal shield, but it\u2019s more of a polite request that good bots usually respect. Understanding its limits helps you set realistic expectations and use it strategically.<\/p>\n\n\n\n

When a search engine visits your site, it first looks for a robots.txt file in the root directory (for example, https:\/\/example.com\/robots.txt). The file tells the crawler which parts of the site it can or cannot access.<\/p>\n\n\n\n

For example:<\/p>\n\n\n\n