Setting a Crawl Delay in Google Webmaster Tools

Google at times may be excessively crawling your website. While GoogleBot indexing your website is a good thing, if the crawl rate is too high it can adversely affect the server.

You can change the crawl rate (the time used by Googlebot to crawl the site) for your website using Google WebMaster Tools. If you do not have a Webmaster Tools account, please read our article “Introduction to Google Webmaster Tools.” If you wish to set up a crawl delay in a robots.txt file please read our article on How to Stop Search Engines from Crawling my site. 

Please keep in mind the crawl rate refers to the speed of GoogleBots requests during the crawl process. It does not have any effect on how often Google crawls or how deeply they crawl your URL structure.

Changing the Crawl Rate Structure

1. Log into Webmaster Tools. 
2. On the Webmaster Tools Home page, click on your site you wish to change the crawl rate for.
3. Under Site configuration, select "Settings".

   

4. In the Crawl rate section, select "Set custom crawl rate".

   

5. Set your new crawl rate by adjusting the slider. In this example, we reduced our crawl rate by 33%. Click "Save" when you are finished.

   

Please keep in mind, this new crawl rate you just set will be valid for 90 days. After the expiration date 90 days from now, the crawl rate will revert back to the original.

If you need further assistance please feel free to contact our support department.