Setting a Crawl Delay in Google Webmaster Tools

Google just like any other search engine needs to send crawlers to your website in order to index the information.

Search engines can crawl your website too fast, and you can block search engines from crawling your website.

With Google, you typically want them to crawl your site, but at a reduced rate so they aren’t slamming your server.

If you don’t already have a Google account you should read our introduction to Google Webmaster Tools article.

Changing Google’s crawl rate in Webmaster Tools

Adjusting the crawl rate for Google won’t affect ranking, just how long it takes them to index your entire website.

If you had 1,000 pages on your website, a search engine could potentially index your entire site in a few minutes.

However this could cause high system resource usage with all of those pages loaded in a short time period.

A Crawl-delay: of 30 seconds would allow crawlers to index your entire 1,000 page website in just 8.3 hours

A Crawl-delay: of 500 seconds would allow crawlers to index your entire 1,000 page website in 5.8 days

Using the steps below you can adjust Google’s crawl rate for your website using Webmaster Tools:

Step 1
Log in to Google Webmaster Tools

Step 2
Click on the website you’d like to manage

Step 3
Click the top-right settings cog, then Site Settings

Step 4
Click on Limit Google’s maximum crawl rate
Adjust the crawl rate to at least 30 seconds apart.

Step 5
Crawl delay settings are only saved for 90 days.
Note down the date you’ll need to set them again

Now the next time Google sends their Googlebot crawler to index your website, they’ll be sure to space out their requests at least 30 seconds in-between each other.

Thoughts on “Setting a Crawl Delay in Google Webmaster Tools

    • This guide is not really for helping your site’s rank but to reduce the usage of your server. For example, if you have a large site and are receiving a large amount of traffic due to a promotion, increasing the crawl delay may reduce the resources your server is using and speed up your site.
      Thank you,
      John-Paul

  • How to do this calculations?

     

    A Crawl-delay: of 30 seconds would allow crawlers to index your entire 1,000 page website in just 8.3 hours

    A Crawl-delay: of 500 seconds would allow crawlers to index your entire 1,000 page website in 5.8 days

    • Hello Suleman,

      Thank you for contacting us. The best crawl rate will depend on several actors, such as the amount of resources your server has available, amount of pages/content on your site, frequency of website update.

      I recommend trying different configurations, to see what is working for your specific sites.

      Thank you,
      John-Paul

    • Hello Brad,

      The crawl-delay in robots.txt is pretty much ignored by Google. It does, however, obey the crawl delays set in the Webmaster tools.

      Kindest Regards,
      Scott M

  • The irony is that my site is hosted on Google App. Engine, so that’s money going for them too 🙂

    Then again, most of my traffic comes from them as well.

    What can we do against the almighty…

  • I cannot change that setting; it says:

    “Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate.”

    On Feb 19 Google crawled more than 2 million pages in my site (taht cost more than $30) and now is starting to spike again.

    What could I do?

     

    • Hello MdeA,

      I’m sorry you’re getting that message. If you’re getting it and it’s coming from Google, you won’t be able to make a change unless you can contact them and find out how. We have no way to control what Google has decided to do for your crawl rate.

      Regards,
      Arnel C.

    • Hello Javadh,

      You would not be able to enable the crawling during certain times of the day as Google does not function like that.

      Best Regards
      TJ Edens

    • Hello Javadh,

      If you do not want to limit Googles crawl rate then I would suggest selecting the optimal option and let Google automatically set its crawl rate.

      Best Regards,
      TJ Edens

Leave a Reply to InMotionFan Cancel reply