Google just like any other search engine needs to send crawlers to your website in order to index the information.

Search engines can crawl your website too fast, and you can block search engines from crawling your website.

With Google, you typically want them to crawl your site, but at a reduced rate so they aren't slamming your server.

If you don't already have a Google account you should read our introduction to Google Webmaster Tools article.

Changing Google's crawl rate in Webmaster Tools

Adjusting the crawl rate for Google won't affect ranking, just how long it takes them to index your entire website.

If you had 1,000 pages on your website, a search engine could potentially index your entire site in a few minutes.

However this could cause high system resource usage with all of those pages loaded in a short time period.

A Crawl-delay: of 30 seconds would allow crawlers to index your entire 1,000 page website in just 8.3 hours

A Crawl-delay: of 500 seconds would allow crawlers to index your entire 1,000 page website in 5.8 days

Using the steps below you can adjust Google's crawl rate for your website using Webmaster Tools:

Step 1
Log in to Google Webmaster Tools

Step 2
Click on the website you'd like to manage

Step 3
Click the top-right settings cog, then Site Settings

Step 4
Click on Limit Google's maximum crawl rate
Adjust the crawl rate to at least 30 seconds apart.

Step 5
Crawl delay settings are only saved for 90 days.
Note down the date you'll need to set them again

Now the next time Google sends their Googlebot crawler to index your website, they'll be sure to space out their requests at least 30 seconds in-between each other.

Did you find this article helpful?

We value your feedback!

Why was this article not helpful? (Check all that apply)
The article is too difficult or too technical to follow.
There is a step or detail missing from the instructions.
The information is incorrect or out-of-date.
It does not resolve the question/problem I have.
How did you find this article?
Please tell us how we can improve our Support Center:
Email Address
Optional, but our team may contact you for more information.
Like this Article?

Post a Comment

Name:
Email Address:
Comment:
Are you a bot?
Submit

Please note: Your name and comment will be displayed, but we will not show your email address.

Write New!
Do you want to publish a tutorial to our support center?

News / Announcements

SSL Certficate Warnings
Updated 2014-04-14 11:34 am EST
Hits: 2004
Heartbleed 0-day OpenSSL security bug
Updated 2014-04-14 04:43 pm EST
Hits: 5279

Related Questions

Here are a few questions related to this article that our customers have asked:
Ooops! It looks like there are no questions about this page.
Would you like to ask a question about this page? If so, click the button below!
Ask a Question

Need more Help?

Search

Ask the Community!

Get help with your questions from our community of like-minded hosting users and InMotion Hosting Staff.

Current Customers

Chat: Click to Chat Now E-mail: support@InMotionHosting.com
Call: 888-321-HOST (4678) Ticket: Submit a Support Ticket

Not a Customer?

Get web hosting from a company that is here to help. Sign up today!