Google just like any other search engine needs to send crawlers to your website in order to index the information.

Search engines can crawl your website too fast, and you can block search engines from crawling your website.

With Google, you typically want them to crawl your site, but at a reduced rate so they aren't slamming your server.

If you don't already have a Google account you should read our introduction to Google Webmaster Tools article.

Changing Google's crawl rate in Webmaster Tools

Adjusting the crawl rate for Google won't affect ranking, just how long it takes them to index your entire website.

If you had 1,000 pages on your website, a search engine could potentially index your entire site in a few minutes.

However this could cause high system resource usage with all of those pages loaded in a short time period.

A Crawl-delay: of 30 seconds would allow crawlers to index your entire 1,000 page website in just 8.3 hours

A Crawl-delay: of 500 seconds would allow crawlers to index your entire 1,000 page website in 5.8 days

Using the steps below you can adjust Google's crawl rate for your website using Webmaster Tools:

Step 1
Log in to Google Webmaster Tools

Step 2
Click on the website you'd like to manage

Step 3
Click the top-right settings cog, then Site Settings

Step 4
Click on Limit Google's maximum crawl rate
Adjust the crawl rate to at least 30 seconds apart.

Step 5
Crawl delay settings are only saved for 90 days.
Note down the date you'll need to set them again

Now the next time Google sends their Googlebot crawler to index your website, they'll be sure to space out their requests at least 30 seconds in-between each other.

Did you find this article helpful?

We value your feedback!

Why was this article not helpful? (Check all that apply)
The article is too difficult or too technical to follow.
There is a step or detail missing from the instructions.
The information is incorrect or out-of-date.
It does not resolve the question/problem I have.
How did you find this article?
Please tell us how we can improve this article:
Email Address

new! - Enter your name and email address above and we will post your feedback in the comments on this page!

Support Center Login

Social Media Login

Related Questions

Here are a few questions related to this article that our customers have asked:
Ooops! It looks like there are no questions about this page.
Would you like to ask a question about this page? If so, click the button below!
Ask a Question
n/a Points
2015-02-17 5:43 pm
if i want google bot fast crawl my site how should i change setting ?
2,033 Points
2015-02-17 5:53 pm
Hello Javadh,

If you do not want to limit Googles crawl rate then I would suggest selecting the optimal option and let Google automatically set its crawl rate.

Best Regards,
TJ Edens
n/a Points
2015-02-17 6:03 pm
i am posting every 30min from 13am to 23am and from 23am to 12 am have not post
2,033 Points
2015-02-17 6:13 pm
Hello Javadh,

It should be crawling your website during those times.

TJ Edens
n/a Points
2015-02-17 6:14 pm
how can i set for this times ?
2,033 Points
2015-02-17 6:16 pm
Hello Javadh,

You would not be able to enable the crawling during certain times of the day as Google does not function like that.

Best Regards
TJ Edens
n/a Points
2015-02-23 5:21 pm

but bing have i think

n/a Points
2015-02-23 4:48 pm

I cannot change that setting; it says:

"Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate."

On Feb 19 Google crawled more than 2 million pages in my site (taht cost more than $30) and now is starting to spike again.

What could I do?


21,373 Points
2015-02-23 5:19 pm
Hello MdeA,

I'm sorry you're getting that message. If you're getting it and it's coming from Google, you won't be able to make a change unless you can contact them and find out how. We have no way to control what Google has decided to do for your crawl rate.

Arnel C.
n/a Points
2015-02-23 5:49 pm

The irony is that my site is hosted on Google App. Engine, so that's money going for them too :)

Then again, most of my traffic comes from them as well.

What can we do against the almighty...

n/a Points
2015-03-19 10:41 pm

Crawl-delay: 1


If your robots txt file shows this does it mean Google will not crawl the site?

23,972 Points
2015-03-23 1:16 pm
Hello Brad,

The crawl-delay in robots.txt is pretty much ignored by Google. It does, however, obey the crawl delays set in the Webmaster tools.

Kindest Regards,
Scott M

Post a Comment

Email Address:
Phone Number:

Please note: Your name and comment will be displayed, but we will not show your email address.

13 Questions & Comments

Post a comment

Back to first comment | top

Need more Help?


Ask the Community!

Get help with your questions from our community of like-minded hosting users and InMotion Hosting Staff.

Current Customers

Chat: Click to Chat Now E-mail:
Call: 888-321-HOST (4678) Ticket: Submit a Support Ticket

Not a Customer?

Get web hosting from a company that is here to help. Sign up today!