InMotion Hosting Support Center

Google just like any other search engine needs to send crawlers to your website in order to index the information.

Search engines can crawl your website too fast, and you can block search engines from crawling your website.

With Google, you typically want them to crawl your site, but at a reduced rate so they aren't slamming your server.

If you don't already have a Google account you should read our introduction to Google Webmaster Tools article.

Changing Google's crawl rate in Webmaster Tools

Adjusting the crawl rate for Google won't affect ranking, just how long it takes them to index your entire website.

If you had 1,000 pages on your website, a search engine could potentially index your entire site in a few minutes.

However this could cause high system resource usage with all of those pages loaded in a short time period.

A Crawl-delay: of 30 seconds would allow crawlers to index your entire 1,000 page website in just 8.3 hours

A Crawl-delay: of 500 seconds would allow crawlers to index your entire 1,000 page website in 5.8 days

Using the steps below you can adjust Google's crawl rate for your website using Webmaster Tools:

Step 1
Log in to Google Webmaster Tools

Step 2
Click on the website you'd like to manage

Step 3
Click the top-right settings cog, then Site Settings

Step 4
Click on Limit Google's maximum crawl rate
Adjust the crawl rate to at least 30 seconds apart.

Step 5
Crawl delay settings are only saved for 90 days.
Note down the date you'll need to set them again

Now the next time Google sends their Googlebot crawler to index your website, they'll be sure to space out their requests at least 30 seconds in-between each other.

Support Center Login

Social Media Login

   
Social Login Joomla

Related Questions

Here are a few questions related to this article that our customers have asked:
Ooops! It looks like there are no questions about this page.
Would you like to ask a question about this page? If so, click the button below!
Ask a Question
n/a Points
2015-02-17 5:43 pm
if i want google bot fast crawl my site how should i change setting ?
Staff
2,702 Points
2015-02-17 5:53 pm
Hello Javadh,

If you do not want to limit Googles crawl rate then I would suggest selecting the optimal option and let Google automatically set its crawl rate.

Best Regards,
TJ Edens
n/a Points
2015-02-17 6:03 pm
i am posting every 30min from 13am to 23am and from 23am to 12 am have not post
Staff
2,702 Points
2015-02-17 6:13 pm
Hello Javadh,

It should be crawling your website during those times.

TJ Edens
n/a Points
2015-02-17 6:14 pm
how can i set for this times ?
Staff
2,702 Points
2015-02-17 6:16 pm
Hello Javadh,

You would not be able to enable the crawling during certain times of the day as Google does not function like that.

Best Regards
TJ Edens
n/a Points
2015-02-23 5:21 pm

but bing have i think

n/a Points
2015-02-23 4:48 pm

I cannot change that setting; it says:

"Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate."

On Feb 19 Google crawled more than 2 million pages in my site (taht cost more than $30) and now is starting to spike again.

What could I do?

 

Staff
24,062 Points
2015-02-23 5:19 pm
Hello MdeA,

I'm sorry you're getting that message. If you're getting it and it's coming from Google, you won't be able to make a change unless you can contact them and find out how. We have no way to control what Google has decided to do for your crawl rate.

Regards,
Arnel C.
n/a Points
2015-02-23 5:49 pm

The irony is that my site is hosted on Google App. Engine, so that's money going for them too :)

Then again, most of my traffic comes from them as well.

What can we do against the almighty...

n/a Points
2015-03-19 10:41 pm

Crawl-delay: 1

 

If your robots txt file shows this does it mean Google will not crawl the site?

Staff
25,902 Points
2015-03-23 1:16 pm
Hello Brad,

The crawl-delay in robots.txt is pretty much ignored by Google. It does, however, obey the crawl delays set in the Webmaster tools.

Kindest Regards,
Scott M

Post a Comment

Name:
Email Address:
Phone Number:
Comment:
Submit

Please note: Your name and comment will be displayed, but we will not show your email address.

13 Questions & Comments

Post a comment

Back to first comment | top

Need more Help?

Search

Ask the Community!

Get help with your questions from our community of like-minded hosting users and InMotion Hosting Staff.

Current Customers

Chat: Click to Chat Now E-mail: support@InMotionHosting.com
Call: 888-321-HOST (4678) Ticket: Submit a Support Ticket

Not a Customer?

Get web hosting from a company that is here to help. Sign up today!