InMotion Hosting Support Center


Category: Databases

2011-11-01 5:23 pm EST

Hits: 10,384
What can be done to block the robots and spiders from tracking my domain?

You must login before you can ask a follow up question.

You must login before you can submit an answer.

Best answer chosen by InMotion Hosting staff


Tim S
13,441 Points
2011-11-02 7:10 am EST
Hi daveengle,

Thanks for posting your question. I'm more than happy to assist you today. There's two methods to keeping spiders from crawling your website. The first, involves creating a robots.txt file. This file will be read first, and specifies whether the bot can crawl the site. To learn more about robots.txt files please visit the site:

The second option is to place a specific "no index, no follow" tag in the header of each webpage you wish to exclude from being crawled. Here's the basic tag to do so:


Also, I'd suggest creating a Google Webmaster Tools account. This will give you the ability to decide how Google's bots crawl your website.

I hope this helps! If you need further assistance please feel free to contact us.

Thank you!

Tim S

You must login before you can post a comment about this answer.

Like this Question?

Need more Help?


Ask the Community!

Get help with your questions from our community of like-minded hosting users and InMotion Hosting Staff.

Current Customers

Chat: Click to Chat Now E-mail:
Call: 888-321-HOST (4678) Ticket: Submit a Support Ticket

Not a Customer?

Get web hosting from a company that is here to help. Sign up today!