robots7

Category: Databases

daveengle
Asked:
2011-11-01 5:23 pm EST

Hits: 10,342
What can be done to block the robots and spiders from tracking my domain?

You must login before you can ask a follow up question.

You must login before you can submit an answer.

Best answer chosen by InMotion Hosting staff

0

Tim S.
Staff
9,967 Points
2011-11-02 7:10 am EST
Hi daveengle,

Thanks for posting your question. I'm more than happy to assist you today. There's two methods to keeping spiders from crawling your website. The first, involves creating a robots.txt file. This file will be read first, and specifies whether the bot can crawl the site. To learn more about robots.txt files please visit the site:

www.robotstxt.org

The second option is to place a specific "no index, no follow" tag in the header of each webpage you wish to exclude from being crawled. Here's the basic tag to do so:

‹META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">


Also, I'd suggest creating a Google Webmaster Tools account. This will give you the ability to decide how Google's bots crawl your website.

I hope this helps! If you need further assistance please feel free to contact us.

Thank you!

Tim S

You must login before you can post a comment about this answer.

Like this Question?

News / Announcements

WordPress wp-login.php brute force attack
Updated 2014-07-17 06:43 pm EST
Hits: 201665

Related Articles

It looks like there are no related articles.
Would you like to ask a question about this page? If so, click the button below!
Ask a Question

Need more Help?

Search

Ask the Community!

Get help with your questions from our community of like-minded hosting users and InMotion Hosting Staff.

Current Customers

Chat: Click to Chat Now E-mail: support@InMotionHosting.com
Call: 888-321-HOST (4678) Ticket: Submit a Support Ticket

Not a Customer?

Get web hosting from a company that is here to help. Sign up today!