{"id":282,"date":"2011-11-02T12:07:31","date_gmt":"2011-11-02T16:07:31","guid":{"rendered":"https:\/\/www.inmotionhosting.com\/support\/2011\/11\/02\/how-to-stop-search-engines-from-crawling-your-website\/"},"modified":"2025-06-12T16:57:25","modified_gmt":"2025-06-12T20:57:25","slug":"how-to-stop-search-engines-from-crawling-your-website","status":"publish","type":"post","link":"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/","title":{"rendered":"How to Stop Search Engines from Crawling your Website"},"content":{"rendered":"

In order for your website to be found by other people, search engine<\/strong> crawlers, also sometimes referred to as bots or spiders, will crawl<\/strong> your website looking for updated text and links to update their search indexes.<\/p>\n

How to Control search engine crawlers with a robots.txt file<\/h2>\n

Website owners can instruct search engines on how they should crawl a website, by using a robots.txt<\/strong> file.<\/p>\n

When a search engine crawls a website, it requests the robots.txt<\/strong> file first and then follows the rules within.<\/p>\n

It’s important to know robots.txt<\/strong> rules don’t have to be followed by bots, and they are a guideline.For instance, to set a Crawl-delay for Google<\/a> this must be done in the Google Webmaster tools<\/a>.<\/p>\n

For bad bots that abuse your site you should look at how to block bad users by User-agent in .htaccess<\/a>.<\/p>\n<\/div>\n

Edit or create robots.txt file<\/h2>\n

The robots.txt<\/strong> file needs to be at the root of your site. If your domain was example.com<\/strong> it should be found:<\/p>\n

On your website<\/strong>:<\/p>\n

https:\/\/example.com\/robots.txt<\/pre>\n
On your server<\/strong>:<\/p>\n
\/home\/userna5\/public_html\/robots.txt<\/pre>\n
You can also create a new file<\/a> and call it robots.txt<\/strong> as just a plain-text file if you don’t already have one.<\/p>\n
Search engine User-agents<\/h2>\n
The most common rule you’d use in a robots.txt<\/strong> file is based on the User-agent<\/strong> of the search engine crawler.<\/p>\n
Search engine crawlers use a User-agent<\/strong> to identify themselves when crawling, here are some common examples:<\/p>\n
Top 3 US search engine User-agents<\/strong>:<\/p>\n
Googlebot \n\nYahoo! \n\nSlurp bingbot<\/pre>\n
Common search engine User-agents blocked<\/strong>:<\/p>\n
AhrefsBot \n\nBaiduspider \n\nEzooms \n\nMJ12bot \n\nYandexBot<\/pre>\n
Search engine crawler access via robots.txt file<\/h2>\n
There are quite a few options when it comes to controlling how your site is crawled with the robots.txt<\/strong> file.<\/p>\n
The User-agent:<\/strong> rule specifies which User-agent the rule applies to, and *<\/strong> is a wildcard matching any User-agent.<\/p>\n
Disallow:<\/strong> sets the files or folders that are not allowed<\/strong> to be crawled.<\/p>\n
\n
Here are some of the most common uses of the robots.txt<\/strong> file:<\/p>\n
Set a crawl delay for all search engines<\/a>
\nAllow all search engines to crawl website<\/a>
\nDisallow all search engines from crawling website<\/a>
\nDisallow one particular search engines from crawling website<\/a>
\nDisallow all search engines from particular folders<\/a>
\nDisallow all search engines from particular files<\/a>
\nDisallow all search engines but one<\/a><\/p>\n<\/div>\n
 <\/a>
\nSet a crawl delay for all search engines<\/strong>:<\/p>\n
If you had 1,000 pages on your website, a search engine could potentially index your entire site in a few minutes.<\/p>\n
However, this could cause high system resource usage<\/a> with all of those pages loaded in a short time period.<\/p>\n
A Crawl-delay:<\/strong> of 30<\/strong> seconds would allow crawlers to index your entire 1,000 page website in just 8.3 hours<\/strong><\/p>\n
A Crawl-delay:<\/strong> of 500<\/strong> seconds would allow crawlers to index your entire 1,000 page website in 5.8 days<\/strong><\/p>\n
You can set the Crawl-delay:<\/strong> for all search engines at once with:<\/p>\n
User-agent:<\/strong> *<\/strong> \nCrawl-delay:<\/strong> 30<\/strong><\/pre>\n
<\/p>\n
<\/a>
\nAllow all search engines to crawl website<\/strong>:By default search engines should be able to crawl your website, but you can also specify they are allowed<\/strong> with:<\/p>\n
User-agent:<\/strong> *<\/strong>\nDisallow:<\/strong><\/pre>\n<\/div>\n
 <\/a>
\nDisallow all search engines from crawling website<\/strong>:<\/p>\n
You can disallow<\/strong> any search engine from crawling your website, with these rules:<\/p>\n
User-agent:<\/strong> *<\/strong>\nDisallow:<\/strong> \/<\/strong><\/pre>\n
<\/p>\n
<\/a>
\nDisallow one particular search engines from crawling website<\/strong>:You can disallow<\/strong> just one specific search engine from crawling your website, with these rules:<\/p>\n
User-agent:<\/strong> Baiduspider<\/strong> \nDisallow:<\/strong> \/<\/strong><\/pre>\n<\/div>\n
 <\/a>
\nDisallow all search engines from particular folders<\/strong>:<\/p>\n
If we had a few directories like \/cgi-bin\/<\/strong>, \/private\/<\/strong>, and \/tmp\/<\/strong> we didn’t want bots to crawl we could use this:<\/p>\n
User-agent:<\/strong> *<\/strong> \nDisallow:<\/strong> \/cgi-bin\/<\/strong> \nDisallow:<\/strong> \/private\/<\/strong> \nDisallow:<\/strong> \/tmp\/<\/strong><\/pre>\n
<\/p>\n
<\/a>
\nDisallow all search engines from particular files<\/strong>:If we had files like contactus.htm<\/strong>, index.htm<\/strong>, and store.htm<\/strong> we didn’t want bots to crawl we could use this:<\/p>\n
User-agent:<\/strong> *<\/strong>\nDisallow:<\/strong> \/contactus.htm<\/strong>\nDisallow:<\/strong> \/index.htm<\/strong> \nDisallow:<\/strong> \/store.htm<\/strong><\/pre>\n<\/div>\n
 <\/a>
\nDisallow all search engines but one<\/strong>:<\/p>\n
If we only wanted to allow Googlebot<\/strong> access to our \/private\/<\/strong> directory and disallow all other bots we could use:<\/p>\n
User-agent:<\/strong> *<\/strong> \nDisallow:<\/strong> \/private\/<\/strong>  \nUser-agent:<\/strong> Googlebot<\/strong> \nDisallow:<\/strong><\/pre>\n
When the Googlebot<\/strong> reads our robots.txt<\/strong> file, it will see it is not disallowed from crawling any directories.<\/p>\n
\r\n
Professional Websites Without the Tech Headaches<\/strong><\/p>\r\n
Our team will build a beautiful WordPress site and take care of updates, security, and maintenance \u2013 so you can focus on running your business.<\/p>\r\n
Let Us Handle It for You<\/a><\/p>\r\n<\/div>\n","protected":false},"excerpt":{"rendered":"
In order for your website to be found by other people, search engine crawlers, also sometimes referred to as bots or spiders, will crawl your website looking for updated text and links to update their search indexes. How to Control search engine crawlers with a robots.txt file Website owners can instruct search engines on how Read More ><\/a><\/p>\n","protected":false},"author":57014,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4313,4288],"tags":[],"class_list":["post-282","post","type-post","status-publish","format-standard","hentry","category-seo","category-website"],"acf":[],"yoast_head":"\nHow to Stop Search Engines from Crawling your Website | InMotion Hosting<\/title>\n<meta name=\"description\" content=\"Stop search engines from crawling your website with a robots.txt file.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Stop Search Engines from Crawling your Website | InMotion Hosting\" \/>\n<meta property=\"og:description\" content=\"Stop search engines from crawling your website with a robots.txt file.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/\" \/>\n<meta property=\"og:site_name\" content=\"InMotion Hosting Support Center\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inmotionhosting\/\" \/>\n<meta property=\"article:published_time\" content=\"2011-11-02T16:07:31+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-12T20:57:25+00:00\" \/>\n<meta name=\"author\" content=\"InMotion Hosting Contributor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@https:\/\/twitter.com\/InMotionHosting\" \/>\n<meta name=\"twitter:site\" content=\"@InMotionHosting\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"InMotion Hosting Contributor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/\"},\"author\":{\"name\":\"InMotion Hosting Contributor\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644\"},\"headline\":\"How to Stop Search Engines from Crawling your Website\",\"datePublished\":\"2011-11-02T16:07:31+00:00\",\"dateModified\":\"2025-06-12T20:57:25+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/\"},\"wordCount\":654,\"commentCount\":166,\"publisher\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\"},\"articleSection\":[\"Search Engine Optimization\",\"Website\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/\",\"name\":\"How to Stop Search Engines from Crawling your Website | InMotion Hosting\",\"isPartOf\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#website\"},\"datePublished\":\"2011-11-02T16:07:31+00:00\",\"dateModified\":\"2025-06-12T20:57:25+00:00\",\"description\":\"Stop search engines from crawling your website with a robots.txt file.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.inmotionhosting.com\/support\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to Stop Search Engines from Crawling your Website\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#website\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/\",\"name\":\"InMotion Hosting Support Center\",\"description\":\"Web Hosting Support & Tutorials\",\"publisher\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.inmotionhosting.com\/support\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\",\"name\":\"InMotion Hosting\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg\",\"contentUrl\":\"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg\",\"width\":696,\"height\":696,\"caption\":\"InMotion Hosting\"},\"image\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/inmotionhosting\/\",\"https:\/\/x.com\/InMotionHosting\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644\",\"name\":\"InMotion Hosting Contributor\",\"description\":\"InMotion Hosting contributors are highly knowledgeable individuals who create relevant content on new trends and troubleshooting techniques to help you achieve your online goals!\",\"sameAs\":[\"https:\/\/www.linkedin.com\/company\/inmotion-hosting\/\",\"https:\/\/x.com\/https:\/\/twitter.com\/InMotionHosting\"],\"url\":\"https:\/\/www.inmotionhosting.com\/support\/author\/inmotion-hosting-contributor\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to Stop Search Engines from Crawling your Website | InMotion Hosting","description":"Stop search engines from crawling your website with a robots.txt file.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/","og_locale":"en_US","og_type":"article","og_title":"How to Stop Search Engines from Crawling your Website | InMotion Hosting","og_description":"Stop search engines from crawling your website with a robots.txt file.","og_url":"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/","og_site_name":"InMotion Hosting Support Center","article_publisher":"https:\/\/www.facebook.com\/inmotionhosting\/","article_published_time":"2011-11-02T16:07:31+00:00","article_modified_time":"2025-06-12T20:57:25+00:00","author":"InMotion Hosting Contributor","twitter_card":"summary_large_image","twitter_creator":"@https:\/\/twitter.com\/InMotionHosting","twitter_site":"@InMotionHosting","twitter_misc":{"Written by":"InMotion Hosting Contributor","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/#article","isPartOf":{"@id":"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/"},"author":{"name":"InMotion Hosting Contributor","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644"},"headline":"How to Stop Search Engines from Crawling your Website","datePublished":"2011-11-02T16:07:31+00:00","dateModified":"2025-06-12T20:57:25+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/"},"wordCount":654,"commentCount":166,"publisher":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#organization"},"articleSection":["Search Engine Optimization","Website"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/","url":"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/","name":"How to Stop Search Engines from Crawling your Website | InMotion Hosting","isPartOf":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#website"},"datePublished":"2011-11-02T16:07:31+00:00","dateModified":"2025-06-12T20:57:25+00:00","description":"Stop search engines from crawling your website with a robots.txt file.","breadcrumb":{"@id":"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.inmotionhosting.com\/support\/website\/how-to-stop-search-engines-from-crawling-your-website\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inmotionhosting.com\/support\/"},{"@type":"ListItem","position":2,"name":"How to Stop Search Engines from Crawling your Website"}]},{"@type":"WebSite","@id":"https:\/\/www.inmotionhosting.com\/support\/#website","url":"https:\/\/www.inmotionhosting.com\/support\/","name":"InMotion Hosting Support Center","description":"Web Hosting Support & Tutorials","publisher":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inmotionhosting.com\/support\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.inmotionhosting.com\/support\/#organization","name":"InMotion Hosting","url":"https:\/\/www.inmotionhosting.com\/support\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/","url":"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg","contentUrl":"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg","width":696,"height":696,"caption":"InMotion Hosting"},"image":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inmotionhosting\/","https:\/\/x.com\/InMotionHosting"]},{"@type":"Person","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644","name":"InMotion Hosting Contributor","description":"InMotion Hosting contributors are highly knowledgeable individuals who create relevant content on new trends and troubleshooting techniques to help you achieve your online goals!","sameAs":["https:\/\/www.linkedin.com\/company\/inmotion-hosting\/","https:\/\/x.com\/https:\/\/twitter.com\/InMotionHosting"],"url":"https:\/\/www.inmotionhosting.com\/support\/author\/inmotion-hosting-contributor\/"}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"primary_category":{"id":4288,"name":"Website","slug":"website","link":"https:\/\/www.inmotionhosting.com\/support\/website\/"},"_links":{"self":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/282","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/users\/57014"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/comments?post=282"}],"version-history":[{"count":11,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/282\/revisions"}],"predecessor-version":[{"id":130473,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/282\/revisions\/130473"}],"wp:attachment":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/media?parent=282"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/categories?post=282"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/tags?post=282"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}