{"id":610,"date":"2013-03-05T20:52:32","date_gmt":"2013-03-06T01:52:32","guid":{"rendered":"https:\/\/www.inmotionhosting.com\/support\/2013\/03\/05\/search-engine-crawlers-automated-robots-not-being-controlled\/"},"modified":"2021-08-16T23:19:56","modified_gmt":"2021-08-17T03:19:56","slug":"search-engine-crawlers-automated-robots-not-being-controlled","status":"publish","type":"post","link":"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/","title":{"rendered":"Search engine crawler and automated robot activity not being controlled"},"content":{"rendered":"

In this article I’m going to talk about the impact that search engine crawlers and automated robots activity not being controlled can have on your account’s resource usage.<\/p>\n

Why do bots need to be controlled?<\/h2>\n

A very common way that information spreads on the Internet is by the use of automated robots that crawl the Internet to find and index new content. This is great if you want to have your website content found on one of the large search engines such as Google or Bing, but at the same time these are automated bot visitors to your website, and they act much differently than a human visitor would.<\/p>\n

A human coming to your website will more than likely take some time to read the current page they’ve landed on, then if they decide to click on another link on your website, it’s going to probably be because it piqued their interest.<\/p>\n

A bot coming to your website is typically on a mission to find everything on your website, so they might start on the front page, and then simply spider out to each and every link on your website one after another till they’ve found everything.<\/p>\n

So one bot visitor, could potentially have the resource usage impact of hundreds, if not thousands of normal human visitors on your account. An extreme amount of resource usage coming from your account can eventually lead to an account suspension<\/a>, so it’s important to realize that typically this could be avoided if you were only allowing human visitors, and only select good bots onto your website.<\/p>\n

How do I control robots?<\/h2>\n

Luckily most rule abiding robots out there will follow a standardized robots.txt<\/strong> rules file, you can read more about how to stop search engines from crawling your website<\/a> which reviews the robots.txt<\/strong> file. You can also read about setting a crawl delay in Google webmaster tools<\/a> which has specific steps for how Google’s crawling robot can be controlled.<\/p>\n

Unfortunately not all automated robots are going to comply with your robots.txt<\/strong> rules, and in these cases it’s best to learn about how to block unwanted users from your site using .htaccess<\/a> which will allow you to make sure these bad robots don’t add to your account’s resource usage.<\/p>\n","protected":false},"excerpt":{"rendered":"

In this article I’m going to talk about the impact that search engine crawlers and automated robots activity not being controlled can have on your account’s resource usage. Why do bots need to be controlled? A very common way that information spreads on the Internet is by the use of automated robots that crawl the Read More ><\/a><\/p>\n","protected":false},"author":57014,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4313],"tags":[],"class_list":["post-610","post","type-post","status-publish","format-standard","hentry","category-seo"],"acf":[],"yoast_head":"\nSearch engine crawler and automated robot activity not being controlled | InMotion Hosting<\/title>\n<meta name=\"description\" content=\"In this article I'm going to talk about the impact that search engine crawlers and automated robots activity not being controlled can have on your account's resource usage.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Search engine crawler and automated robot activity not being controlled | InMotion Hosting\" \/>\n<meta property=\"og:description\" content=\"In this article I'm going to talk about the impact that search engine crawlers and automated robots activity not being controlled can have on your account's resource usage.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/\" \/>\n<meta property=\"og:site_name\" content=\"InMotion Hosting Support Center\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inmotionhosting\/\" \/>\n<meta property=\"article:published_time\" content=\"2013-03-06T01:52:32+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-08-17T03:19:56+00:00\" \/>\n<meta name=\"author\" content=\"InMotion Hosting Contributor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@https:\/\/twitter.com\/InMotionHosting\" \/>\n<meta name=\"twitter:site\" content=\"@InMotionHosting\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"InMotion Hosting Contributor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/\"},\"author\":{\"name\":\"InMotion Hosting Contributor\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644\"},\"headline\":\"Search engine crawler and automated robot activity not being controlled\",\"datePublished\":\"2013-03-06T01:52:32+00:00\",\"dateModified\":\"2021-08-17T03:19:56+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/\"},\"wordCount\":398,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\"},\"articleSection\":[\"Search Engine Optimization\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/\",\"name\":\"Search engine crawler and automated robot activity not being controlled | InMotion Hosting\",\"isPartOf\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#website\"},\"datePublished\":\"2013-03-06T01:52:32+00:00\",\"dateModified\":\"2021-08-17T03:19:56+00:00\",\"description\":\"In this article I'm going to talk about the impact that search engine crawlers and automated robots activity not being controlled can have on your account's resource usage.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.inmotionhosting.com\/support\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Search engine crawler and automated robot activity not being controlled\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#website\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/\",\"name\":\"InMotion Hosting Support Center\",\"description\":\"Web Hosting Support & Tutorials\",\"publisher\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.inmotionhosting.com\/support\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\",\"name\":\"InMotion Hosting\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg\",\"contentUrl\":\"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg\",\"width\":696,\"height\":696,\"caption\":\"InMotion Hosting\"},\"image\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/inmotionhosting\/\",\"https:\/\/x.com\/InMotionHosting\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644\",\"name\":\"InMotion Hosting Contributor\",\"description\":\"InMotion Hosting contributors are highly knowledgeable individuals who create relevant content on new trends and troubleshooting techniques to help you achieve your online goals!\",\"sameAs\":[\"https:\/\/www.linkedin.com\/company\/inmotion-hosting\/\",\"https:\/\/x.com\/https:\/\/twitter.com\/InMotionHosting\"],\"url\":\"https:\/\/www.inmotionhosting.com\/support\/author\/inmotion-hosting-contributor\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Search engine crawler and automated robot activity not being controlled | InMotion Hosting","description":"In this article I'm going to talk about the impact that search engine crawlers and automated robots activity not being controlled can have on your account's resource usage.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/","og_locale":"en_US","og_type":"article","og_title":"Search engine crawler and automated robot activity not being controlled | InMotion Hosting","og_description":"In this article I'm going to talk about the impact that search engine crawlers and automated robots activity not being controlled can have on your account's resource usage.","og_url":"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/","og_site_name":"InMotion Hosting Support Center","article_publisher":"https:\/\/www.facebook.com\/inmotionhosting\/","article_published_time":"2013-03-06T01:52:32+00:00","article_modified_time":"2021-08-17T03:19:56+00:00","author":"InMotion Hosting Contributor","twitter_card":"summary_large_image","twitter_creator":"@https:\/\/twitter.com\/InMotionHosting","twitter_site":"@InMotionHosting","twitter_misc":{"Written by":"InMotion Hosting Contributor","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/#article","isPartOf":{"@id":"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/"},"author":{"name":"InMotion Hosting Contributor","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644"},"headline":"Search engine crawler and automated robot activity not being controlled","datePublished":"2013-03-06T01:52:32+00:00","dateModified":"2021-08-17T03:19:56+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/"},"wordCount":398,"commentCount":0,"publisher":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#organization"},"articleSection":["Search Engine Optimization"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/","url":"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/","name":"Search engine crawler and automated robot activity not being controlled | InMotion Hosting","isPartOf":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#website"},"datePublished":"2013-03-06T01:52:32+00:00","dateModified":"2021-08-17T03:19:56+00:00","description":"In this article I'm going to talk about the impact that search engine crawlers and automated robots activity not being controlled can have on your account's resource usage.","breadcrumb":{"@id":"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.inmotionhosting.com\/support\/website\/seo\/search-engine-crawlers-automated-robots-not-being-controlled\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inmotionhosting.com\/support\/"},{"@type":"ListItem","position":2,"name":"Search engine crawler and automated robot activity not being controlled"}]},{"@type":"WebSite","@id":"https:\/\/www.inmotionhosting.com\/support\/#website","url":"https:\/\/www.inmotionhosting.com\/support\/","name":"InMotion Hosting Support Center","description":"Web Hosting Support & Tutorials","publisher":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inmotionhosting.com\/support\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.inmotionhosting.com\/support\/#organization","name":"InMotion Hosting","url":"https:\/\/www.inmotionhosting.com\/support\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/","url":"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg","contentUrl":"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg","width":696,"height":696,"caption":"InMotion Hosting"},"image":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inmotionhosting\/","https:\/\/x.com\/InMotionHosting"]},{"@type":"Person","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644","name":"InMotion Hosting Contributor","description":"InMotion Hosting contributors are highly knowledgeable individuals who create relevant content on new trends and troubleshooting techniques to help you achieve your online goals!","sameAs":["https:\/\/www.linkedin.com\/company\/inmotion-hosting\/","https:\/\/x.com\/https:\/\/twitter.com\/InMotionHosting"],"url":"https:\/\/www.inmotionhosting.com\/support\/author\/inmotion-hosting-contributor\/"}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"primary_category":null,"_links":{"self":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/610","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/users\/57014"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/comments?post=610"}],"version-history":[{"count":3,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/610\/revisions"}],"predecessor-version":[{"id":84881,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/610\/revisions\/84881"}],"wp:attachment":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/media?parent=610"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/categories?post=610"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/tags?post=610"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}