{"id":583,"date":"2013-02-21T20:00:49","date_gmt":"2013-02-22T01:00:49","guid":{"rendered":"https:\/\/www.inmotionhosting.com\/support\/2013\/02\/21\/identify-and-block-bad-robots-from-website\/"},"modified":"2025-03-21T18:36:36","modified_gmt":"2025-03-21T22:36:36","slug":"identify-and-block-bad-robots-from-website","status":"publish","type":"post","link":"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/","title":{"rendered":"Identify and block bad robots from website"},"content":{"rendered":"\n<p>In this article I\u2019m going to teach you how you can identify and then block bad robots from your website, that could be possibly using up system resources on your server.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"#what\">What is a Bad Robot?<\/a><\/li>\n\n\n\n<li><a href=\"#identify\">Identify a Bad Robot<\/a><\/li>\n\n\n\n<li><a href=\"#block\">Block a Bad Robot<\/a><\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"what\">What is a Bad Robot?<\/h2>\n\n\n\n<p>There are many reasons that an automated robot would be trying to crawl through your website, the most common is for the large search engines such as Google or Bing to be able to find all the content on your website, so they can then serve it up to their users via search queries they are doing on those services.<\/p>\n\n\n\n<p>These robots are supposed to follow rules that you place in a <strong>robots.txt<\/strong> file, such as how frequently they are allowed to request pages, and from what directories they are allowed to crawl through. They should also be supplying a consistent valid <strong>User-Agent<\/strong> string that identifies the requests as a bot request.<\/p>\n\n\n\n<p>A bad robot usually will ignore <strong>robots.txt<\/strong> rules, request pages too quickly, re-visit your site too frequently, attempt to harvest email addresses, or in general simply provide no value back to your website. When a good robot crawls your site, this is typically so other people can find your content and then be directed to it from a search engine. When a bad robot is crawling through your site it could be for malicious intensions such as trying to copy your content so that they can use it as their own.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"identify\">Identify a Bad Robot<\/h2>\n\n\n\n<p>Using the steps below, I\u2019ll show you some good steps to take in order to verify if a robot is a good or bad one.<\/p>\n\n\n\n<p class=\"alert alert-info\">Please note in order to follow these steps you would need to be on either a <a href=\"https:\/\/www.inmotionhosting.com\/cloud-vs-vps\">VPS<\/a> (Virtual Private Server) or dedicated server that has SSH access. If you\u2019re on a shared server you could read our guide on <a href=\"https:\/\/www.inmotionhosting.com\/support\/edu\/cpanel\/enable-raw-access-logs-in-cpanel\/\">enabling raw access log archiving in cPanel<\/a> to be able to view the same data, but it would have to be on your own local computer.<\/p>\n\n\n\n<ol class=\"wp-block-list article_list\">\n<li><a href=\"https:\/\/www.inmotionhosting.com\/support\/server\/ssh\/how-to-login-ssh\/\">Login to your server via SSH<\/a>.<\/li>\n\n\n\n<li>Navigate to your user\u2019s home directory where the Apache access logs are stored, in this case our username is <strong>userna5<\/strong>, so we\u2019ll use the following command:<br><pre>cd ~userna5\/access-logs\/<\/pre><\/li>\n\n\n\n<li>We can now use the following command to see all <strong>User-Agents<\/strong> that have requests to our <strong>example.com<\/strong> website:<br><pre>cat\u00a0example.com\u00a0| awk -F'\"' '{print $6}' | sort | uniq -c | sort -n<\/pre><br>This gives us back the output:<br><code>638 Mozilla\/5.0 (compatible; Googlebot\/2.1; +https:\/\/www.google.com\/bot.html)<\/code><br><code>1015 msnbot-UDiscovery\/2.0b (+https:\/\/search.msn.com\/msnbot.htm)<\/code><br><code>1344 Mozilla\/5.0 (en-US) AppleWebKit\/537.4 (KHTML, like Gecko) Chrome\/22.0.1229 Safari\/537.4 pss-webkit-request<\/code><br><code>21937 \u2013<\/code><br>So in this case, there have been (21,937) requests that aren\u2019t supplying a <strong>User-Agent<\/strong> string. This is an immediate red flag, as typically any human visitor requesting a page from your website should have the <strong>User-Agent<\/strong> of their web-browser in each request.<br>We can also see that the next highest level of requests came from something calling itself <strong>pss-webkit-request<\/strong>, followed by <strong>msnbot-UDiscovery\/2.0b<\/strong>, and then <strong>Googlebot\/2.1<\/strong><\/li>\n\n\n\n<li>First we can see all of the requests that didn\u2019t provide a <strong>User-Agent<\/strong> string, and then view all of the unique IP addresses that sent those requests in, with the following code:<br><pre>cat example.com | awk -F'\"' '$6 ~ \"-\"' | awk '{print $1}' | sort -n | uniq -c | sort -n<\/pre><br>In this example here are the top IPs that had requests without a User-Agent string:<br><code>421 74.125.176.94 17. 434 74.125.176.85 18. 463 74.125.176.95<\/code><\/li>\n\n\n\n<li>We can now search for these IP address against our access log to see what might going on with their requests. The following command is going to look for what <strong>User-Agent<\/strong> strings are coming from the one <strong>74.125.176.95<\/strong> IP address which had the most requests:<br><pre>grep 74.125.176.95 example.com | awk -F'\"' '{print $6}' | sort | uniq -c | sort -n<\/pre><br>This gives us back:<br><code>7 Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/537.17 (KHTML, like Gecko) Chrome\/24.0.1312.57 Safari\/537.17<\/code><br><code>11 Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit\/537.17 (KHTML, like Gecko) Chrome\/24.0.1312.57 Safari\/537.17 29 Mozilla\/5.0 (en-US) AppleWebKit\/537.4 (KHTML, like Gecko) Chrome\/22.0.1229 Safari\/537.4 pss-webkit-request 434 \u2013<\/code><br>Again this is another red flag, typically requests coming from a single IP address will use the same <strong>User-Agent<\/strong> string for each request. We can go a step further and run the following command to find out some information about the IP address:<br><pre>whois 74.125.176.95<\/pre>Some of the pertinent information that command gives us back for that IP is:<br><code>NetRange: 74.125.0.0 \u2013 74.125.255.255 30. <\/code><br><code>NetName: GOOGLE<\/code><br><code>OrgName: Google Inc.<\/code><br><code>OrgId: GOGL<\/code><br><code>Address: 1600 Amphitheatre Parkway<\/code><br><code>City: Mountain View<\/code><br><code>StateProv: CA<\/code><br><code>PostalCode: 94043<\/code><br><code>Country: US<\/code><br><code>RegDate: 2000-03-30<\/code><br><code>Updated: 2011-09-24<\/code><br><code>Ref: https:\/\/whois.arin.net\/rest\/org\/GOGL<\/code><br><br>This is an IP address that belongs to Google, and so are all the other IPs we saw coming from the <strong>74.125<\/strong> IP range. However this isn\u2019t coming from the official Googlebot crawler which would be identified as such by the <strong>User-Agent<\/strong> string, but instead these are requests from their AppEngine\/Cloud service.<br><br>So these requests are from custom crawlers more than likely that other users have made, and in some cases they could simply be trying to index your content for their own purposes instead of providing links back to you, which would fall under our definition of a bad robot.<\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"block\">Block a Bad Robot<\/h2>\n\n\n\n<p>Now that you understand a bit about how to identify a possible bad robot, the next step would be to probably block that bad robot if they\u2019ve been causing problems with your website usage.<\/p>\n\n\n\n<p>Using the steps below I\u2019ll show you how we can block the entire range of <strong>74.125<\/strong> IPs we were seeing from accessing the <strong>example.com<\/strong> website, but still allow them to request if they do happen to mention <strong>Google<\/strong> in their <strong>User-Agent<\/strong> string of the request.<\/p>\n\n\n\n<ol class=\"wp-block-list article_list\">\n<li>Edit the <strong>.htaccess<\/strong> file for your website with the following command:<br><pre>vim ~userna5\/public_html\/.htaccess<\/pre><br>Once the <strong>vim<\/strong> text-editor has loaded the file, hit i to enter <strong>Insert<\/strong> mode, enter in the following code (in most SSH clients you can also right-click to paste text from the clipboard):<br><code>ErrorDocument 503 \"Site disabled for crawling\" <\/code><br><code>RewriteEngine On <\/code><br><code>RewriteCond %{HTTP_USER_AGENT} !^.*(Google).*$ <\/code><br><code>RewriteCond %{REMOTE_ADDR} ^74.125 <\/code><br><code>RewriteRule .* \u2013 [R=503,L]<\/code><br><br>Once you\u2019ve entered that information in, hit <strong>Esc<\/strong> to exit <strong>Insert<\/strong> mode, then hold down <strong>Shift<\/strong> and type in <strong>ZZ<\/strong> to save the file.<br>These rules will first check if the <strong>User-Agent<\/strong> has the word <strong>Google<\/strong> in it anywhere, if it does not, it moves onto the next rule, which checks to see if the IP address begins with <strong>74.125<\/strong>. If the IP matches then it will serve them a <strong>503<\/strong> response of <strong>Site disabled for crawling<\/strong> which uses up very minimal server resources, instead of allowing them to hit your various website pages that could be causing a lot of usage.<\/li>\n\n\n\n<li>As a bonus you can then use the following command to check up on how many requests you\u2019re saving your server from having to serve to these bad bots with this command:<br><pre>cat ~userna5\/access-logs\/example.com | grep \"74.125\" | awk '$9 ~ 503' | cut -d[ -f2 | cut -d] -f1 | awk -F: '{print $2\":00\"}' | sort -n | uniq -c | sed 's\/[ ]*\/\/'<\/pre><br>This gives you back the amount of bots blocked per hour with your rule:<br><code>2637 14:00 <\/code><br><code>2823 15:00 <\/code><br><code>2185 16:00<\/code><\/li>\n<\/ol>\n\n\n\n<p>You should now understand how to identify and then block bad robots from causing usage problems on your website with their excessive requests.<\/p>\n\n\n<div class=\"jumbotron\"><p style=\"margin-top:0;\"><img decoding=\"async\" src=\"https:\/\/design.inmotionhosting.com\/assets\/icons\/custom\/security.svg\" align=\"right\" alt=\"Secure VPS Hosting\" height=\"80\">Enjoy high-performance, lightning-fast servers with increased security and maximum up-time with our <a href=\"https:\/\/www.inmotionhosting.com\/vps-hosting\">Secure VPS Hosting<\/a>!<\/p>\r\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/design.inmotionhosting.com\/assets\/icons\/standard\/check-blue.svg\" alt=\"check mark\" width=\"24\" height=\"24\" \/>Linux VPS    <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/design.inmotionhosting.com\/assets\/icons\/standard\/check-blue.svg\" alt=\"check mark\" width=\"24\" height=\"24\" \/>cPanel or Control Web Panel    <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/design.inmotionhosting.com\/assets\/icons\/standard\/check-blue.svg\" alt=\"check mark\" width=\"24\" height=\"24\" \/>Scalable    <img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/design.inmotionhosting.com\/assets\/icons\/standard\/check-blue.svg\" alt=\"check mark\" width=\"24\" height=\"24\" \/>Website Migration Assistance<\/p>\r\n<p style=\"margin:10px 0;\"><a class=\"btn btn-primary btn-lg\" href=\"https:\/\/www.inmotionhosting.com\/vps-hosting\/linux-vps\">Linux VPS Hosting<\/a><\/p><\/div>\n","protected":false},"excerpt":{"rendered":"<p>In this article I\u2019m going to teach you how you can identify and then block bad robots from your website, that could be possibly using up system resources on your server. What is a Bad Robot? There are many reasons that an automated robot would be trying to crawl through your website, the most common<a class=\"moretag\" href=\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/\"> Read More ><\/a><\/p>\n","protected":false},"author":57014,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4288],"tags":[],"class_list":["post-583","post","type-post","status-publish","format-standard","hentry","category-website"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.1.1 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Identify and block bad robots from website | InMotion Hosting<\/title>\n<meta name=\"description\" content=\"In this article I&#039;m going to teach you how you can identify and then block bad robots from your website, that could be possibly using up system resources on your server.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Identify and block bad robots from website | InMotion Hosting\" \/>\n<meta property=\"og:description\" content=\"In this article I&#039;m going to teach you how you can identify and then block bad robots from your website, that could be possibly using up system resources on your server.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/\" \/>\n<meta property=\"og:site_name\" content=\"InMotion Hosting Support Center\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inmotionhosting\/\" \/>\n<meta property=\"article:published_time\" content=\"2013-02-22T01:00:49+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-03-21T22:36:36+00:00\" \/>\n<meta name=\"author\" content=\"InMotion Hosting Contributor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@https:\/\/twitter.com\/InMotionHosting\" \/>\n<meta name=\"twitter:site\" content=\"@InMotionHosting\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"InMotion Hosting Contributor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/\"},\"author\":{\"name\":\"InMotion Hosting Contributor\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644\"},\"headline\":\"Identify and block bad robots from website\",\"datePublished\":\"2013-02-22T01:00:49+00:00\",\"dateModified\":\"2025-03-21T22:36:36+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/\"},\"wordCount\":1042,\"commentCount\":7,\"publisher\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\"},\"articleSection\":[\"Website\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/\",\"name\":\"Identify and block bad robots from website | InMotion Hosting\",\"isPartOf\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#website\"},\"datePublished\":\"2013-02-22T01:00:49+00:00\",\"dateModified\":\"2025-03-21T22:36:36+00:00\",\"description\":\"In this article I'm going to teach you how you can identify and then block bad robots from your website, that could be possibly using up system resources on your server.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.inmotionhosting.com\/support\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Identify and block bad robots from website\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#website\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/\",\"name\":\"InMotion Hosting Support Center\",\"description\":\"Web Hosting Support &amp; Tutorials\",\"publisher\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.inmotionhosting.com\/support\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\",\"name\":\"InMotion Hosting\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg\",\"contentUrl\":\"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg\",\"width\":696,\"height\":696,\"caption\":\"InMotion Hosting\"},\"image\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/inmotionhosting\/\",\"https:\/\/x.com\/InMotionHosting\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644\",\"name\":\"InMotion Hosting Contributor\",\"description\":\"InMotion Hosting contributors are highly knowledgeable individuals who create relevant content on new trends and troubleshooting techniques to help you achieve your online goals!\",\"sameAs\":[\"https:\/\/www.linkedin.com\/company\/inmotion-hosting\/\",\"https:\/\/x.com\/https:\/\/twitter.com\/InMotionHosting\"],\"url\":\"https:\/\/www.inmotionhosting.com\/support\/author\/inmotion-hosting-contributor\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Identify and block bad robots from website | InMotion Hosting","description":"In this article I'm going to teach you how you can identify and then block bad robots from your website, that could be possibly using up system resources on your server.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/","og_locale":"en_US","og_type":"article","og_title":"Identify and block bad robots from website | InMotion Hosting","og_description":"In this article I'm going to teach you how you can identify and then block bad robots from your website, that could be possibly using up system resources on your server.","og_url":"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/","og_site_name":"InMotion Hosting Support Center","article_publisher":"https:\/\/www.facebook.com\/inmotionhosting\/","article_published_time":"2013-02-22T01:00:49+00:00","article_modified_time":"2025-03-21T22:36:36+00:00","author":"InMotion Hosting Contributor","twitter_card":"summary_large_image","twitter_creator":"@https:\/\/twitter.com\/InMotionHosting","twitter_site":"@InMotionHosting","twitter_misc":{"Written by":"InMotion Hosting Contributor","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/#article","isPartOf":{"@id":"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/"},"author":{"name":"InMotion Hosting Contributor","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644"},"headline":"Identify and block bad robots from website","datePublished":"2013-02-22T01:00:49+00:00","dateModified":"2025-03-21T22:36:36+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/"},"wordCount":1042,"commentCount":7,"publisher":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#organization"},"articleSection":["Website"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/","url":"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/","name":"Identify and block bad robots from website | InMotion Hosting","isPartOf":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#website"},"datePublished":"2013-02-22T01:00:49+00:00","dateModified":"2025-03-21T22:36:36+00:00","description":"In this article I'm going to teach you how you can identify and then block bad robots from your website, that could be possibly using up system resources on your server.","breadcrumb":{"@id":"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.inmotionhosting.com\/support\/website\/identify-and-block-bad-robots-from-website\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inmotionhosting.com\/support\/"},{"@type":"ListItem","position":2,"name":"Identify and block bad robots from website"}]},{"@type":"WebSite","@id":"https:\/\/www.inmotionhosting.com\/support\/#website","url":"https:\/\/www.inmotionhosting.com\/support\/","name":"InMotion Hosting Support Center","description":"Web Hosting Support &amp; Tutorials","publisher":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inmotionhosting.com\/support\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.inmotionhosting.com\/support\/#organization","name":"InMotion Hosting","url":"https:\/\/www.inmotionhosting.com\/support\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/","url":"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg","contentUrl":"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg","width":696,"height":696,"caption":"InMotion Hosting"},"image":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inmotionhosting\/","https:\/\/x.com\/InMotionHosting"]},{"@type":"Person","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644","name":"InMotion Hosting Contributor","description":"InMotion Hosting contributors are highly knowledgeable individuals who create relevant content on new trends and troubleshooting techniques to help you achieve your online goals!","sameAs":["https:\/\/www.linkedin.com\/company\/inmotion-hosting\/","https:\/\/x.com\/https:\/\/twitter.com\/InMotionHosting"],"url":"https:\/\/www.inmotionhosting.com\/support\/author\/inmotion-hosting-contributor\/"}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"primary_category":{"id":4288,"name":"Website","slug":"website","link":"https:\/\/www.inmotionhosting.com\/support\/website\/"},"_links":{"self":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/583","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/users\/57014"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/comments?post=583"}],"version-history":[{"count":19,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/583\/revisions"}],"predecessor-version":[{"id":129810,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/583\/revisions\/129810"}],"wp:attachment":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/media?parent=583"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/categories?post=583"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/tags?post=583"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}