{"id":507,"date":"2013-01-15T22:15:44","date_gmt":"2013-01-16T03:15:44","guid":{"rendered":"https:\/\/www.inmotionhosting.com\/support\/2013\/01\/15\/locate-problematic-user-agents\/"},"modified":"2021-11-19T01:56:10","modified_gmt":"2021-11-19T06:56:10","slug":"locate-problematic-user-agents","status":"publish","type":"post","link":"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/","title":{"rendered":"Locate problematic user agents"},"content":{"rendered":"

In this article we’ll review how you can locate possibly problematic user agents from requests on your site, that could be causing additional resource usage on your server.<\/p>\n

This guide is meant for VPS or dedicated server customers that have SSH access to their server. If you happened to have setup a server load monitoring bash script<\/a>, or you’re utilizing one of the tools available mentioned in our advanced server load monitoring<\/a> article, and see that your server’s load average has been spiking, sometimes it’s a good idea to see if there are any particular user agents in your access logs that seem to be causing this.<\/p>\n

Locate high amounts of duplicate user agents<\/h2>\n

You can look at your Apache access logs in order to see a high amount of duplicate requests from one certain user agent using the steps below.<\/p>\n

    \n
  1. Login to your server via SSH<\/a>.<\/li>\n
  2. Navigate to the home directory for the website you’d like to investigate. In this example our cPanel username is userna5<\/strong>, and our domain name is example.com<\/strong>:cd \/home\/userna5\/access-logs<\/code><\/li>\n
  3. You can use the awk<\/strong> command to only print certain columns of the Apache log, we will then pipe |<\/strong> that to the sort<\/strong> command so that all of the user agents are sorted by name, we’ll then pipe that to the uniq -c<\/strong> command to uniquely count up how many times each user agent occurs, then finally we’ll pipe all that to the sort -n<\/strong> command so it sorts the user agents by how many total requests they had: awk -F\"\"\" '{print $(NF-1)}' example.com | sort | uniq -c | sort -n<\/code>You should get back something similar to this:\n

    1308 facebookexternalhit\/1.0 (+https:\/\/www.facebook.com\/externalhit_uatext.php)
    \n1861 facebookexternalhit\/1.1 (+https:\/\/www.facebook.com\/externalhit_uatext.php)
    \n1931 msnbot-media\/1.1 (+https:\/\/search.msn.com\/msnbot.htm)
    \n3293 Mozilla\/5.0 (compatible; AhrefsBot\/4.0; +https:\/\/ahrefs.com\/robot\/)<\/code><\/li>\n

  4. Now we can see that the AhrefsBot\/4.0<\/strong>  search engine crawler has far more requests than any other user agent currently. In this case let’s say that this website doesn’t necessarily want to be indexed by this search engine, and they are just worried about Google and Bing (MSN) crawling them. Then we could go ahead and use the robots.txt file to stop a search engine from crawling your website<\/a>.<\/li>\n
  5. If the requests from this user-agent continue to flood in and are causing a current issue on your server, the robots.txt rules won’t stop the requests until they request the rules again. You can go ahead and use our guide on how to block bad users based on their user agent string<\/a> to stop them immediately from being able to access your site.<\/li>\n<\/ol>\n

    You should understand how to locate possible problematic user agents from hitting your site and causing issues.<\/p>\n","protected":false},"excerpt":{"rendered":"

    In this article we’ll review how you can locate possibly problematic user agents from requests on your site, that could be causing additional resource usage on your server. This guide is meant for VPS or dedicated server customers that have SSH access to their server. If you happened to have setup a server load monitoring Read More ><\/a><\/p>\n","protected":false},"author":57014,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4364],"tags":[],"class_list":["post-507","post","type-post","status-publish","format-standard","hentry","category-server-usage"],"acf":[],"yoast_head":"\nLocate problematic user agents | InMotion Hosting<\/title>\n<meta name=\"description\" content=\"In this article we'll review how you can locate possibly problematic user agents from requests on your site, that could be causing additional resource usage on your server.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Locate problematic user agents | InMotion Hosting\" \/>\n<meta property=\"og:description\" content=\"In this article we'll review how you can locate possibly problematic user agents from requests on your site, that could be causing additional resource usage on your server.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/\" \/>\n<meta property=\"og:site_name\" content=\"InMotion Hosting Support Center\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/inmotionhosting\/\" \/>\n<meta property=\"article:published_time\" content=\"2013-01-16T03:15:44+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-11-19T06:56:10+00:00\" \/>\n<meta name=\"author\" content=\"InMotion Hosting Contributor\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@https:\/\/twitter.com\/InMotionHosting\" \/>\n<meta name=\"twitter:site\" content=\"@InMotionHosting\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"InMotion Hosting Contributor\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/\"},\"author\":{\"name\":\"InMotion Hosting Contributor\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644\"},\"headline\":\"Locate problematic user agents\",\"datePublished\":\"2013-01-16T03:15:44+00:00\",\"dateModified\":\"2021-11-19T06:56:10+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/\"},\"wordCount\":427,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\"},\"articleSection\":[\"Server Usage\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/\",\"name\":\"Locate problematic user agents | InMotion Hosting\",\"isPartOf\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#website\"},\"datePublished\":\"2013-01-16T03:15:44+00:00\",\"dateModified\":\"2021-11-19T06:56:10+00:00\",\"description\":\"In this article we'll review how you can locate possibly problematic user agents from requests on your site, that could be causing additional resource usage on your server.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.inmotionhosting.com\/support\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Locate problematic user agents\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#website\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/\",\"name\":\"InMotion Hosting Support Center\",\"description\":\"Web Hosting Support & Tutorials\",\"publisher\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.inmotionhosting.com\/support\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#organization\",\"name\":\"InMotion Hosting\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg\",\"contentUrl\":\"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg\",\"width\":696,\"height\":696,\"caption\":\"InMotion Hosting\"},\"image\":{\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/inmotionhosting\/\",\"https:\/\/x.com\/InMotionHosting\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644\",\"name\":\"InMotion Hosting Contributor\",\"description\":\"InMotion Hosting contributors are highly knowledgeable individuals who create relevant content on new trends and troubleshooting techniques to help you achieve your online goals!\",\"sameAs\":[\"https:\/\/www.linkedin.com\/company\/inmotion-hosting\/\",\"https:\/\/x.com\/https:\/\/twitter.com\/InMotionHosting\"],\"url\":\"https:\/\/www.inmotionhosting.com\/support\/author\/inmotion-hosting-contributor\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Locate problematic user agents | InMotion Hosting","description":"In this article we'll review how you can locate possibly problematic user agents from requests on your site, that could be causing additional resource usage on your server.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/","og_locale":"en_US","og_type":"article","og_title":"Locate problematic user agents | InMotion Hosting","og_description":"In this article we'll review how you can locate possibly problematic user agents from requests on your site, that could be causing additional resource usage on your server.","og_url":"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/","og_site_name":"InMotion Hosting Support Center","article_publisher":"https:\/\/www.facebook.com\/inmotionhosting\/","article_published_time":"2013-01-16T03:15:44+00:00","article_modified_time":"2021-11-19T06:56:10+00:00","author":"InMotion Hosting Contributor","twitter_card":"summary_large_image","twitter_creator":"@https:\/\/twitter.com\/InMotionHosting","twitter_site":"@InMotionHosting","twitter_misc":{"Written by":"InMotion Hosting Contributor","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/#article","isPartOf":{"@id":"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/"},"author":{"name":"InMotion Hosting Contributor","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644"},"headline":"Locate problematic user agents","datePublished":"2013-01-16T03:15:44+00:00","dateModified":"2021-11-19T06:56:10+00:00","mainEntityOfPage":{"@id":"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/"},"wordCount":427,"commentCount":0,"publisher":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#organization"},"articleSection":["Server Usage"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/","url":"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/","name":"Locate problematic user agents | InMotion Hosting","isPartOf":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#website"},"datePublished":"2013-01-16T03:15:44+00:00","dateModified":"2021-11-19T06:56:10+00:00","description":"In this article we'll review how you can locate possibly problematic user agents from requests on your site, that could be causing additional resource usage on your server.","breadcrumb":{"@id":"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.inmotionhosting.com\/support\/server\/server-usage\/locate-problematic-user-agents\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.inmotionhosting.com\/support\/"},{"@type":"ListItem","position":2,"name":"Locate problematic user agents"}]},{"@type":"WebSite","@id":"https:\/\/www.inmotionhosting.com\/support\/#website","url":"https:\/\/www.inmotionhosting.com\/support\/","name":"InMotion Hosting Support Center","description":"Web Hosting Support & Tutorials","publisher":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.inmotionhosting.com\/support\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.inmotionhosting.com\/support\/#organization","name":"InMotion Hosting","url":"https:\/\/www.inmotionhosting.com\/support\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/","url":"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg","contentUrl":"https:\/\/www.inmotionhosting.com\/support\/wp-content\/uploads\/2023\/02\/inmotion-hosting-logo-yoast.jpg","width":696,"height":696,"caption":"InMotion Hosting"},"image":{"@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/inmotionhosting\/","https:\/\/x.com\/InMotionHosting"]},{"@type":"Person","@id":"https:\/\/www.inmotionhosting.com\/support\/#\/schema\/person\/f9a4fc454cd1df128ee8e898d30d4644","name":"InMotion Hosting Contributor","description":"InMotion Hosting contributors are highly knowledgeable individuals who create relevant content on new trends and troubleshooting techniques to help you achieve your online goals!","sameAs":["https:\/\/www.linkedin.com\/company\/inmotion-hosting\/","https:\/\/x.com\/https:\/\/twitter.com\/InMotionHosting"],"url":"https:\/\/www.inmotionhosting.com\/support\/author\/inmotion-hosting-contributor\/"}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"primary_category":null,"_links":{"self":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/507","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/users\/57014"}],"replies":[{"embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/comments?post=507"}],"version-history":[{"count":5,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/507\/revisions"}],"predecessor-version":[{"id":92064,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/posts\/507\/revisions\/92064"}],"wp:attachment":[{"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/media?parent=507"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/categories?post=507"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.inmotionhosting.com\/support\/wp-json\/wp\/v2\/tags?post=507"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}