Using a META tag to block bots from indexing a webpage Updated on August 16, 2021 by Carrie Smaha 2 Minutes, 3 Seconds to Read You can use a specific <META> tag to tell search engine robots to not index the content of a specific webpage. If you would like to find out how to stop bots from accessing your entire website please read our article on creating a Robots.txt file. You can also use the same tag to direct the bot to not scan the page for links to follow as well. This is a great tag to use if you are only trying to restrict one page on your website. Here’s a basic ‘no follow, no index’ tag in HTML code: <html> <head> <title>...</title> <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> </head> There are three important peices of information you need to keep in mind when using the <META> tag: Robots can ignore the tag if they wish. Typically, the bots that will ignore the <META> tag are rogue bots that are scanning the web for security vulnerablities to exploit or email harvesters used by spammers to grab email addresses. the NOFOLLOW directive only applies to links on the page. If the bot finds a link on another page without a NOFOLLOW it will still arrive on the page The NOFOLLOW in this tag is NOT the same as the rel=”nofollow” attribute you can set on an <a href> tag Writing a ROBOTS META Tag Like other <META> tags used in HTML they should be placed in the HEAD section of your HTML code just like in the example above. <META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW”> So lets break the above example down further. The “NAME” attribute will always be “ROBOTS“. The “CONTENT” attribute can have four different values: “INDEX”, “NOINDEX”, “FOLLOW”, “NOFOLLOW” Multiple comma separated values are allowed but only certain combinations make sense. If there’s no <META> tag for ROBOTS present the default is “INDEX, FOLLOW” so there’s no need to specify that. You can have the following: <META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW"> <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW"> <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"> If you need further assistance with blocking robots from indexing your webpages, please read our article on How to stop Search Engines from crawling your Website. If you need furhter assistance you can always contact our support department 24 hours a day 7 days a week. Share this Article Carrie Smaha Senior Manager Marketing Operations Carrie enjoys working on demand generation and product marketing projects that tap into multi-touch campaign design, technical SEO, content marketing, software design, and business operations. More Articles by Carrie Related Articles How to Fix the “550 No Such User Here” Email Error What is Node.js? How to Deploy Websites Generated by AI Builders on Your Server How to Use robots.txt Disallow to Block Crawlers and Protect Site Performance Content Security Policy (CSP) Headers – Complete Reference Guide Troubleshooting SSL Connection Errors: How to Fix HTTPS Issues How to Check and Repair a Database in phpMyAdmin 21 Reasons Your Website is Slow and How to Speed it Up Website Transfer Resources Ensure a Successful Website Transfer
