Robots Meta Tag

Example Robots Meta Tag;

<meta name="robots" content="index, nofollow" />

I could use the above meta tag to tell Google to index the page but not to follow any links on the page, if for some reason, I did not want the page to appear in Google search results.

By default, Googlebot will index a page and follow links to it. So there’s no need to tag pages with content values of INDEX or FOLLOW. GOOGLE

There are various instructions you can make use of in your Robots Meta Tag, but remember Google by default WILL index and follow links, so you have NO need to include that as a command – you can leave the robots meta out completely – and probably should if you don’t have a clue.

Googlebot understands any combination of lowercase and uppercase. GOOGLE.

Valid values for Robots Meta Tag ”CONTENT” attribute are: “INDEX“, “NOINDEX“, “FOLLOW“, and  “NOFOLLOW“.

Example Usage:

• META NAME=”ROBOTS” CONTENT=”NOINDEX, FOLLOW”
• META NAME=”ROBOTS” CONTENT=”INDEX, NOFOLLOW”
• META NAME=”ROBOTS” CONTENT=”NOINDEX, NOFOLLOW”
• META NAME=”ROBOTS” CONTENT=”NOARCHIVE”
• META NAME=”GOOGLEBOT” CONTENT=”NOSNIPPET”

Google will understand the following and interprets the following robots meta tag values:

• NOINDEX – prevents the page from being included in the index.
• NOFOLLOW – prevents Googlebot from following any links on the page. (Note that this is different from the link-level NOFOLLOW attribute, which prevents Googlebot from following an individual link.)
• NOARCHIVE – prevents a cached copy of this page from being available in the search results.
• NOSNIPPET – prevents a description from appearing below the page in the search results, as well as prevents caching of the page.
• NOODP – blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.
• NONE – equivalent to “NOINDEX, NOFOLLOW”.

Robots META Tag Quick Reference

Terms	Googlebot	Slurp	BingBot	Teoma
NoIndex	YES	YES	YES	YES
NoFollow	YES	YES	YES	YES
NoArchive	YES	YES	YES	YES
NoSnippet	YES	NO	NO	NO
NoODP	YES	YES	YES	NO
NoYDIR	NO	YES	NO	NO
NoImageIndex	YES	NO	NO	NO
NoTranslate	YES	NO	NO	NO
Unavailable_After	YES	NO	NO	NO

I’ve included the robots meta tag in my tutorial as this IS one of only a few meta tags / HTML head elements I focus on when it comes to managing Googlebot and Bingbot. At a page level – it is a powerful way to control if your pages are returned in search results pages.

These meta tags go in the [HEAD] section of a [HTML] page and represent the only tags for Google I care about. Just about everything else you can put in the [HEAD] of your HTML document is quite unnecessary and maybe even pointless (for Google optimisation, anyway).