Robots Meta Tag
The content of the robots meta tag can contain:
- index - meaning index this page in search engines
- noindex - meaning don't index it
- follow - meaning follow the links on this page to find more pages to index
- nofollow - meaning don't follow the links
- all - supported by some search engines as equivalent to index,follow
- none - supported by some search engines as equivalent to noindex,nofollow
- noarchive - instruction for Google to not archive the page
- nosnippet - instruction for Google to not create a snippet of the page to display in search results
Where more than one value is used they need to be separated by commas.
Decent search engines will follow these instructions. Spam spiders will of course ignore them.
