SEO – Restrict page or link not to crawl by Search Engine

SEO – Restrict page or link not to crawl by Search Engine
2 votes, 5.00 avg. rating (96% score)


There are other ways to instruct search engine crawlers how to behave when indexing and following links besides using a robots.txt file.

Restrict Page not to get crawled

Add a noindex meta tag to the head section of a page

<meta name="robots" content="noindex" />

This will not prevent crawlers from visiting the page, but it should stop them from indexing the page and listing it among the search results.

Restrict link not to get crawled

Apply the “nofollow” value to the rel attribute of a hyperlink

<a href="/Account/Register" rel="nofollow">Register</a>

This does not prevent the crawler or bot from actually following the link. Different crawlers behave in different ways, but most of the leading search engine crawlers will not include the link in their calculations for assessing the value of the target. nofollow is more often used for external links than internal links.


Few of my clients asked for this to SharePoint public site pages. Hope this tip helps

November 27, 2014 В· Adi В· No Comments
Tags: ,  В· Posted in: SEO

Leave a Reply

What is 11 + 10 ?
Please leave these two fields as-is:
IMPORTANT! To be able to proceed, you need to solve the following simple math (so we know that you are a human) :-)