While searching the Internet I stumbled upon a very interesting article on Seomoz about how to hide your content from Search Engines (Google, Yahoo, MSN, etc).

When we’re talking about Search Engine Optimization we want to highlight content and promote pages/sites, try to bring them to the surface as much as possible and get the best rankings for certain keywords. But a lot of people want exactly the opposite, to hide and obfuscate that content, try to make it invisible to search engines.

There are mainly two reasons from my point of view and these are:

1. Confidentiality.

Many persons just want to keep their work hidden from vicious predators that SQL inject passwords, scrape content and generally steal others work.

2. Duplicate Content.

One of the very basic SEO rules when building a site is to stay away from duplicate content as much as possible. When a page is too similar with another (content, keyword descriptors, etc) search engines may think these pages were created only for the soul purpose of having as much content as possible so they penalize these pages. In the last years Google and other top search engines struggled to bring the best most relevant content to the top results so it would be wise to write long, quality articles to avoid duplicate content. The pages who are thought to be duplicates are moved to the supplementals section which is at the end of the results which means no traffic at all

Anyway the best methods and the most accessible ones too for avoiding these 2 problems are:

1. Using the robots.txt which can be found in the root of your site: www.myexample.com/robots.txt.

2. Using META descriptors : By adding < META NAME=”ROBOTS” CONTENT=”NOINDEX,FOLLOW” > in the source code of your page (works per page only) will allow spiders to crawl your page, even your links but the only thing which will not get indexed will be your Content.