01 April 2010

Robots.txt Vs Robots Meta Tag

 Robots.txt

While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results.

In order to use a robots.txt file, you'll need to have access to the root of your domain (if you're not sure, check with your web hoster). If you don't have access to the root of a domain, you can restrict access using the robots meta tag.

Robots Meta Tag

To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index.

When we see the noindex meta tag on a page, Google will completely drop the page from our search results, even if other pages link to it. Other search engines, however, may interpret this directive differently. As a result, a link to the page can still appear in their search results.

Note that because we have to crawl your page in order to see the noindex meta tag, there's a small chance that Googlebot won't see and respect the noindex meta tag. If your page is still appearing in results, it's probably because we haven't crawled your site since you added the tag. (Also, if you've used your robots.txt file to block this page, we won't be able to see the tag either.)

If the content is currently in our index, we will remove it after the next time we crawl it. To expedite removal, use the URL removal request tool in Google Webmaster Tools.

Source Link: Google Webmaster Central

14 comments:

Sid said...

Thanks for sharing your ideas...

seo said...

This Blog is really best than other,Here not only blog is good but also we can achieve lot of knowledge about seo.important link is here mention which is imp link .

Bradley Holton said...

This site is extremely informative and has proved to be handy. Full of information and productive articles.

Regards,
Bradley Holton.

Bradley Holton said...

A good site with excellent articles. Thanks for such a wonderful informative and entertaining read. Quotations are a great way to inspire you to perform at your best and to remember sage advice from the smartest minds in the world.

Regards,
Bradley Holton.

offshore open source developers said...

thanks for sharing this.

SEO Company India said...

Really great. Hoping you will be there with more updates. :)

Complete SEO Packages said...

Our Professional SEO Expert use only natural SEO techniques of Buy One way Link & Link Building Service, for details visit http://www.links-building.com/.

plc scada training said...

Thank you. :D I’m glad you found it useful post

t28 Tower Noida said...

Hi,I got a lot of information from this blog.

rize tower noida

the paras rize t28
rise tower noida
rise tower in noida
the paras rise t28

t28 Tower Noida said...

Hi,I got a lot of information from this blog.

rize tower noida

the paras rize t28
rise tower noida
rise tower in noida
the paras rise t28

Unknown said...

Thanks for your information.We specialize in achieving top placement of website on three major search engines: Google, Bing, and Yahoo.


search engine optimization services

Fun Vihar said...

Wow I Appreciate you, and thanks for sharing this valuble information and it is

very useful for me and all.http://www.seoservicesinhyderabad.co.in/top-seo-

companies-in-hyderabad-india/

Anonymous said...

Wow I Appreciate you, and thanks for sharing this valuble information and it is

very useful for me and all.http://www.seoservicesinhyderabad.co.in/top-seo-

companies-in-hyderabad-india/

Search Engine Optimization Company India said...

Great Blog Post!!

Thanks for sharing a valuable information.

Keep up sharing..