Robots.txt SEO | How to Write robots.txt for Google?

As we all seo’s know about robots.txt file in main root directory. Robots.txt is seen by search engine robots (crawlers) for information which files or folders to be indexed or not. Robots.txt is very important file if we are working on e commerce based site or portal. If our site, portal or e-commerce shopping cart has more than 1000 pages, robots.txt plays a role of captain for all pages.

Main role of robots.txt is to guide robots to index and noindex some of pages or directories on server, we you can say it root directory or website.

Some of seo’s are still confused and not full knowledge about robots.txt file. in last week I need to take interview for some of seo experienced professionals with 2+ years of experience, they all were well knowledge but they were just little bit confused for robots.txt. That inspired me to write some basic information and syntax about robots.txt.

Google Robot: googleBot
Yahoo Bot : slurp
MSN Bot : msnbot
Altavista : scooter

If you are media partner of Google and running Google Ads on Your Website, you should write this in robots.txt

Google bot that crawls the Ad sense on your site: User-agent: mediapartners-Google

If you want to invite all crawlers to crawl and index all pages and root directories, no need to write more tags, simple write this one:

User-agent: *

* is wild card character and allow robots to crawl all pages on your site.

If you want to disallow some file(s) or folder not to be crawled or indexed, just write,

Disallow: /folder name/
Disallow: /file-name.htm

Mention Sitemap in robots.txt
Sitemap: http://www.site-name.com/sitemap.xml
Sitemap: http://www.site-name.com/sitemap.htm

You can mention xml, htm(l), php or simple txt sitemap with proper direction in robots.txt

I would like some feed back or suggestions from you. Post comment and share some more information. Thanks.

Tags: How to write robots.txt, robots.txt google, file robots.txt, sitemap robots.txt, robots.txt disallow, robots.txt seo

Search Industry Updates: SEO , robots.txt
Advertisers


Share This Information with your friends