What is a Bot ?

A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.

Note that "recursive" here doesn't limit the definition to any specific traversal algorithm; even if a robot applies some heuristic to the selection and order of documents to visit and spaces out requests over a long space of time, it is still a robot.

Normal Web browsers are not robots, because they are operated by a human, and don't automatically retrieve referenced documents (other than inline images).


Robot Exclusion :

The robots exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is, otherwise, publicly viewable.

Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard complements Sitemaps, a robot inclusion standard for websites.


Some Examples :

The following codes can be used in the robots.txt file for the purposes written alongwith.

1. Allowing all the robots to index the site.
   
   Code:

   User-agent: *
   Disallow:

2. Disallowing all the robots to index the site.
   
   Code:

   User-agent: *
   Disallow: /

3. Disallowing only the specified directories to all the robots.
   
   Code:

   User-agent: *
   Disallow: /cgi-bin/
   Disallow: /secret/
   Disallow: /another-secret/

4. Preventing one robot from indexing a specific directory
   
   Code:

   User-agent: A-Bad-Robot
   Disallow: /some-secret/

5. Disallowing only a single file to all robots.
   
   Code:

   User-agent: *
   Disallow: /private/secret-file.htm

6. Specifying a Sitemap :
   
   Code:

   Sitemap: http://www.yoursite.com/path/to/sitemap.xml.gz

7. Delaying the Crawling :
   
   Code:

   User-agent: *
   Crawl-delay: 5

   This will instruct all the robots to request different documents at an interval of 5 seconds.
   
   This can be used to lower the no. of requests you receive per second,
   if you are being flooded by a huge no. of bots and requests.
   And hence lower the server load too.

thanx,
Shadab.

Note : Googlebot doesn't accept the crawl delay


I can add a feature for spiders who accept Sitemap like googlebot, Msnbot, Yahoo :
Add :
Just be careful the syntax :
- no space between 'Sitemap' and ':'
- a space between ':' and your address,
- and last you must put the full URL and must be into your website. You can't add an URL from another website
- and of course you must have a correct sitemap page at sitemap.php, lol (you could use another name or extension)

This seems cool, thanks mate.

this is an awesome post...do you mind if i use it on my forums and provide you with a vBSEO linkback??

Let me know

Yes, and
Thanks you for your appreciation.




Thanx,
Shadab.

Thank's Shadab !
I should in my future site , I will add a site map .. ! Because my visitors want to see everything in the site

Welcome mate !

I highly recommend you use a sitemap if your site contains a LOT of deep pages.
Makes it easy for the search engines to index and crawl all your page.

:twisted:

wow, this is neat, i'll probably be using this on my site, thanks!

fantastic post! Great for beginner webmasters..