Robots.txt

ISRdesign · # 1 (**permalink**)

Examples

This example allows all robots to visit all files because the wildcard "*" specifies all robots:

Code:

User-agent: *
Disallow:

This example keeps all robots out:

Code:

User-agent: *
Disallow: /

All HERE

Quote:

Questions ? ask me

Shadab · # 2 (**permalink**)

The very basics of the robots.txt file.
Thanks for posting.

At GP, i used this :

Code:

User-agent: *
Disallow: /archive/

to de-index the archived posts from Google.

ISRdesign · # 3 (**permalink**)

why you don't wanna add /archive/ to google?

Shadab · # 4 (**permalink**)

Because it doesn't exist anymore!
As I renamed it to /sitemap/

So, i used that code to remove the 'old' non-existent content from the Google Index.

ISRdesign · # 5 (**permalink**)

yes, i understand