Robot.Txt Generator

Navigation

Sponsors

Save this to Del.icio.us

Robot.txt Generator

Put Robot.txt Generator on Your Site

Background Info

Robot.txt Generator
Robot.txt allow you to determine which parts of your sites a bot can access and which parts of the site it cannot. Using the tool is pretty self-explanatory. However when specifying the URLs that should be excluded note that it should be 1 url per line.

Example (As shown in the textarea):

URL: You Type:
http://www.YOUR-DOMAIN.com/index.html /index.html
http://www.YOUR-DOMAIN.com/blog/private-info.html /blog/private-info.html
http://www.YOUR-DOMAIN.com/cgi-bin/ /cgi-bin/

What to Do With the Generated File

Once you have generated a code you need to copy and paste it into a text file called "robot.txt" and then upload it into the root directory of your server (usually "public_html").

Specifying Different Rules For Different User Agents

When you are specifying different rules for different bots, you need to treat each one seperately. To clarify what is meant please look at the following example:

In Robot.txt

User-agent: Slurp
Disallow: /images/

User-agent: fast
Disallow: /

etc...

Now fill in the form below to generate your custom robot.txt code.


Allowed User Agents:

Exclude: Nothing   Everything   The URL Below:


[ Back to Website Tools and Tutorials ]


Forums | Search | Donate | Links | Contact | Templates | Privacy Policy
Copyright © 2017 SiteToolCenter.com

Website Design

Best Viewed With Mozilla FireFox