|
Background Info
Robot.txt allow you to determine which parts of your sites a bot can
access and which parts of the site it cannot. Using the tool is pretty
self-explanatory. However when specifying the URLs that should be excluded
note that it should be 1 url per line.
Example (As shown in the textarea):
|
URL:
|
|
You Type:
|
|
http://www.YOUR-DOMAIN.com/index.html
|
|
/index.html
|
|
http://www.YOUR-DOMAIN.com/blog/private-info.html
|
|
/blog/private-info.html
|
|
http://www.YOUR-DOMAIN.com/cgi-bin/
|
|
/cgi-bin/
|
What to Do With the Generated File
Once you have generated a code you need to copy and paste it into
a text file called "robot.txt" and then upload it into the root directory
of your server (usually "public_html").
Specifying Different Rules For Different User Agents
When you are specifying different rules for different bots, you need to
treat each one seperately. To clarify what is meant please look at the
following example:
|
In Robot.txt
|
|
User-agent: Slurp
Disallow: /images/
User-agent: fast
Disallow: /
etc...
|
Now fill in the form below to generate your custom robot.txt code.
[
Back to Website Tools and Tutorials ]
|
