If you'd like to place this robot.txt generator on your website, copy and paste the HTML code below into you webpage. Please note that by using this tool you are agreeing to keep the link intact .
<form action = "http://www.SiteToolCenter.com/website-tools-and-tutorials/robot-txt.php" method = "post"> <table> <tr> <td align = "left" valign = "top"> Allowed User Agents: </td> <td align = "left" valign = "top"> <select name = "userAgent"> <option value="*" >All Agents</option> <option value="ia_archiver" >Alexa Agent</option> <option value="Fast">AllTheWeb Agent</option> <option value="Scooter" >AltaVista Agent</option> <option value="ArchitextSpider" >AOL Netfind/Excite Agent</option> <option value="Googlebot" >Google Agent</option> <option value="Googlebot-Images">Google Images</option> <option value="Slurp" >HotBot Agent</option> <option value="InfoSeek Sidewinder" >InfoSeek Agent</option> <option value="Lycos_Spider_(T-Rex)" >Lycos Agent</option> <option value="MSNBot" >MSN Search Crawler</option> <option value="Gulliver" >Northern Lights Agent</option> <option value="WebCrawler/3.0" >WebCrawler Agent</option> </select> <br> <br> </td> </tr> <tr> <td align = "left" valign = "top"> Exclude: </td> <td align = "left" valign = "top"> <input type = "radio" name = "exclude" value = "nothing" checked> Nothing <input type = "radio" name = "exclude" value = "all"> Everything <input type = "radio" name = "exclude" value = "urls"> The URL Below:<br> <textarea name = "urls" cols = "40" rows = "5"> ** Examples ** /index.html /blog/private-info.html /cgi-bin/</textarea> </td> </tr> <tr> <td align = "left" colspan = "2"> <br> <input type = "submit" value = "Generate Robot.txt Code"> </td> </tr> <tr> <td colspan = "2"> <a href="http://www.SiteToolCenter.com">More Website Tools</a> </td> </tr> </table> </form>
Forums | Search | Donate | Links | Contact | Templates | Privacy Policy Copyright © 2010 SiteToolCenter.com
Best Viewed With Mozilla FireFox
