http://en.wikipedia.org/wiki/Robots_exclusion_stan ... - jeeves[5] 40% -

http://www.searchtools.com/robots/robots-txt.html - jeeves[2] 33% -

http://www.robotstxt.org - jeeves[1] 25% -

Robots.txt Information Information on the robots.txt and how it effects your website. Also includes a free robots.txt generator http://www.robotstxt.ca - yahoo[4] 23% -

http://www.javascriptkit.com/howto/robots.shtml - jeeves[3] 20% -

Creating and Using a robots.txt File A robots.txt is a file placed on your server to tell the various search engine spiders not to crawl or index certain sections or pages of your site. ... http://www.outfront.net/tutorials_02/adv_tech/robo ... - yahoo[6] 20% -

http://www.sxw.org.uk/computing/robots/check.html - jeeves[4] 18% -

Manual:robots.txt - MediaWiki robots.txt files are part of the Robots Exclusion Standard. ... because some robots like Googlebot accept this wildcard extension to the robots.txt standard, which stops ... http://www.mediawiki.org/wiki/Manual:Robots.txt - yahoo[8] 18% -

Robots.txt Generator - McAnerin International Inc. robots.txt generator designed by an SEO for public use. Includes tutorial. ... The robots.txt is a very simple text file that is placed on your root directory. ... http://www.mcanerin.com/EN/search-engine/robots-tx ... - yahoo[10] 15% -

http://www.clickability.co.uk/robotstxt.html - jeeves[6] 13% -