User-agent: * Disallow: /admin/ Disallow: /_rhnew/ Disallow: /_rhnew/admin/ # test of semicolon after disallow User-agent: inktomi Disallow; /_rhnew/ # test of empty user agent field User-agent: Disallow: / # updated 2002-03-22 (disallow rtestprob links) # updated 2002-06-25 (disallow info/slides links, info/robots/) # updated 2002-07-25 (disallow /searchtools/ which is an alias) # updated 2005-09-09 (rearranged as per Enrico's advice) # updated 2007-01-24 (added site directory to disallow list) # updated 2007-01-25 (removed rss disallow, file is gone) # updated 2008-06-24 (added wildcard tests lines 6-10) # updated 2008-07-02 (added allows for site subidrectory)