User-agent: ShopWiki Disallow: / User-agent: Twiceler Disallow: / User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler www.cuill.com/robots.html Disallow: / User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Disallow: / User-agent: Twiceler www.cuill.com/twiceler/robot.html Disallow: / User-agent: Twiceler-0.9 http://www.cuill.com/twiceler/robot.html Disallow: / User-agent: TurnitinBot Disallow: / User-agent: Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/) Disallow: / User-agent: Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org) Disallow: / User-agent: dotbot Disallow: / User-agent: * Crawl-delay: 20 Disallow: /forgot-password Disallow: /join Disallow: /login Disallow: */inftrees$ Disallow: */inffast$ Disallow: */trees$ Disallow: */common$ Disallow: */crc32$ Disallow: */adler32$