User-agent: Googlebot-Image User-agent: Mediapartners-Google* User-agent: Yahoo-MMCrawler User-agent: psbot User-agent: IsraBot User-agent: yahoo-blogs/v3.9 # # My additions User-agent: LmCrawler User-agent: msnbot-MM Disallow:/ # Begin block Bad-Robots from robots.txt if they obey lol User-agent: asterias User-agent: Offline Explorer User-agent: Zealbot User-agent: Zeus User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow:/ # Begin Exclusion From Directories from robots.txt User-agent: * Disallow: /admin/ Disallow: /cgi-bin/ Disallow: /files/ Disallow: /htaccess/ Disallow: /images/ Disallow: /templates/ # Delay crawl of site - # some do some dont read this # whole integers in seconds per page Crawl-delay: 5