# http://en.wikipedia.org/wiki/Robots.txt User-agent: * Disallow: Crawl-delay: 5 # Allow 5 seconds between each request Request-rate: 1/5 # maximum rate is one page every 5 seconds Disallow: /bin/ Sitemap: http://www.voola.com/sitemap.aspx