User-agent: * # rules apply to ALL Web robots Disallow: /scripts/EducationalGame/ # do not crawl the educational games directory Disallow: /scripts/educationalGame/ # do not crawl the educational games directory Disallow: /cfide/ Disallow: /CFIDE/ Disallow: /cfdocs/ Disallow: /docs/kids/L&W/ Disallow: /docs/justforfun/ Crawl-delay: 120 # Number of seconds between each request to the same server. Request-rate: 1/10 # Maximum of one page per five seconds. Visit-time: 0500-0700 # Crawl the web site from 5AM to 7AM.