Robots.txt Generator
Create robots.txt files to control search engine crawling
Configuration
Rules
Rule #1
Generated robots.txt
Configure rules and generate robots.txt
Common Use Cases
Block Admin Areas
Prevent crawling of admin and private sections
Control Crawl Rate
Set delays to reduce server load from crawlers
Sitemap Reference
Tell crawlers where to find your XML sitemap
Bot-Specific Rules
Different rules for different search engines