AI Crawler robots.txt Rule Builder
Generate explicit User-agent rules for search citation crawlers versus bulk training dataset scrapers.
Crawler Directives
| User-Agent Token | Operator & Role | Allow |
|---|---|---|
| OAI-SearchBot | OpenAI ChatGPT Search Citations | |
| PerplexityBot | Perplexity Answer Citations | |
| ChatGPT-User | User-Triggered ChatGPT Browsing | |
| Claude-SearchBot | Anthropic Search Grounding | |
| GPTBot | OpenAI Model Training Crawler | |
| ClaudeBot | Anthropic Model Training | |
| Google-Extended | Google Gemini Training Control | |
| Applebot-Extended | Apple Intelligence Training | |
| Meta-ExternalAgent | Meta Llama Training Crawler | |
| CCBot | Common Crawl Training Dataset |