SITECUBE /llms.txt Validator · AI Crawler Registry · Web-to-Token Observatory

AI Crawler robots.txt Rule Builder

Generate explicit User-agent rules for search citation crawlers versus bulk training dataset scrapers.

Crawler Directives

User-Agent Token Operator & Role Allow
OAI-SearchBot OpenAI ChatGPT Search Citations
PerplexityBot Perplexity Answer Citations
ChatGPT-User User-Triggered ChatGPT Browsing
Claude-SearchBot Anthropic Search Grounding
GPTBot OpenAI Model Training Crawler
ClaudeBot Anthropic Model Training
Google-Extended Google Gemini Training Control
Applebot-Extended Apple Intelligence Training
Meta-ExternalAgent Meta Llama Training Crawler
CCBot Common Crawl Training Dataset

Generated /robots.txt