Free AI Crawler Checker — Test GPTBot, OAI-SearchBot, PerplexityBot & ClaudeBot Access
Inspect any website’s live /robots.txt policy across 10 AI search citation bots and foundation model training crawlers.
Test Live AI Bot Permissions for Any Domain
10-Crawler Governance Matrix for cloudflare.com
| User-Agent Token | Crawler Category | Operator & Traffic Impact | Effective Policy |
|---|---|---|---|
| OAI-SearchBot | AI Search / Citation | ChatGPT Search Citations | ALLOW (Crawls Root) |
| PerplexityBot | AI Search / Citation | Perplexity Answer Citations | ALLOW (Crawls Root) |
| ChatGPT-User | AI Search / Citation | Live User Agent Browsing | ALLOW (Crawls Root) |
| Claude-SearchBot | AI Search / Citation | Anthropic Search Grounding | ALLOW (Crawls Root) |
| GPTBot | Model Training Corpus | Foundation Model Training | ALLOW (Crawls Root) |
| ClaudeBot | Model Training Corpus | Foundation Model Training | ALLOW (Crawls Root) |
| Google-Extended | Model Training Corpus | Gemini Training Control | ALLOW (Crawls Root) |
| Applebot-Extended | Model Training Corpus | Apple Intelligence Training | ALLOW (Crawls Root) |
| Meta-ExternalAgent | Model Training Corpus | Llama Training & Indexing | ALLOW (Crawls Root) |
| CCBot | Model Training Corpus | Open Dataset Training Corpus | ALLOW (Crawls Root) |
Frequently Asked Questions: AI Crawler & GPTBot Access
Technical ReferenceHow do I check if my website blocks GPTBot, OAI-SearchBot, or PerplexityBot?
Enter your domain into Sitecube’s live AI Crawler Checker above. It fetches your production /robots.txt over HTTPS, parses User-agent and Allow/Disallow rules across all 10 major AI crawlers, and separates Search Citation Bots (which send clickable referral traffic) from Foundation Training Bots.
What is the difference between OAI-SearchBot and GPTBot?
OAI-SearchBot indexes webpages specifically to surface clickable source citations inside ChatGPT Search answers, whereas GPTBot crawls content to train OpenAI foundation models. Allowing OAI-SearchBot while disallowing GPTBot preserves AI referral traffic without opting into model training.
Does blocking Google-Extended hurt Google Search SEO rankings?
No. Google’s documentation confirms that Google-Extended only controls Gemini model training and Vertex AI grounding datasets—it has zero effect on Googlebot indexing or traditional Google Search rankings.