Free AI Crawler Checker — Test GPTBot, OAI-SearchBot, PerplexityBot & ClaudeBot Access

Inspect any website’s live /robots.txt policy across 10 AI search citation bots and foundation model training crawlers.

Test Live AI Bot Permissions for Any Domain

10-Crawler Governance Matrix for cloudflare.com

Open to Search & Training ADA 88/100
User-Agent Token Crawler Category Operator & Traffic Impact Effective Policy
OAI-SearchBot AI Search / Citation ChatGPT Search Citations ALLOW (Crawls Root)
PerplexityBot AI Search / Citation Perplexity Answer Citations ALLOW (Crawls Root)
ChatGPT-User AI Search / Citation Live User Agent Browsing ALLOW (Crawls Root)
Claude-SearchBot AI Search / Citation Anthropic Search Grounding ALLOW (Crawls Root)
GPTBot Model Training Corpus Foundation Model Training ALLOW (Crawls Root)
ClaudeBot Model Training Corpus Foundation Model Training ALLOW (Crawls Root)
Google-Extended Model Training Corpus Gemini Training Control ALLOW (Crawls Root)
Applebot-Extended Model Training Corpus Apple Intelligence Training ALLOW (Crawls Root)
Meta-ExternalAgent Model Training Corpus Llama Training & Indexing ALLOW (Crawls Root)
CCBot Model Training Corpus Open Dataset Training Corpus ALLOW (Crawls Root)
Want to allow AI search traffic while blocking model training scrapers? Open 1-Click AI robots.txt Rule Builder →

Frequently Asked Questions: AI Crawler & GPTBot Access

Technical Reference

How do I check if my website blocks GPTBot, OAI-SearchBot, or PerplexityBot?

Enter your domain into Sitecube’s live AI Crawler Checker above. It fetches your production /robots.txt over HTTPS, parses User-agent and Allow/Disallow rules across all 10 major AI crawlers, and separates Search Citation Bots (which send clickable referral traffic) from Foundation Training Bots.

What is the difference between OAI-SearchBot and GPTBot?

OAI-SearchBot indexes webpages specifically to surface clickable source citations inside ChatGPT Search answers, whereas GPTBot crawls content to train OpenAI foundation models. Allowing OAI-SearchBot while disallowing GPTBot preserves AI referral traffic without opting into model training.

Does blocking Google-Extended hurt Google Search SEO rankings?

No. Google’s documentation confirms that Google-Extended only controls Gemini model training and Vertex AI grounding datasets—it has zero effect on Googlebot indexing or traditional Google Search rankings.