AI Crawler User-agent List (2026): Search Citation Bots vs. Model Training Scrapers

Complete engineering reference for OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, and CCBot.

Search Open: 31/31 Train Restricted: 1/31

1. The Critical Distinction: Search Grounding vs. Pre-Training Crawlers

RFC 9309 Governance

One of the most costly mistakes technical teams make in /robots.txt is applying a blanket block across every AI-related User-agent without distinguishing between real-time search citation bots and bulk pre-training corpus scrapers.

Major AI providers now separate their crawler fleets into two distinct operational roles:

  • Search Grounding & User-Action Agents (e.g., OAI-SearchBot, PerplexityBot, Claude-SearchBot, ChatGPT-User): Fetch pages to answer live user queries with clickable source citations and referral links. Blocking these removes your domain from AI search answers.
  • Foundation Model Training Crawlers (e.g., GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot): Scrape public web text into multi-trillion-token pre-training and post-training datasets without per-query attribution. Blocking these opts out of model weight training while keeping search visibility intact.

2. Complete 10-Bot AI Crawler Technical Directory

Exact robots.txt Tokens
User-agent Token Operator Category Impact if Disallowed IP / rDNS Verification
OAI-SearchBot OpenAI Search Citations Removes site from ChatGPT Search link citations Published JSON IP ranges (openai.com/searchbot.json)
ChatGPT-User OpenAI Live User Fetch Blocks live URL reading when a user pastes your link Published JSON IP ranges (openai.com/chatgpt-user.json)
PerplexityBot Perplexity AI Search Citations Excludes domain from Perplexity answer citations Published JSON CIDR list (perplexity.ai/perplexitybot.json)
Claude-SearchBot Anthropic Search Citations Blocks real-time Claude web search grounding Anthropic crawler IP ranges
GPTBot OpenAI Model Training Opts out of OpenAI generative model training sets Published JSON IP ranges (openai.com/gptbot.json)
ClaudeBot Anthropic Model Training Opts out of Anthropic Claude training datasets Anthropic published CIDR blocks
Google-Extended Google Model Training Opts out of Gemini training; zero impact on Google Search Shares Google ASN / reverse DNS (*.googlebot.com)
Applebot-Extended Apple Model Training Opts out of Apple Intelligence training; Siri/Spotlight stays active Reverse DNS *.applebot.apple.com
Meta-ExternalAgent Meta Model Training Opts out of Llama model training & Meta AI corpus Meta crawler ASN (AS32934)
CCBot Common Crawl Open Dataset Excludes pages from open Common Crawl WARC archives AWS Common Crawl crawler fleet

3. Recommended Balanced /robots.txt Policy

Need custom per-bot toggles? Open Interactive Builder →

4. Crawler Governance & Security FAQ

Googlebot & WAF Notes

Does blocking Google-Extended hurt Google Search or AI Overviews?

No. Google explicitly documents that Google-Extended is a standalone product token used solely to control whether content helps train Gemini foundation models and Vertex AI generative APIs. Traditional Google Search rankings and Search AI Overviews are governed by Googlebot. Never disallow Googlebot if you want organic Google Search traffic.

Why do bots still hit my origin after updating /robots.txt?

Compliant crawlers cache /robots.txt for up to 24 hours (per RFC 9309) before applying rule changes. Conversely, non-compliant scrapers spoof standard browser or bot User-Agent strings. To block spoofed bots without blocking real search indexers, enforce ASN and reverse-DNS verification at your CDN/WAF edge (for example, Cloudflare Verified Bots) rather than matching substring User-Agent headers alone.

How can I audit my domain’s live crawler rules?

Enter your domain in the Sitecube Host Inspector at the top of this page. Sitecube fetches your live /robots.txt over Port 443 and evaluates effective wildcard (*) plus bot-specific precedence across all 10 AI crawlers.