anthropic.com

B+ (81/100) Open to Search & Training

Home \ Anthropic

Raw HTML Tokens: ~43,602 tok
Clean Text Tokens: ~494 tok (1.1% signal)
/llms.txt Manifest: Not deployed
Edge / TTFB: Cloudflare · 408 ms
AI & /llms.txt Readiness Signal-to-Noise Ratio Token Savings via /llms.txt Network TTFB TLS & HTTP Security DNS & Edge Stack
51/100 1.1% (494 / 43,602 tok) 0% (No /llms.txt) 408 ms (84/100) 86/100 Cloudflare (Next.js)

Web-to-Token Efficiency & Agent Ingestion Cost

~4 bytes/token UTF-8 heuristic
Ingestion Source Payload Est. Tokens Cost / 1k Reads (Flash $0.35) Cost / 1k Reads (Pro $1.25)
Raw HTML DOM (GET /) 170.3 KB 43,602 $15.2607 $54.5025
Stripped Readable Text 1.9 KB 494 $0.1729 $0.6175
/llms.txt Manifest Not deployed — agents must parse 43,602 raw HTML tokens
HTML Signal-to-Noise Ratio: 1.1% readable text vs. 98.9% markup/script overhead. Token Optimization Guide →

/llms.txt Spec Linter & Agent Endpoint Probe

Status: Not Deployed · Spec Rules →
GET /llms.txt Missing — Generate spec-compliant /llms.txt for anthropic.com →
GET /llms-full.txt Not deployed
GET /.well-known/mcp.json Not advertised
JSON-LD Schema Entities None detected in initial HTML

AI Crawler /robots.txt Access Matrix

Open to Search & Training · Bot Directory →
User-Agent Token Purpose / Operator Effective Directive
OAI-SearchBot ChatGPT Search Citations Allow: /
PerplexityBot Perplexity Answer Citations Allow: /
ChatGPT-User Live User Agent Browsing Allow: /
Claude-SearchBot Anthropic Search Grounding Allow: /
GPTBot Foundation Model Training Allow: /
ClaudeBot Foundation Model Training Allow: /
Google-Extended Gemini Training Control Allow: /
Applebot-Extended Apple Intelligence Training Allow: /
Meta-ExternalAgent Llama Training & Indexing Allow: /
CCBot Open Dataset Training Corpus Allow: /

HTTP Response & Handshake Timing

GET https://www.anthropic.com/
DNS Lookup 14 ms HTML Document 170.3 KB
TCP Connect 10 ms Script Tags 46
TLS Handshake 173 ms Stylesheets 12
Time to First Byte (TTFB) 408 ms Images (with alt) 0 / 5
Total Transfer 424 ms HTTP/3 (Alt-Svc) Yes (h3)

Edge Network & Application Stack

Header & DOM Signatures
Edge / Reverse Proxy Cloudflare
Application Framework Next.js
Server Header cloudflare
Mail Provider (MX) Google Workspace
Telemetry Signatures Zero Client Trackers
TXT Verifications Google Search Console, Microsoft Entra, Stripe Verified, Atlassian, Slack

TLS Certificate & HTTP Security Policy

Port 443 X.509
Certificate Issuer Let's Encrypt (76 days remaining)
Strict-Transport-Security Present
Content-Security-Policy Present
X-Content-Type-Options Present
X-Frame-Options Present
Referrer-Policy Not set
Permissions-Policy Not set
SPF / DMARC Policy SPF: Yes · DMARC: Yes

Authoritative DNS & Document Head

A / AAAA / NS / Canonical
A (IPv4) 160.79.104.10
AAAA (IPv6) 2607:6bc0::10
NS (Nameservers) isla.ns.cloudflare.com, randy.ns.cloudflare.com
HTML Title (16c) Home \ Anthropic
/sitemap.xml Valid XML Sitemap

Hosted /llms.txt Mirror & Status Badge

Platform Setup Guide →
1-Line <head> Tag (for Shopify, Webflow, Framer, Wix, Squarespace & WordPress):
Sitecube status for anthropic.com
GitHub / Docs Markdown Badge:
HTML Badge:

Technical Synthesis & Agentic Ingestion Analysis: anthropic.com

Automated Engineering Assessment

1. HTML-to-Token Efficiency & LLM Context Window Cost

An unauthenticated HTTPS GET request to https://www.anthropic.com/ returns an initial HTML payload of 170.3 KB, corresponding to approximately 43,602 UTF-8 tokens when ingested raw by an AI coding agent, retrieval pipeline, or headless browser. After stripping inline <script>, <style>, <svg>, and DOM markup tags, the human-readable text footprint measures 494 tokens (1.9 KB)—yielding an HTML Signal-to-Noise Ratio of 1.1%. Because anthropic.com does not currently serve a machine-readable /llms.txt manifest at its root path, autonomous agents must either parse the full 43,602-token HTML payload ($54.5025 per 1,000 frontier reads) or run a DOM-stripping pass before grounding.

2. AI Crawler Governance & /robots.txt Posture

Based on live inspection of https://anthropic.com/robots.txt, anthropic.com maintains a Open to Search & Training posture. Specifically, AI search citation crawlers (OAI-SearchBot: Allowed, PerplexityBot: Allowed) and foundation model training crawlers (GPTBot: Allowed, ClaudeBot: Allowed, CCBot: Allowed) are governed explicitly by the host’s root robots policy.

3. Edge Delivery, TLS Handshake & Security Header Assessment

Traffic for anthropic.com resolves to 160.79.104.10 via Cloudflare running Next.js, achieving a measured Time to First Byte (TTFB) of 408 ms (DNS lookup: 14 ms, TCP connect: 10 ms, TLS handshake: 173 ms). The port 443 TLS certificate is issued by Let's Encrypt with 76 days remaining before expiration (valid through 2027-02-01). Outbound email authentication via SPF is present and DMARC (_dmarc.anthropic.com) is enforced.

Comparable Hosts in Web-to-Token & /llms.txt Registry

Full Index (32 hosts) →
Host Rating /llms.txt HTML Tokens Text Tokens Signal % Crawler Posture TTFB
sitecube.com A+ 98 2,099 tok 16,563 1,347 8.1% Open to Search & Training 96 ms
cloudflare.com A+ 96 4,294 tok 333,631 471 0.1% Open to Search & Training 294 ms
trendyflash.com A+ 95 3,256 tok 12,230 1,739 14.2% Open to Search & Training 85 ms
multifactor.com A+ 94 656 tok 98,275 673 0.7% Open to Search & Training 109 ms
tierones.io A+ 93 481 tok 3,744 693 18.5% Open to Search & Training 30 ms
aiimpacted.org A+ 93 1,355 tok 17,701 3,599 20.3% Open to Search & Training 119 ms
github.com A+ 92 7,165 tok 131,479 1,185 0.9% Open to Search & Training 26 ms
vercel.com A+ 92 1,385 tok 149,553 1,193 0.8% Open to Search & Training 66 ms
render.com A+ 92 13,304 tok 190,754 2,018 1.1% Open to Search & Training 306 ms
cloq.com A 91 1,169 tok 2,109 690 32.7% Open to Search & Training 46 ms
postman.com A 91 5,375 tok 118,892 11,040 9.3% Open to Search & Training 769 ms
neon.tech A 90 7,418 tok 110,734 1,629 1.5% Open to Search & Training 106 ms