SITECUBE /llms.txt Validator · AI Crawler Registry · Web-to-Token Observatory

anthropic.com

B+ (81/100) Open to Search & Training

Home \ Anthropic

Raw HTML Tokens: ~43,594 tok
Clean Text Tokens: ~494 tok (1.1% signal)
/llms.txt Manifest: Not deployed
Edge / TTFB: Cloudflare · 358 ms
AI & /llms.txt Readiness Signal-to-Noise Ratio Token Savings via /llms.txt Network TTFB TLS & HTTP Security DNS & Edge Stack
51/100 1.1% (494 / 43,594 tok) 0% (No /llms.txt) 358 ms (84/100) 86/100 Cloudflare (Next.js)

Web-to-Token Efficiency & Agent Ingestion Cost

~4 bytes/token UTF-8 heuristic
Ingestion Source Payload Est. Tokens Cost / 1k Reads (Flash $0.35) Cost / 1k Reads (Pro $1.25)
Raw HTML DOM (GET /) 170.3 KB 43,594 $15.2579 $54.4925
Stripped Readable Text 1.9 KB 494 $0.1729 $0.6175
/llms.txt Manifest Not deployed — agents must parse 43,594 raw HTML tokens
HTML Signal-to-Noise Ratio: 1.1% readable text vs. 98.9% markup/script overhead.

/llms.txt Spec Linter & Agent Endpoint Probe

Status: Not Deployed
GET /llms.txt Missing — Generate spec-compliant /llms.txt for anthropic.com →
GET /llms-full.txt Not deployed
GET /.well-known/mcp.json Not advertised
JSON-LD Schema Entities None detected in initial HTML

AI Crawler /robots.txt Access Matrix

Open to Search & Training
User-Agent Token Purpose / Operator Effective Directive
OAI-SearchBot ChatGPT Search Citations Allow: /
PerplexityBot Perplexity Answer Citations Allow: /
ChatGPT-User Live User Agent Browsing Allow: /
Claude-SearchBot Anthropic Search Grounding Allow: /
GPTBot Foundation Model Training Allow: /
ClaudeBot Foundation Model Training Allow: /
Google-Extended Gemini Training Control Allow: /
Applebot-Extended Apple Intelligence Training Allow: /
Meta-ExternalAgent Llama Training & Indexing Allow: /
CCBot Open Dataset Training Corpus Allow: /

HTTP Response & Handshake Timing

GET https://www.anthropic.com/
DNS Lookup 14 ms HTML Document 170.3 KB
TCP Connect 8 ms Script Tags 48
TLS Handshake 151 ms Stylesheets 12
Time to First Byte (TTFB) 358 ms Images (with alt) 0 / 5
Total Transfer 364 ms HTTP/3 (Alt-Svc) Yes (h3)

Edge Network & Application Stack

Header & DOM Signatures
Edge / Reverse Proxy Cloudflare
Application Framework Next.js
Server Header cloudflare
Mail Provider (MX) Google Workspace
Telemetry Signatures Zero Client Trackers
TXT Verifications Google Search Console, Stripe Verified, Microsoft Entra, Atlassian, Slack

TLS Certificate & HTTP Security Policy

Port 443 X.509
Certificate Issuer Let's Encrypt (78 days remaining)
Strict-Transport-Security Present
Content-Security-Policy Present
X-Content-Type-Options Present
X-Frame-Options Present
Referrer-Policy Not set
Permissions-Policy Not set
SPF / DMARC Policy SPF: Yes · DMARC: Yes

Authoritative DNS & Document Head

A / AAAA / NS / Canonical
A (IPv4) 160.79.104.10
AAAA (IPv6) 2607:6bc0::10
NS (Nameservers) isla.ns.cloudflare.com, randy.ns.cloudflare.com
HTML Title (16c) Home \ Anthropic
/sitemap.xml Valid XML Sitemap

Hosted /llms.txt Mirror & Status Badge

Platform Setup Guide →
1-Line <head> Tag (for Shopify, Webflow, Framer, Wix, Squarespace & WordPress):
Sitecube status for anthropic.com
GitHub / Docs Markdown Badge:
HTML Badge:

Technical Synthesis & Agentic Ingestion Analysis: anthropic.com

Automated Engineering Assessment

1. HTML-to-Token Efficiency & LLM Context Window Cost

An unauthenticated HTTPS GET request to https://www.anthropic.com/ returns an initial HTML payload of 170.3 KB, corresponding to approximately 43,594 UTF-8 tokens when ingested raw by an AI coding agent, retrieval pipeline, or headless browser. After stripping inline <script>, <style>, <svg>, and DOM markup tags, the human-readable text footprint measures 494 tokens (1.9 KB)—yielding an HTML Signal-to-Noise Ratio of 1.1%. Because anthropic.com does not currently serve a machine-readable /llms.txt manifest at its root path, autonomous agents must either parse the full 43,594-token HTML payload ($54.4925 per 1,000 frontier reads) or run a DOM-stripping pass before grounding.

2. AI Crawler Governance & /robots.txt Posture

Based on live inspection of https://anthropic.com/robots.txt, anthropic.com maintains a Open to Search & Training posture. Specifically, AI search citation crawlers (OAI-SearchBot: Allowed, PerplexityBot: Allowed) and foundation model training crawlers (GPTBot: Allowed, ClaudeBot: Allowed, CCBot: Allowed) are governed explicitly by the host’s root robots policy.

3. Edge Delivery, TLS Handshake & Security Header Assessment

Traffic for anthropic.com resolves to 160.79.104.10 via Cloudflare running Next.js, achieving a measured Time to First Byte (TTFB) of 358 ms (DNS lookup: 14 ms, TCP connect: 8 ms, TLS handshake: 151 ms). The port 443 TLS certificate is issued by Let's Encrypt with 78 days remaining before expiration (valid through 2027-02-01). Outbound email authentication via SPF is present and DMARC (_dmarc.anthropic.com) is enforced.

Comparable Hosts in Web-to-Token & /llms.txt Registry

Full Index (28 hosts) →
Host Rating /llms.txt HTML Tokens Text Tokens Signal % Crawler Posture TTFB
sitecube.com A+ 98 1,744 tok 12,387 1,227 9.9% Open to Search & Training 66 ms
cloudflare.com A+ 96 4,294 tok 333,631 471 0.1% Open to Search & Training 251 ms
trendyflash.com A+ 95 3,120 tok 12,197 1,705 14% Open to Search & Training 36 ms
tierones.io A+ 93 481 tok 3,744 693 18.5% Open to Search & Training 30 ms
dosefacts.org A+ 93 10,613 tok 29,181 7,309 25% Open to Search & Training 107 ms
github.com A+ 92 7,165 tok 131,355 1,163 0.9% Open to Search & Training 27 ms
vercel.com A+ 92 1,385 tok 149,553 1,193 0.8% Open to Search & Training 57 ms
aiimpacted.org A+ 92 1,355 tok 17,717 3,605 20.3% Open to Search & Training 173 ms
cloq.com A 91 1,169 tok 2,109 690 32.7% Open to Search & Training 34 ms
multifactor.com A 91 656 tok 98,275 673 0.7% Open to Search & Training 329 ms
supabase.com A 89 879 tok 337,513 1,905 0.6% Open to Search & Training 55 ms
shopify.com A 89 3,161 tok 172,960 513 0.3% Open to Search & Training 95 ms