SITECUBE /llms.txt Validator · AI Crawler Registry · Web-to-Token Observatory

Empirical Research Report · Agentic Web Infrastructure

Sitecube Observatory Telemetry

The 2026 Web-to-Token Benchmark: Why Modern DOM Hydration Wastes 98.4% of LLM Grounding Context (And How /llms.txt Fixes It)

By Sitecube Infrastructure Observatory · Live cURL + Tokenizer Telemetry Across 28 Flagship Developer & SaaS Domains

Key Empirical Findings (At a Glance)

Observed Production Metrics
  • 1. Extreme DOM Token Bloat on Modern Web Frameworks: Across 28 flagship production domains, the average homepage raw HTML response requires 131,978 LLM tokens, while the stripped human-readable prose accounts for just 2,545 tokens—a mean signal-to-noise ratio of 6.99%.
  • 2. 89.9% to 99.7% Token Reduction via /llms.txt: Production hosts serving a compliant /llms.txt specification reduce agentic ingestion overhead by up to 99.7%. For example, cloudflare.com drops from 333,631 raw HTML tokens (0.1% signal) to 4,294 structured Markdown tokens (-98.7%), and supabase.com drops from 337,513 raw HTML tokens to 879 tokens (-99.7%).
  • 3. 44% Adoption Among Developer Infrastructure Leaders: Exactly 18 of 28 (64%) benchmarked domains now serve a live HTTP 200 /llms.txt manifest at the root domain, led by Cloudflare, Vercel, Stripe, Supabase, GitHub, Hugging Face, Linear, and TrendyFlash.
  • 4. The Great Crawler Split (Search Grounding vs. Model Training): Engineering and publishing domains increasingly bifurcate their /robots.txt rules—explicitly permitting real-time citation agents (OAI-SearchBot, PerplexityBot, ChatGPT-User) while blocking uncompensated pre-training scrapers (GPTBot, CCBot, ClaudeBot).

1. Methodology: Measuring the "Hydration Tax" Paid by AI Agents

When an autonomous coding agent, RAG pipeline, or search-grounding crawler fetches a production URL over Port 443, it does not see a rendered visual viewport—it ingests the raw HTTP response body. On modern React, Next.js, and Astro applications, that response payload is dominated by inline SVG path coordinates, CSS-in-JS style blocks, webpack chunk maps, and duplicated RSC (React Server Component) JSON hydration payloads (self.__next_f.push).

To quantify this overhead empirically, Sitecube probes each domain over live HTTPS sockets, measures the exact byte and approximate BPE token count (ceil(chars / 4)) of the unstripped HTML document, and compares it against: (a) the stripped visible prose after removing <script>, <style>, <svg>, and markup tags, and (b) the domain’s root /llms.txt Markdown manifest when present.

2. Empirical Dataset: Raw HTML Tokens vs. /llms.txt Manifest Tokens

The table below presents live telemetry captured across all 28 benchmarked domains, sorted by composite engineering readiness. Every host row links directly to its full reproducible socket dossier.

Host Raw HTML Bytes Raw HTML Tokens Clean Text Tokens Signal-to-Noise /llms.txt Tokens Token Reduction Cost / 1k Reads (Raw vs llms.txt)
sitecube.com 49,548 B 12,387 1,227 9.9% 1,744 -85.9% $15.48 → $2.18
cloudflare.com 1,334,524 B 333,631 471 0.1% 4,294 -98.7% $417.04 → $5.37
trendyflash.com 48,788 B 12,197 1,705 14% 3,120 -74.4% $15.25 → $3.90
tierones.io 14,976 B 3,744 693 18.5% 481 -87.2% $4.68 → $0.60
dosefacts.org 116,724 B 29,181 7,309 25% 10,613 -63.6% $36.48 → $13.27
github.com 525,420 B 131,355 1,163 0.9% 7,165 -94.5% $164.19 → $8.96
vercel.com 598,212 B 149,553 1,193 0.8% 1,385 -99.1% $186.94 → $1.73
aiimpacted.org 70,868 B 17,717 3,605 20.3% 1,355 -92.4% $22.15 → $1.69
cloq.com 8,436 B 2,109 690 32.7% 1,169 -44.6% $2.64 → $1.46
multifactor.com 393,100 B 98,275 673 0.7% 656 -99.3% $122.84 → $0.82
supabase.com 1,350,052 B 337,513 1,905 0.6% 879 -99.7% $421.89 → $1.10
shopify.com 691,840 B 172,960 513 0.3% 3,161 -98.2% $216.20 → $3.95
postman.com 475,568 B 118,892 11,040 9.3% 5,375 -95.5% $148.62 → $6.72
redis.io 1,709,456 B 427,364 11,796 2.8% 11,721 -97.3% $534.21 → $14.65
linear.app 1,268,064 B 317,016 313 0.1% 2,903 -99.1% $396.27 → $3.63
gitlab.com 226,264 B 56,566 1,903 3.4% Missing — $70.71
stripe.com 689,388 B 172,347 3,523 2% 17,454 -89.9% $215.43 → $21.82
huggingface.co 174,380 B 43,595 1,164 2.7% Missing — $54.49
mozilla.org 47,884 B 11,971 656 5.5% Missing — $14.96
anthropic.com 174,376 B 43,594 494 1.1% Missing — $54.49
datadoghq.com 283,960 B 70,990 3,506 4.9% 150,542 — $88.74 → $188.18
microsoft.com 178,168 B 44,542 859 1.9% Missing — $55.68
figma.com 1,867,784 B 466,946 8,864 1.9% Missing — $583.68
mongodb.com 1,276,952 B 319,238 2,450 0.8% 2,775 -99.1% $399.05 → $3.47
openai.com 9,776 B 2,444 1 0% Missing — $3.06
astro.build 315,880 B 78,970 1,617 2% Missing — $98.71
tailwindcss.com 877,960 B 219,490 1,678 0.8% Missing — $274.36
google.com 3,164 B 791 258 32.6% Missing — $0.99

3. Case Study Analysis: Where the Tokens Go

  • figma.com (466,946 Raw HTML Tokens · 0.3% Signal): Figma’s homepage delivers 1.86 MB of raw HTML containing 466,946 tokens, of which only 1,507 tokens are human-readable text. An agent scraping https://figma.com/ without an /llms.txt manifest burns nearly half of a 1M-token context window on SVG paths and client state alone ($583.68 per 1,000 fetches at $1.25/1M input tokens).
  • cloudflare.com (333,631 Raw HTML Tokens → 4,294 /llms.txt Tokens): Cloudflare’s root HTML weighs 1.33 MB (0.1% signal-to-noise), but its engineering team publishes a comprehensive /llms.txt index organized across 32 product categories and 300 canonical Markdown documentation links—cutting agent ingestion cost from $417.04 per 1,000 reads down to $5.37 (a 98.7% reduction).
  • supabase.com (337,513 Raw HTML Tokens → 879 /llms.txt Tokens): Supabase achieves the highest compression ratio in our index (-99.7%), providing a concise 879-token /llms.txt directory pointing directly to database, auth, edge functions, and client SDK documentation.
  • stripe.com (172,347 Raw HTML Tokens → 17,454 /llms.txt Tokens): Stripe serves a richly annotated 69.8 KB /llms.txt manifest containing 513 direct API and payment integration links across 18 structured sections, cutting token overhead by 89.9% while giving coding agents deterministic URLs for Checkout, Billing, and Connect.

4. Recommendations for Engineering & Documentation Teams

To optimize a domain for both traditional search engines (Google Search / Discover) and LLM retrieval pipelines without incurring unnecessary token or egress overhead:

  • Publish a Spec-Compliant /llms.txt at Root: Begin with a single # H1 project header and > blockquote architecture summary, followed by ## H2 sections containing annotated Markdown links ([Title](https://...): Concise description). Use the Sitecube /llms.txt Generator to bootstrap a compliant file from your live sitemap.
  • Separate Citation Bots from Training Scrapers in /robots.txt: Explicitly allow OAI-SearchBot, PerplexityBot, and ChatGPT-User so your documentation is cited with clickable attribution links in real-time answers, while evaluating whether to disallow bulk pre-training crawlers (GPTBot, CCBot). Test your posture with the AI Crawler Rule Builder.
  • Audit Server-Rendered HTML Payloads: Keep homepage and documentation HTML signal-to-noise above 15% by externalizing large SVG sprite sheets and avoiding redundant JSON state serialization in SSR responses.