# Sitecube.com — /llms.txt Validator, AI Crawler Registry & Web-to-Token Observatory

> Sitecube.com is a technical observatory and specification linter that measures any domain’s HTML-to-LLM token footprint (`rawHtmlTokens` vs. `cleanTextTokens`), signal-to-noise ratio, `/llms.txt` compliance, `/.well-known/mcp.json` discovery, AI crawler `robots.txt` directives across 10 bots (`OAI-SearchBot`, `PerplexityBot`, `ChatGPT-User`, `Claude-SearchBot`, `GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `Meta-ExternalAgent`, `CCBot`), TLS security headers, and Port 443 TTFB latency.

## Primary Research & Benchmarks
- [The 2026 Web-to-Token Benchmark Study](https://sitecube.com/research/llms-txt-token-efficiency-2026): Empirical study across indexed production domains quantifying DOM hydration token waste vs. `/llms.txt` token reduction (up to -99.7%).
- [Global Host Registry & Token Leaderboard](https://sitecube.com/leaderboard): Comparative index of raw HTML tokens, clean text tokens, signal-to-noise ratios, and TTFB timings.
- [Methodology & Scoring Specification](https://sitecube.com/about): How Sitecube computes BPE token footprints, `/llms.txt` lint scores, and security posture.

## Interactive Engineering & FinOps Utilities
- [Live Host & Token Inspector](https://sitecube.com/): Real-time socket, DOM token, and `/llms.txt` audit for any domain.
- [/llms.txt Specification Generator](https://sitecube.com/tools/llms-txt-generator): Crawl any live host’s metadata, canonical links, and XML sitemap to build a compliant `/llms.txt` file.
- [AI Crawler robots.txt Rule Builder](https://sitecube.com/tools/ai-robots-generator): Configure explicit `User-agent` rules separating AI search citation bots from model training scrapers.
- [Cloud Compute, Egress & LLM Token Rate Estimator](https://sitecube.com/tools/cloud-cost-calculator): Compare monthly unit costs across Cloudflare Workers, Vercel Pro, AWS Lambda, Shared Cloud, and LLM API tiers.

## Verified Host Dossiers (26 Production Domains)
- [sitecube.com — Score 98/100 (A+)](https://sitecube.com/site/sitecube.com): Raw HTML 11,832 tok (9.6% signal) | llms.txt: 1,600 tok | Posture: Open to Search & Training | TTFB: 99ms
- [cloudflare.com — Score 96/100 (A+)](https://sitecube.com/site/cloudflare.com): Raw HTML 333,631 tok (0.1% signal) | llms.txt: 4,294 tok | Posture: Open to Search & Training | TTFB: 291ms
- [trendyflash.com — Score 95/100 (A+)](https://sitecube.com/site/trendyflash.com): Raw HTML 12,123 tok (13.7% signal) | llms.txt: 2,987 tok | Posture: Open to Search & Training | TTFB: 112ms
- [multifactor.com — Score 94/100 (A+)](https://sitecube.com/site/multifactor.com): Raw HTML 98,275 tok (0.7% signal) | llms.txt: 656 tok | Posture: Open to Search & Training | TTFB: 97ms
- [github.com — Score 92/100 (A+)](https://sitecube.com/site/github.com): Raw HTML 131,504 tok (0.9% signal) | llms.txt: 7,165 tok | Posture: Open to Search & Training | TTFB: 35ms
- [vercel.com — Score 92/100 (A+)](https://sitecube.com/site/vercel.com): Raw HTML 149,859 tok (0.6% signal) | llms.txt: 1,385 tok | Posture: Open to Search & Training | TTFB: 53ms
- [aiimpacted.org — Score 92/100 (A+)](https://sitecube.com/site/aiimpacted.org): Raw HTML 17,598 tok (20.3% signal) | llms.txt: 971 tok | Posture: Open to Search & Training | TTFB: 55ms
- [tierones.io — Score 92/100 (A+)](https://sitecube.com/site/tierones.io): Raw HTML 3,744 tok (18.5% signal) | llms.txt: 481 tok | Posture: Open to Search & Training | TTFB: 95ms
- [cloq.com — Score 91/100 (A)](https://sitecube.com/site/cloq.com): Raw HTML 2,109 tok (32.7% signal) | llms.txt: 1,169 tok | Posture: Open to Search & Training | TTFB: 51ms
- [supabase.com — Score 89/100 (A)](https://sitecube.com/site/supabase.com): Raw HTML 337,513 tok (0.6% signal) | llms.txt: 879 tok | Posture: Open to Search & Training | TTFB: 48ms
- [dosefacts.org — Score 89/100 (A)](https://sitecube.com/site/dosefacts.org): Raw HTML 29,008 tok (25.1% signal) | llms.txt: 10,936 tok | Posture: Open to Search & Training | TTFB: 335ms
- [postman.com — Score 89/100 (A)](https://sitecube.com/site/postman.com): Raw HTML 118,892 tok (9.3% signal) | llms.txt: 5,375 tok | Posture: Open to Search & Training | TTFB: 947ms
- [linear.app — Score 87/100 (A)](https://sitecube.com/site/linear.app): Raw HTML 317,016 tok (0.1% signal) | llms.txt: 2,903 tok | Posture: Open to Search & Training | TTFB: 452ms
- [gitlab.com — Score 85/100 (A)](https://sitecube.com/site/gitlab.com): Raw HTML 56,566 tok (3.4% signal) | No llms.txt | Posture: Open to Search & Training | TTFB: 399ms
- [stripe.com — Score 85/100 (A)](https://sitecube.com/site/stripe.com): Raw HTML 172,347 tok (2% signal) | llms.txt: 17,454 tok | Posture: Open to Search & Training | TTFB: 503ms
- [datadoghq.com — Score 83/100 (B+)](https://sitecube.com/site/datadoghq.com): Raw HTML 70,990 tok (4.9% signal) | llms.txt: 150,542 tok | Posture: Open to Search & Training | TTFB: 276ms
- [huggingface.co — Score 82/100 (B+)](https://sitecube.com/site/huggingface.co): Raw HTML 43,957 tok (2.6% signal) | No llms.txt | Posture: Open to Search & Training | TTFB: 69ms
- [anthropic.com — Score 81/100 (B+)](https://sitecube.com/site/anthropic.com): Raw HTML 43,572 tok (1.1% signal) | No llms.txt | Posture: Open to Search & Training | TTFB: 352ms
- [mozilla.org — Score 79/100 (B+)](https://sitecube.com/site/mozilla.org): Raw HTML 11,971 tok (5.5% signal) | No llms.txt | Posture: Open to Search & Training | TTFB: 259ms
- [microsoft.com — Score 77/100 (B)](https://sitecube.com/site/microsoft.com): Raw HTML 44,551 tok (1.9% signal) | No llms.txt | Posture: Open to Search & Training | TTFB: 120ms
- [figma.com — Score 77/100 (B)](https://sitecube.com/site/figma.com): Raw HTML 466,946 tok (1.9% signal) | No llms.txt | Posture: Search Allowed · Training Restricted | TTFB: 1210ms
- [openai.com — Score 74/100 (B)](https://sitecube.com/site/openai.com): Raw HTML 2,444 tok (0% signal) | No llms.txt | Posture: Open to Search & Training | TTFB: 27ms
- [tailwindcss.com — Score 74/100 (B)](https://sitecube.com/site/tailwindcss.com): Raw HTML 219,494 tok (0.8% signal) | No llms.txt | Posture: Open to Search & Training | TTFB: 51ms
- [shopify.com — Score 73/100 (B)](https://sitecube.com/site/shopify.com): Raw HTML 2,167 tok (1% signal) | No llms.txt | Posture: Open to Search & Training | TTFB: 20ms
- [astro.build — Score 73/100 (B)](https://sitecube.com/site/astro.build): Raw HTML 78,970 tok (2% signal) | No llms.txt | Posture: Open to Search & Training | TTFB: 80ms
- [google.com — Score 70/100 (B)](https://sitecube.com/site/google.com): Raw HTML 791 tok (32.6% signal) | No llms.txt | Posture: Open to Search & Training | TTFB: 426ms