SITECUBE /llms.txt Validator · AI Crawler Registry · Web-to-Token Observatory
Recent: cloudflare.com vercel.com stripe.com github.com openai.com trendyflash.com

/llms.txt Validator, AI Crawler Registry & Web-to-Token Observatory

Live HTTP/TLS socket diagnostics, HTML-to-LLM token footprint analysis, and crawler governance across production domains.

Hosts: 28 /llms.txt: 18/28 Mean TTFB: 313 ms

Agentic Web & Token Telemetry Summary

Live DOM, /llms.txt & Crawler Probes
Indexed Production Hosts 28 verified domains
/llms.txt Manifest Adoption 18 of 28 hosts (64%)
Mean HTML Signal-to-Noise 7% readable text vs. markup/script tokens
Flagship Research Report 2026 Web-to-Token Empirical Study (28 Hosts) →

Specification Linter & FinOps Utilities

Direct Tools
CMS /llms.txt Setup Guide 1-line <head> & 301 redirect guide for Shopify, Webflow, Framer, Wix & WP
/llms.txt Spec Builder Crawl any domain’s HTML head, nav links & sitemap to build an llms.txt file
AI robots.txt Builder Configure User-agent rules for OAI-SearchBot, GPTBot, ClaudeBot & CCBot
Cloud & Token Estimator Compare Cloudflare Workers, Vercel Pro, AWS Lambda & LLM API rates

Observed Host Registry (Web-to-Token Footprint · /llms.txt · Crawler Posture · TTFB)

Click any host to inspect full token economics & /llms.txt linter dossier
# Host Grade Score /llms.txt Raw HTML Tokens Clean Text Tokens Signal % AI Crawler Posture Edge / Stack TTFB
1 sitecube.com A+ 98/100 1,744 tok 12,387 1,227 9.9% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 66 ms
2 cloudflare.com A+ 96/100 4,294 tok 333,631 471 0.1% Open to Search & Training Cloudflare · Astro 251 ms
3 trendyflash.com A+ 95/100 3,120 tok 12,197 1,705 14% Open to Search & Training Hostinger Edge CDN · Next.js 36 ms
4 tierones.io A+ 93/100 481 tok 3,744 693 18.5% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 30 ms
5 dosefacts.org A+ 93/100 10,613 tok 29,181 7,309 25% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 107 ms
6 github.com A+ 92/100 7,165 tok 131,355 1,163 0.9% Open to Search & Training GitHub Pages · Custom SSR / Static 27 ms
7 vercel.com A+ 92/100 1,385 tok 149,553 1,193 0.8% Open to Search & Training Vercel · Next.js 57 ms
8 aiimpacted.org A+ 92/100 1,355 tok 17,717 3,605 20.3% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 173 ms
9 cloq.com A 91/100 1,169 tok 2,109 690 32.7% Open to Search & Training Hostinger Edge CDN · Custom SSR / Static 34 ms
10 multifactor.com A 91/100 656 tok 98,275 673 0.7% Open to Search & Training Cloudflare · Next.js 329 ms
11 supabase.com A 89/100 879 tok 337,513 1,905 0.6% Open to Search & Training Vercel · Next.js 55 ms
12 shopify.com A 89/100 3,161 tok 172,960 513 0.3% Open to Search & Training Cloudflare · Shopify 95 ms
13 postman.com A 89/100 5,375 tok 118,892 11,040 9.3% Open to Search & Training Cloudflare · Next.js 1254 ms
14 redis.io A 88/100 11,721 tok 427,364 11,796 2.8% Open to Search & Training Cloudflare · Next.js 130 ms
15 linear.app A 87/100 2,903 tok 317,016 313 0.1% Open to Search & Training Cloudflare · Next.js 452 ms
16 gitlab.com A 85/100 — 56,566 1,903 3.4% Open to Search & Training Cloudflare · Nuxt 402 ms
17 stripe.com A 85/100 17,454 tok 172,347 3,523 2% Open to Search & Training Direct Origin · Next.js 502 ms
18 huggingface.co B+ 82/100 — 43,595 1,164 2.7% Open to Search & Training AWS CloudFront · Custom SSR / Static 40 ms
19 mozilla.org B+ 81/100 — 11,971 656 5.5% Open to Search & Training Fastly · Custom SSR / Static 245 ms
20 anthropic.com B+ 81/100 — 43,594 494 1.1% Open to Search & Training Cloudflare · Next.js 358 ms
21 datadoghq.com B+ 81/100 150,542 tok 70,990 3,506 4.9% Open to Search & Training AWS CloudFront · Custom SSR / Static 481 ms
22 microsoft.com B 77/100 — 44,542 859 1.9% Open to Search & Training Direct Origin · Custom SSR / Static 107 ms
23 figma.com B 77/100 — 466,946 8,864 1.9% Search Allowed · Training Restricted Netlify Edge · Next.js 1024 ms
24 mongodb.com B 75/100 2,775 tok 319,238 2,450 0.8% Open to Search & Training AWS CloudFront · Next.js 1633 ms
25 openai.com B 74/100 — 2,444 1 0% Open to Search & Training Cloudflare · Custom SSR / Static 27 ms
26 astro.build B 73/100 — 78,970 1,617 2% Open to Search & Training Cloudflare · Astro 49 ms
27 tailwindcss.com B 71/100 — 219,490 1,678 0.8% Open to Search & Training Cloudflare · Next.js 375 ms
28 google.com B 70/100 — 791 258 32.6% Open to Search & Training Direct Origin · Custom SSR / Static 426 ms