/llms.txt, Crawler Policy & Web-to-Token Registry
Observed HTML token footprints, signal-to-noise ratios, /llms.txt manifest sizes, and AI crawler directives across 26 hosts.
Hosts: 26
/llms.txt Active: 15/26 (58%)
Mean TTFB: 249 ms
| # | Host | Rating | /llms.txt | Raw HTML Tokens | Clean Text Tokens | Signal % | AI Crawler Posture | Edge / Stack | TTFB |
|---|---|---|---|---|---|---|---|---|---|
| 1 | sitecube.com | A+ 98 | 1,600 tok | 11,832 | 1,132 | 9.6% | Open to Search & Training | Hostinger Edge CDN · Custom SSR / Static | 99 ms |
| 2 | cloudflare.com | A+ 96 | 4,294 tok | 333,631 | 471 | 0.1% | Open to Search & Training | Cloudflare · Astro | 291 ms |
| 3 | trendyflash.com | A+ 95 | 2,987 tok | 12,123 | 1,660 | 13.7% | Open to Search & Training | Hostinger Edge CDN · Next.js | 112 ms |
| 4 | multifactor.com | A+ 94 | 656 tok | 98,275 | 673 | 0.7% | Open to Search & Training | Cloudflare · Next.js | 97 ms |
| 5 | github.com | A+ 92 | 7,165 tok | 131,504 | 1,159 | 0.9% | Open to Search & Training | GitHub Pages · Custom SSR / Static | 35 ms |
| 6 | vercel.com | A+ 92 | 1,385 tok | 149,859 | 889 | 0.6% | Open to Search & Training | Vercel · Next.js | 53 ms |
| 7 | aiimpacted.org | A+ 92 | 971 tok | 17,598 | 3,568 | 20.3% | Open to Search & Training | Direct Origin · Custom SSR / Static | 55 ms |
| 8 | tierones.io | A+ 92 | 481 tok | 3,744 | 693 | 18.5% | Open to Search & Training | Direct Origin · Custom SSR / Static | 95 ms |
| 9 | cloq.com | A 91 | 1,169 tok | 2,109 | 690 | 32.7% | Open to Search & Training | Hostinger Edge CDN · Custom SSR / Static | 51 ms |
| 10 | supabase.com | A 89 | 879 tok | 337,513 | 1,905 | 0.6% | Open to Search & Training | Vercel · Next.js | 48 ms |
| 11 | dosefacts.org | A 89 | 10,936 tok | 29,008 | 7,291 | 25.1% | Open to Search & Training | Direct Origin · Custom SSR / Static | 335 ms |
| 12 | postman.com | A 89 | 5,375 tok | 118,892 | 11,040 | 9.3% | Open to Search & Training | Cloudflare · Next.js | 947 ms |
| 13 | linear.app | A 87 | 2,903 tok | 317,016 | 313 | 0.1% | Open to Search & Training | Cloudflare · Next.js | 452 ms |
| 14 | gitlab.com | A 85 | — | 56,566 | 1,903 | 3.4% | Open to Search & Training | Cloudflare · Nuxt | 399 ms |
| 15 | stripe.com | A 85 | 17,454 tok | 172,347 | 3,523 | 2% | Open to Search & Training | Direct Origin · Next.js | 503 ms |
| 16 | datadoghq.com | B+ 83 | 150,542 tok | 70,990 | 3,506 | 4.9% | Open to Search & Training | AWS CloudFront · Custom SSR / Static | 276 ms |
| 17 | huggingface.co | B+ 82 | — | 43,957 | 1,148 | 2.6% | Open to Search & Training | AWS CloudFront · Custom SSR / Static | 69 ms |
| 18 | anthropic.com | B+ 81 | — | 43,572 | 494 | 1.1% | Open to Search & Training | Cloudflare · Next.js | 352 ms |
| 19 | mozilla.org | B+ 79 | — | 11,971 | 656 | 5.5% | Open to Search & Training | Fastly · Custom SSR / Static | 259 ms |
| 20 | microsoft.com | B 77 | — | 44,551 | 859 | 1.9% | Open to Search & Training | Direct Origin · Custom SSR / Static | 120 ms |
| 21 | figma.com | B 77 | — | 466,946 | 8,864 | 1.9% | Search Allowed · Training Restricted | AWS CloudFront · Next.js | 1210 ms |
| 22 | openai.com | B 74 | — | 2,444 | 1 | 0% | Open to Search & Training | Cloudflare · Custom SSR / Static | 27 ms |
| 23 | tailwindcss.com | B 74 | — | 219,494 | 1,678 | 0.8% | Open to Search & Training | Cloudflare · Next.js | 51 ms |
| 24 | shopify.com | B 73 | — | 2,167 | 22 | 1% | Open to Search & Training | Cloudflare · Custom SSR / Static | 20 ms |
| 25 | astro.build | B 73 | — | 78,970 | 1,617 | 2% | Open to Search & Training | Cloudflare · Astro | 80 ms |
| 26 | google.com | B 70 | — | 791 | 258 | 32.6% | Open to Search & Training | Direct Origin · Custom SSR / Static | 426 ms |