Agentic Web Specifications, /llms.txt & Crawler Governance Guides
Technical reference documentation grounded in live HTTP/TLS socket and tokenizer telemetry across 31 production domains.
1. What Is /llms.txt? Specification, Syntax & Examples
Core Spec
Complete Markdown grammar for /llms.txt and /llms-full.txt, how it differs from /robots.txt and /sitemap.xml, common specification lint errors, and live production breakdowns across 21 verified adopters.
2. How to Add /llms.txt to Shopify, WordPress, Webflow & Framer
9 CMS Platforms
Step-by-step instructions to deploy /llms.txt in 15 seconds—even on hosted platforms with locked root directories—using standard <link rel="alternate" type="text/markdown"> autodiscovery or 301 redirects.
3. AI Crawler User-agent Directory & robots.txt Rules
10 Tracked Bots
Technical reference comparing real-time search citation crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) against foundation model training scrapers (GPTBot, ClaudeBot, Google-Extended, CCBot), including ASN/IP verification.
4. Reducing HTML-to-LLM Token Bloat & SSR Hydration Overhead
Token FinOps
Why modern React/Next.js RSC hydration payloads waste up to 99% of LLM context tokens, how to measure DOM Signal-to-Noise ratio, and how to serve clean Markdown via HTTP Accept: text/markdown content negotiation.
5. /.well-known/mcp.json & Model Context Protocol Discovery
Agentic Endpoints
How domains advertise structured read/query endpoints to autonomous coding and research agents via /.well-known/mcp.json, with CORS, caching, security headers, and live production schema examples (3 active in index).
6. The 2026 Web-to-Token Empirical Benchmark Study
Original Dataset
Reproducible telemetry across 31 flagship engineering and SaaS domains measuring raw HTML byte payloads, BPE token counts, stripped prose ratios, and inference cost savings from /llms.txt.
Interactive Verification & Specification Builders
Zero-Signup Engineering Utilities| Live Domain & /llms.txt Inspector | Run an instant HTTPS socket, tokenizer, /llms.txt syntax lint, and 10-bot robots.txt audit on any host |
| /llms.txt Specification Generator | Crawl any live website’s document metadata, canonical navigation, and XML sitemap into compliant Markdown |
| AI Crawler robots.txt Rule Builder | Generate deterministic Allow / Disallow directives separating AI citation search from bulk training |
| Cloud Egress & LLM Token Cost Estimator | Compare monthly serverless compute, bandwidth egress, and frontier vs. flash LLM API inference rates |