Agentic Web Specifications, /llms.txt & Crawler Governance Guides

Technical reference documentation grounded in live HTTP/TLS socket and tokenizer telemetry across 31 production domains.

Guides: 5 Live Tools: 3 Verified Hosts: 31

1. What Is /llms.txt? Specification, Syntax & Examples

Core Spec

Complete Markdown grammar for /llms.txt and /llms-full.txt, how it differs from /robots.txt and /sitemap.xml, common specification lint errors, and live production breakdowns across 21 verified adopters.

Read /llms.txt Specification Guide →

2. How to Add /llms.txt to Shopify, WordPress, Webflow & Framer

9 CMS Platforms

Step-by-step instructions to deploy /llms.txt in 15 seconds—even on hosted platforms with locked root directories—using standard <link rel="alternate" type="text/markdown"> autodiscovery or 301 redirects.

Open CMS Platform Setup Guide →

3. AI Crawler User-agent Directory & robots.txt Rules

10 Tracked Bots

Technical reference comparing real-time search citation crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) against foundation model training scrapers (GPTBot, ClaudeBot, Google-Extended, CCBot), including ASN/IP verification.

Explore AI Crawler Directory →

4. Reducing HTML-to-LLM Token Bloat & SSR Hydration Overhead

Token FinOps

Why modern React/Next.js RSC hydration payloads waste up to 99% of LLM context tokens, how to measure DOM Signal-to-Noise ratio, and how to serve clean Markdown via HTTP Accept: text/markdown content negotiation.

Read HTML Token Optimization Guide →

5. /.well-known/mcp.json & Model Context Protocol Discovery

Agentic Endpoints

How domains advertise structured read/query endpoints to autonomous coding and research agents via /.well-known/mcp.json, with CORS, caching, security headers, and live production schema examples (3 active in index).

Read /.well-known/mcp.json Guide →

6. The 2026 Web-to-Token Empirical Benchmark Study

Original Dataset

Reproducible telemetry across 31 flagship engineering and SaaS domains measuring raw HTML byte payloads, BPE token counts, stripped prose ratios, and inference cost savings from /llms.txt.

View 2026 Empirical Dataset →

Interactive Verification & Specification Builders

Zero-Signup Engineering Utilities
Live Domain & /llms.txt Inspector Run an instant HTTPS socket, tokenizer, /llms.txt syntax lint, and 10-bot robots.txt audit on any host
/llms.txt Specification Generator Crawl any live website’s document metadata, canonical navigation, and XML sitemap into compliant Markdown
AI Crawler robots.txt Rule Builder Generate deterministic Allow / Disallow directives separating AI citation search from bulk training
Cloud Egress & LLM Token Cost Estimator Compare monthly serverless compute, bandwidth egress, and frontier vs. flash LLM API inference rates