What Is /llms.txt? Complete Specification, Syntax, Examples & Linter Rules

How the /llms.txt and /llms-full.txt Markdown standards give AI search engines, coding assistants, and RAG pipelines clean, low-token domain context.

1. Executive Definition: Why /llms.txt Exists

Root Path: https://example.com/llms.txt

/llms.txt is a plain-text Markdown file served at the root of a domain (https://yourdomain.com/llms.txt) that provides Large Language Models (LLMs), AI search engines, and autonomous coding agents with a concise, curated map of a website’s architecture and canonical documentation URLs.

Unlike traditional search engine crawlers that index millions of pages asynchronously into an inverted keyword index, LLM agents operate inside finite, metered context windows at inference time. When an agent fetches a modern web page directly, up to 95%–99.8% of the HTTP response bytes consist of JavaScript hydration bundles, inline SVG vector paths, utility CSS classes, and navigation menus. Publishing /llms.txt gives agents a deterministic, human-and-machine-readable index measured in hundreds or a few thousand tokens rather than hundreds of thousands of noisy DOM tokens.

2. /llms.txt vs. /llms-full.txt vs. /robots.txt vs. /sitemap.xml

Protocol Comparison
Root File Format Primary Role Typical Size Target Consumer
/llms.txt Markdown (UTF-8) Curated architecture summary & annotated high-priority URLs 1 KB – 30 KB (~250–7,500 tok) Coding agents, AI search grounding, RAG routers
/llms-full.txt Markdown (UTF-8) Complete concatenated documentation prose in one file 50 KB – 2 MB (~12k–500k tok) 1M-context frontier models, IDE doc indexing
/robots.txt RFC 9309 Plain Text Crawl permission & disallow access rules per User-agent 0.5 KB – 10 KB All automated web crawlers & training scrapers
/sitemap.xml XML Sitemap 0.9 Exhaustive list of every indexable URL & <lastmod> date 10 KB – 50 MB (thousands of URLs) Googlebot, Bingbot, traditional search indexers
Key Distinction: /robots.txt tells crawlers what they are allowed to fetch; /sitemap.xml lists every URL; /llms.txt explains what matters most with semantic descriptions.

3. Required Markdown Grammar & Order

Sitecube Linter Rules
1. Primary # H1 Header Required (Line 1): A single top-level Markdown heading naming the project, company, or documentation site (e.g., # Stripe Documentation).
2. > Blockquote Summary Recommended: A concise 1–3 sentence > summary immediately below the H1 stating what the product or domain does in unambiguous technical terms.
3. Orientation Notes Optional Prose: Brief body paragraphs (no headings) clarifying conventions, API versions, or authentication prerequisites.
4. ## H2 Link Sections Core Structure: Categorized H2 sections containing Markdown bullet lists formatted as - [Link Title](https://...): Concise 1-line explanation.
5. ## Optional Section Reserved Heading: Links placed under ## Optional signal to LLM harnesses that those URLs may be skipped when operating under tight token budgets.
6. HTTP Delivery Headers Must return HTTP 200 with Content-Type: text/plain; charset=utf-8 or text/markdown; charset=utf-8 (never an HTML SPA shell or soft-404 page).

Reference /llms.txt Template

4. Live Production /llms.txt Implementations (Observed Telemetry)

View All 31 Indexed Hosts →
Production Host Raw HTML Tokens /llms.txt Tokens Token Compression Edge / Stack Live Audit Dossier
sitecube.com 16,563 tok 2,099 tok -87.3% Hostinger Edge CDN · Custom SSR / Static Inspect Linter Report →
cloudflare.com 333,631 tok 4,294 tok -98.7% Cloudflare · Astro Inspect Linter Report →
trendyflash.com 12,230 tok 3,256 tok -73.4% Hostinger Edge CDN · Next.js Inspect Linter Report →
multifactor.com 98,275 tok 656 tok -99.3% Cloudflare · Next.js Inspect Linter Report →
tierones.io 3,744 tok 481 tok -87.2% Hostinger Edge CDN · Custom SSR / Static Inspect Linter Report →
dosefacts.org 29,181 tok 10,613 tok -63.6% Hostinger Edge CDN · Custom SSR / Static Inspect Linter Report →
aiimpacted.org 17,701 tok 1,355 tok -92.3% Hostinger Edge CDN · Custom SSR / Static Inspect Linter Report →
github.com 131,479 tok 7,165 tok -94.6% GitHub Pages · Custom SSR / Static Inspect Linter Report →
vercel.com 149,553 tok 1,385 tok -99.1% Vercel · Next.js Inspect Linter Report →
cloq.com 2,109 tok 1,169 tok -44.6% Hostinger Edge CDN · Custom SSR / Static Inspect Linter Report →

5. Common Implementation Pitfalls & Technical FAQ

Production QA Checklist

Why does my single-page app fail /llms.txt validation even when HTTP status is 200?

Client-side routers (Vite, React Router, misconfigured Nginx try_files $uri /index.html) often return the main HTML <!DOCTYPE html> shell with an HTTP 200 status code for every unknown route, including /llms.txt. Sitecube’s validator explicitly rejects HTML bodies and soft-404 JSON payloads. Ensure your web server or CDN serves static files in public/llms.txt before falling back to your SPA router.

Does adding /llms.txt affect Google Search rankings?

Google Search continues to crawl HTML pages via Googlebot and XML sitemaps. Publishing /llms.txt does not replace standard HTML metadata, structured JSON-LD, or /sitemap.xml; rather, it complements them by making your documentation dramatically cheaper and more accurate for AI coding assistants (Cursor, Antigravity, Claude Code, Windsurf) and LLM retrieval pipelines to ground against—reducing hallucinated API parameters and broken citation links.

What if my CMS (Shopify, Webflow, Squarespace, Wix) blocks root file uploads?

You can either configure a 301 Redirect from /llms.txt to a hosted Markdown manifest, or place a standard HTML autodiscovery tag inside your <head>: <link rel="alternate" type="text/markdown" title="llms.txt" href="..." />. Follow our CMS /llms.txt Setup Guide or generate a compliant file from your live sitemap using the /llms.txt Specification Generator.