SITECUBE /llms.txt Validator · AI Crawler Registry · Web-to-Token Observatory

github.com

A+ (92/100) Open to Search & Training

GitHub · Change is constant. GitHub keeps you ahead. · GitHub · GitHub

Raw HTML Tokens: ~131,391 tok
Clean Text Tokens: ~1,159 tok (0.9% signal)
/llms.txt Manifest: 7,165 tok (Spec Compliant)
Edge / TTFB: GitHub Pages · 43 ms
AI & /llms.txt Readiness Signal-to-Noise Ratio Token Savings via /llms.txt Network TTFB TLS & HTTP Security DNS & Edge Stack
84/100 0.9% (1,159 / 131,391 tok) -94.5% fewer tokens 43 ms (98/100) 93/100 GitHub Pages (Custom SSR / Static)

Web-to-Token Efficiency & Agent Ingestion Cost

~4 bytes/token UTF-8 heuristic
Ingestion Source Payload Est. Tokens Cost / 1k Reads (Flash $0.35) Cost / 1k Reads (Pro $1.25)
Raw HTML DOM (GET /) 513.2 KB 131,391 $45.9869 $164.2388
Stripped Readable Text 4.5 KB 1,159 $0.4057 $1.4488
/llms.txt Manifest 28.0 KB 7,165 $2.5078 $8.9563
HTML Signal-to-Noise Ratio: 0.9% readable text vs. 99.1% markup/script overhead.

/llms.txt Spec Linter & Agent Endpoint Probe

Status: Spec Compliant
GET /llms.txt HTTP 200 · 28,658 bytes (~7,165 tokens · 178 lines)
Spec Check: # H1 Title Pass ("GitHub")
Spec Check: > Summary Pass ("GitHub is a developer platform for building, shipping, and maintaining software. It provides cloud-based Git repository hosting, CI/CD via GitHub Actions, project management with I")
## H2 Sections & Links 12 sections · 117 canonical markdown links
GET /llms-full.txt HTTP 200 · 28,658 bytes (~7,165 tokens)
GET /.well-known/mcp.json Not advertised
JSON-LD Schema Entities None detected in initial HTML
Live Payload from https://github.com/llms.txt
# GitHub

> GitHub is a developer platform for building, shipping, and maintaining software. It provides cloud-based Git repository hosting, CI/CD via GitHub Actions, project management with Issues and Projects, code review via pull requests, AI-powered development with GitHub Copilot, and APIs (REST and GraphQL) for automation and integration.

## How to use

To retrieve full article content, page lists, or search results programmatically, please use the APIs below. To find a specific article, 

AI Crawler /robots.txt Access Matrix

Open to Search & Training
User-Agent Token Purpose / Operator Effective Directive
OAI-SearchBot ChatGPT Search Citations Allow: /
PerplexityBot Perplexity Answer Citations Allow: /
ChatGPT-User Live User Agent Browsing Allow: /
Claude-SearchBot Anthropic Search Grounding Allow: /
GPTBot Foundation Model Training Allow: /
ClaudeBot Foundation Model Training Allow: /
Google-Extended Gemini Training Control Allow: /
Applebot-Extended Apple Intelligence Training Allow: /
Meta-ExternalAgent Llama Training & Indexing Allow: /
CCBot Open Dataset Training Corpus Allow: /

HTTP Response & Handshake Timing

GET https://github.com/
DNS Lookup 3 ms HTML Document 513.2 KB
TCP Connect 17 ms Script Tags 12
TLS Handshake 10 ms Stylesheets 39
Time to First Byte (TTFB) 43 ms Images (with alt) 7 / 21
Total Transfer 55 ms HTTP/3 (Alt-Svc) Not advertised

Edge Network & Application Stack

Header & DOM Signatures
Edge / Reverse Proxy GitHub Pages
Application Framework Custom SSR / Static
Server Header github.com
Mail Provider (MX) Microsoft 365
Telemetry Signatures Zero Client Trackers
TXT Verifications Microsoft Entra, Google Search Console, Stripe Verified, OpenAI Domain Verified, Atlassian, Anthropic Verified

TLS Certificate & HTTP Security Policy

Port 443 X.509
Certificate Issuer Sectigo Limited (72 days remaining)
Strict-Transport-Security Present
Content-Security-Policy Present
X-Content-Type-Options Present
X-Frame-Options Present
Referrer-Policy Present
Permissions-Policy Not set
SPF / DMARC Policy SPF: Yes · DMARC: Yes

Authoritative DNS & Document Head

A / AAAA / NS / Canonical
A (IPv4) 4.187.247.160
AAAA (IPv6) None
NS (Nameservers) ns-421.awsdns-52.com, ns-1707.awsdns-21.co.uk, dns3.p08.nsone.net, dns4.p08.nsone.net, ns-1283.awsdns-32.org, ns-520.awsdns-01.net
HTML Title (70c) GitHub · Change is constant. GitHub keeps you ahead. · GitHub · GitHub
/sitemap.xml Not found

Hosted /llms.txt Mirror & Status Badge

Platform Setup Guide →
1-Line <head> Tag (for Shopify, Webflow, Framer, Wix, Squarespace & WordPress):
Sitecube status for github.com
GitHub / Docs Markdown Badge:
HTML Badge:

Technical Synthesis & Agentic Ingestion Analysis: github.com

Automated Engineering Assessment

1. HTML-to-Token Efficiency & LLM Context Window Cost

An unauthenticated HTTPS GET request to https://github.com/ returns an initial HTML payload of 513.2 KB, corresponding to approximately 131,391 UTF-8 tokens when ingested raw by an AI coding agent, retrieval pipeline, or headless browser. After stripping inline <script>, <style>, <svg>, and DOM markup tags, the human-readable text footprint measures 1,159 tokens (4.5 KB)—yielding an HTML Signal-to-Noise Ratio of 0.9%. Because github.com publishes a valid /llms.txt manifest (measuring 7,165 tokens across 178 lines), AI retrieval agents that read /llms.txt instead of scraping the raw DOM reduce token consumption by 94.5%—cutting 1,000-read inference input costs from $164.2388 to $8.9563 at frontier model rates ($1.25/1M tokens), or $2.5078 on Gemini 4 Hydrogen ($0.35/1M).

2. Indexed Structure Inside github.com/llms.txt

Sitecube’s specification linter verified 12 H2 section headings and 117 Markdown links inside https://github.com/llms.txt under the primary heading "GitHub". Documented sections include: How to use</code>, <code>Building with GitHub, for coding agents and automation</code>, <code>GitHub Copilot</code>, <code>Authentication</code>, <code>Get started</code>, <code>Account and profile</code>, <code>GitHub Pages</code>, <code>GitHub Actions. Key canonical endpoints referenced in the manifest include Versions API (https://docs.github.com/api/pagelist/versions), Languages API (https://docs.github.com/api/pagelist/languages), Page List API (https://docs.github.com/api/pagelist/en/free-pro-team@latest), Article API (https://docs.github.com/api/article), Article Body API (https://docs.github.com/api/article/body).

3. Edge Delivery, TLS Handshake & Security Header Assessment

Traffic for github.com resolves to 4.187.247.160 via GitHub Pages running Custom SSR / Static, achieving a measured Time to First Byte (TTFB) of 43 ms (DNS lookup: 3 ms, TCP connect: 17 ms, TLS handshake: 10 ms). The port 443 TLS certificate is issued by Sectigo Limited with 72 days remaining before expiration (valid through 2027-01-27). Outbound email authentication via SPF is present and DMARC (_dmarc.github.com) is enforced.

Comparable Hosts in Web-to-Token & /llms.txt Registry

Full Index (28 hosts) →
Host Rating /llms.txt HTML Tokens Text Tokens Signal % Crawler Posture TTFB
sitecube.com A+ 98 1,744 tok 12,387 1,227 9.9% Open to Search & Training 66 ms
cloudflare.com A+ 96 4,294 tok 333,631 471 0.1% Open to Search & Training 290 ms
trendyflash.com A+ 95 3,120 tok 12,197 1,705 14% Open to Search & Training 36 ms
multifactor.com A+ 94 656 tok 98,275 673 0.7% Open to Search & Training 109 ms
tierones.io A+ 93 481 tok 3,744 693 18.5% Open to Search & Training 30 ms
dosefacts.org A+ 93 10,613 tok 29,181 7,309 25% Open to Search & Training 107 ms
github.com A+ 92 7,165 tok 131,391 1,159 0.9% Open to Search & Training 43 ms
vercel.com A+ 92 1,385 tok 149,553 1,193 0.8% Open to Search & Training 57 ms
aiimpacted.org A+ 92 1,355 tok 17,717 3,605 20.3% Open to Search & Training 173 ms
cloq.com A 91 1,169 tok 2,109 690 32.7% Open to Search & Training 34 ms
supabase.com A 89 879 tok 337,513 1,905 0.6% Open to Search & Training 55 ms
shopify.com A 89 3,161 tok 172,951 513 0.3% Open to Search & Training 104 ms