SITECUBE /llms.txt Validator · AI Crawler Registry · Web-to-Token Observatory

gitlab.com

A (85/100) Open to Search & Training

GitLab - Speed with control for agentic software engineering

Raw HTML Tokens: ~56,566 tok
Clean Text Tokens: ~1,903 tok (3.4% signal)
/llms.txt Manifest: Not deployed
Edge / TTFB: Cloudflare · 404 ms
AI & /llms.txt Readiness Signal-to-Noise Ratio Token Savings via /llms.txt Network TTFB TLS & HTTP Security DNS & Edge Stack
57/100 3.4% (1,903 / 56,566 tok) 0% (No /llms.txt) 404 ms (84/100) 100/100 Cloudflare (Nuxt)

Web-to-Token Efficiency & Agent Ingestion Cost

~4 bytes/token UTF-8 heuristic
Ingestion Source Payload Est. Tokens Cost / 1k Reads (Flash $0.35) Cost / 1k Reads (Pro $1.25)
Raw HTML DOM (GET /) 221.0 KB 56,566 $19.7981 $70.7075
Stripped Readable Text 7.4 KB 1,903 $0.6661 $2.3788
/llms.txt Manifest Not deployed — agents must parse 56,566 raw HTML tokens
HTML Signal-to-Noise Ratio: 3.4% readable text vs. 96.6% markup/script overhead.

/llms.txt Spec Linter & Agent Endpoint Probe

Status: Not Deployed
GET /llms.txt Missing — Generate spec-compliant /llms.txt for gitlab.com →
GET /llms-full.txt Not deployed
GET /.well-known/mcp.json Not advertised
JSON-LD Schema Entities WebSite, WebPage, ReadAction, Organization, Corporation, PostalAddress, Person, DefinedTerm, ImageObject, QuantitativeValue

AI Crawler /robots.txt Access Matrix

Open to Search & Training
User-Agent Token Purpose / Operator Effective Directive
OAI-SearchBot ChatGPT Search Citations Allow: /
PerplexityBot Perplexity Answer Citations Allow: /
ChatGPT-User Live User Agent Browsing Allow: /
Claude-SearchBot Anthropic Search Grounding Allow: /
GPTBot Foundation Model Training Allow: /
ClaudeBot Foundation Model Training Allow: /
Google-Extended Gemini Training Control Allow: /
Applebot-Extended Apple Intelligence Training Allow: /
Meta-ExternalAgent Llama Training & Indexing Allow: /
CCBot Open Dataset Training Corpus Allow: /

HTTP Response & Handshake Timing

GET https://about.gitlab.com/
DNS Lookup 14 ms HTML Document 221 KB
TCP Connect 11 ms Script Tags 13
TLS Handshake 171 ms Stylesheets 2
Time to First Byte (TTFB) 404 ms Images (with alt) 37 / 39
Total Transfer 407 ms HTTP/3 (Alt-Svc) Yes (h3)

Edge Network & Application Stack

Header & DOM Signatures
Edge / Reverse Proxy Cloudflare
Application Framework Nuxt
Server Header cloudflare
Mail Provider (MX) Google Workspace
Telemetry Signatures GA4 / GTM
TXT Verifications Microsoft Entra, Google Search Console, Stripe Verified, OpenAI Domain Verified, Atlassian

TLS Certificate & HTTP Security Policy

Port 443 X.509
Certificate Issuer Sectigo Limited (177 days remaining)
Strict-Transport-Security Present
Content-Security-Policy Present
X-Content-Type-Options Present
X-Frame-Options Present
Referrer-Policy Present
Permissions-Policy Present
SPF / DMARC Policy SPF: Yes · DMARC: Yes

Authoritative DNS & Document Head

A / AAAA / NS / Canonical
A (IPv4) 172.65.251.78
AAAA (IPv6) 2606:4700:90::f22e:fbec:5bed:a9b9
NS (Nameservers) diva.ns.cloudflare.com, jermaine.ns.cloudflare.com
HTML Title (60c) GitLab - Speed with control for agentic software engineering
/sitemap.xml Valid XML Sitemap

Hosted /llms.txt Mirror & Status Badge

Platform Setup Guide →
1-Line <head> Tag (for Shopify, Webflow, Framer, Wix, Squarespace & WordPress):
Sitecube status for gitlab.com
GitHub / Docs Markdown Badge:
HTML Badge:

Technical Synthesis & Agentic Ingestion Analysis: gitlab.com

Automated Engineering Assessment

1. HTML-to-Token Efficiency & LLM Context Window Cost

An unauthenticated HTTPS GET request to https://about.gitlab.com/ returns an initial HTML payload of 221.0 KB, corresponding to approximately 56,566 UTF-8 tokens when ingested raw by an AI coding agent, retrieval pipeline, or headless browser. After stripping inline <script>, <style>, <svg>, and DOM markup tags, the human-readable text footprint measures 1,903 tokens (7.4 KB)—yielding an HTML Signal-to-Noise Ratio of 3.4%. Because gitlab.com does not currently serve a machine-readable /llms.txt manifest at its root path, autonomous agents must either parse the full 56,566-token HTML payload ($70.7075 per 1,000 frontier reads) or run a DOM-stripping pass before grounding.

2. AI Crawler Governance & /robots.txt Posture

Based on live inspection of https://gitlab.com/robots.txt, gitlab.com maintains a Open to Search & Training posture. Specifically, AI search citation crawlers (OAI-SearchBot: Allowed, PerplexityBot: Allowed) and foundation model training crawlers (GPTBot: Allowed, ClaudeBot: Allowed, CCBot: Allowed) are governed explicitly by the host’s root robots policy.

3. Edge Delivery, TLS Handshake & Security Header Assessment

Traffic for gitlab.com resolves to 172.65.251.78 via Cloudflare running Nuxt, achieving a measured Time to First Byte (TTFB) of 404 ms (DNS lookup: 14 ms, TCP connect: 11 ms, TLS handshake: 171 ms). The port 443 TLS certificate is issued by Sectigo Limited with 177 days remaining before expiration (valid through 2027-05-12). Outbound email authentication via SPF is present and DMARC (_dmarc.gitlab.com) is enforced.

Comparable Hosts in Web-to-Token & /llms.txt Registry

Full Index (28 hosts) →
Host Rating /llms.txt HTML Tokens Text Tokens Signal % Crawler Posture TTFB
sitecube.com A+ 98 1,744 tok 12,387 1,227 9.9% Open to Search & Training 66 ms
cloudflare.com A+ 96 4,294 tok 333,631 471 0.1% Open to Search & Training 290 ms
trendyflash.com A+ 95 3,120 tok 12,197 1,705 14% Open to Search & Training 90 ms
multifactor.com A+ 94 656 tok 98,275 673 0.7% Open to Search & Training 109 ms
tierones.io A+ 93 481 tok 3,744 693 18.5% Open to Search & Training 30 ms
dosefacts.org A+ 93 10,613 tok 29,181 7,309 25% Open to Search & Training 107 ms
github.com A+ 92 7,165 tok 131,391 1,159 0.9% Open to Search & Training 43 ms
vercel.com A+ 92 1,385 tok 149,553 1,193 0.8% Open to Search & Training 57 ms
aiimpacted.org A+ 92 1,355 tok 17,717 3,605 20.3% Open to Search & Training 173 ms
cloq.com A 91 1,169 tok 2,109 690 32.7% Open to Search & Training 34 ms
supabase.com A 89 879 tok 337,513 1,905 0.6% Open to Search & Training 55 ms
shopify.com A 89 3,161 tok 172,951 513 0.3% Open to Search & Training 104 ms