huggingface.co

B+ (81/100) Open to Search & Training

Hugging Face – The AI community building the future.

Raw HTML Tokens: ~43,975 tok
Clean Text Tokens: ~1,156 tok (2.6% signal)
/llms.txt Manifest: Not deployed
Edge / TTFB: AWS CloudFront · 176 ms
AI & /llms.txt Readiness Signal-to-Noise Ratio Token Savings via /llms.txt Network TTFB TLS & HTTP Security DNS & Edge Stack
51/100 2.6% (1,156 / 43,975 tok) 0% (No /llms.txt) 176 ms (92/100) 79/100 AWS CloudFront (Custom SSR / Static)

Web-to-Token Efficiency & Agent Ingestion Cost

~4 bytes/token UTF-8 heuristic
Ingestion Source Payload Est. Tokens Cost / 1k Reads (Flash $0.35) Cost / 1k Reads (Pro $1.25)
Raw HTML DOM (GET /) 171.8 KB 43,975 $15.3913 $54.9688
Stripped Readable Text 4.5 KB 1,156 $0.4046 $1.4450
/llms.txt Manifest Not deployed — agents must parse 43,975 raw HTML tokens
HTML Signal-to-Noise Ratio: 2.6% readable text vs. 97.4% markup/script overhead. Token Optimization Guide →

/llms.txt Spec Linter & Agent Endpoint Probe

Status: Not Deployed · Spec Rules →
GET /llms.txt Missing — Generate spec-compliant /llms.txt for huggingface.co →
GET /llms-full.txt Not deployed
GET /.well-known/mcp.json Not advertised
JSON-LD Schema Entities None detected in initial HTML

AI Crawler /robots.txt Access Matrix

Open to Search & Training · Bot Directory →
User-Agent Token Purpose / Operator Effective Directive
OAI-SearchBot ChatGPT Search Citations Allow: /
PerplexityBot Perplexity Answer Citations Allow: /
ChatGPT-User Live User Agent Browsing Allow: /
Claude-SearchBot Anthropic Search Grounding Allow: /
GPTBot Foundation Model Training Allow: /
ClaudeBot Foundation Model Training Allow: /
Google-Extended Gemini Training Control Allow: /
Applebot-Extended Apple Intelligence Training Allow: /
Meta-ExternalAgent Llama Training & Indexing Allow: /
CCBot Open Dataset Training Corpus Allow: /

HTTP Response & Handshake Timing

GET https://huggingface.co/
DNS Lookup 5 ms HTML Document 171.8 KB
TCP Connect 52 ms Script Tags 8
TLS Handshake 54 ms Stylesheets 2
Time to First Byte (TTFB) 176 ms Images (with alt) 26 / 28
Total Transfer 185 ms HTTP/3 (Alt-Svc) Yes (h3)

Edge Network & Application Stack

Header & DOM Signatures
Edge / Reverse Proxy AWS CloudFront
Application Framework Custom SSR / Static
Server Header Protected Edge
Mail Provider (MX) Google Workspace
Telemetry Signatures Stripe.js
TXT Verifications Anthropic Verified, Google Search Console, OpenAI Domain Verified, Stripe Verified, Microsoft Entra

TLS Certificate & HTTP Security Policy

Port 443 X.509
Certificate Issuer Amazon (102 days remaining)
Strict-Transport-Security Present
Content-Security-Policy Not set
X-Content-Type-Options Not set
X-Frame-Options Present
Referrer-Policy Present
Permissions-Policy Not set
SPF / DMARC Policy SPF: Yes · DMARC: Yes

Authoritative DNS & Document Head

A / AAAA / NS / Canonical
A (IPv4) 18.161.125.35, 18.161.125.68, 18.161.125.127, 18.161.125.84
AAAA (IPv6) 2600:9000:2078:800:17:b174:6d00:93a1, 2600:9000:2078:8800:17:b174:6d00:93a1, 2600:9000:2078:8a00:17:b174:6d00:93a1, 2600:9000:2078:a000:17:b174:6d00:93a1
NS (Nameservers) ns-1452.awsdns-53.org, ns-137.awsdns-17.com, ns-1955.awsdns-52.co.uk, ns-919.awsdns-50.net
HTML Title (52c) Hugging Face – The AI community building the future.
/sitemap.xml Valid XML Sitemap

Hosted /llms.txt Mirror & Status Badge

Platform Setup Guide →
1-Line <head> Tag (for Shopify, Webflow, Framer, Wix, Squarespace & WordPress):
Sitecube status for huggingface.co
GitHub / Docs Markdown Badge:
HTML Badge:

Technical Synthesis & Agentic Ingestion Analysis: huggingface.co

Automated Engineering Assessment

1. HTML-to-Token Efficiency & LLM Context Window Cost

An unauthenticated HTTPS GET request to https://huggingface.co/ returns an initial HTML payload of 171.8 KB, corresponding to approximately 43,975 UTF-8 tokens when ingested raw by an AI coding agent, retrieval pipeline, or headless browser. After stripping inline <script>, <style>, <svg>, and DOM markup tags, the human-readable text footprint measures 1,156 tokens (4.5 KB)—yielding an HTML Signal-to-Noise Ratio of 2.6%. Because huggingface.co does not currently serve a machine-readable /llms.txt manifest at its root path, autonomous agents must either parse the full 43,975-token HTML payload ($54.9688 per 1,000 frontier reads) or run a DOM-stripping pass before grounding.

2. AI Crawler Governance & /robots.txt Posture

Based on live inspection of https://huggingface.co/robots.txt, huggingface.co maintains a Open to Search & Training posture. Specifically, AI search citation crawlers (OAI-SearchBot: Allowed, PerplexityBot: Allowed) and foundation model training crawlers (GPTBot: Allowed, ClaudeBot: Allowed, CCBot: Allowed) are governed explicitly by the host’s root robots policy.

3. Edge Delivery, TLS Handshake & Security Header Assessment

Traffic for huggingface.co resolves to 18.161.125.35 via AWS CloudFront running Custom SSR / Static, achieving a measured Time to First Byte (TTFB) of 176 ms (DNS lookup: 5 ms, TCP connect: 52 ms, TLS handshake: 54 ms). The port 443 TLS certificate is issued by Amazon with 102 days remaining before expiration (valid through 2027-02-27). Outbound email authentication via SPF is present and DMARC (_dmarc.huggingface.co) is enforced.

Comparable Hosts in Web-to-Token & /llms.txt Registry

Full Index (32 hosts) →
Host Rating /llms.txt HTML Tokens Text Tokens Signal % Crawler Posture TTFB
sitecube.com A+ 98 2,099 tok 16,563 1,347 8.1% Open to Search & Training 96 ms
cloudflare.com A+ 96 4,294 tok 333,631 471 0.1% Open to Search & Training 294 ms
trendyflash.com A+ 95 3,256 tok 12,230 1,739 14.2% Open to Search & Training 85 ms
multifactor.com A+ 94 656 tok 98,275 673 0.7% Open to Search & Training 109 ms
tierones.io A+ 93 481 tok 3,744 693 18.5% Open to Search & Training 30 ms
aiimpacted.org A+ 93 1,355 tok 17,701 3,599 20.3% Open to Search & Training 119 ms
github.com A+ 92 7,165 tok 131,479 1,185 0.9% Open to Search & Training 26 ms
vercel.com A+ 92 1,385 tok 149,553 1,193 0.8% Open to Search & Training 66 ms
render.com A+ 92 13,304 tok 190,754 2,018 1.1% Open to Search & Training 306 ms
cloq.com A 91 1,169 tok 2,109 690 32.7% Open to Search & Training 34 ms
postman.com A 91 5,375 tok 118,892 11,040 9.3% Open to Search & Training 769 ms
neon.tech A 90 7,418 tok 110,734 1,629 1.5% Open to Search & Training 106 ms