Notable Absences

Some of the most interesting data points are who doesn't have an llms.txt:

• Anthropic and OpenAI — the two leading LLM companies don't serve one. OpenAI returns a 403; Anthropic a 404. The irony writes itself.
• Google, Apple, Meta — Big Tech is absent across the board (though AWS docs, a separate domain, does have one).
• Cloudflare — builds infrastructure for the web but doesn't have one. Neither does Fastly or Netlify.
• HuggingFace — the largest open-source AI platform. No llms.txt.
• Every single news organization — NYT, BBC, Reuters, Washington Post, Bloomberg, TechCrunch, The Verge. Zero adoption. This likely reflects the industry's broader adversarial stance toward AI scraping.

The media industry's total absence is telling. While developer tools are racing to make their content AI-accessible, publishers are still fighting to keep AI out. These are fundamentally different bets on the same future.

What's Actually in the Files

Not all llms.txt files are created equal. After reading through all 62 files, a few distinct patterns emerge:

Pattern 1: The Documentation Index (most common)

The majority of files are structured catalogs of documentation pages. Stripe, Vercel, Datadog, Pinecone, and others essentially dump their entire doc tree into markdown links with one-line descriptions.

# Stripe Documentation

When installing Stripe packages, always check the npm registry for the latest version rather than relying on memorized version numbers.

- [Authentication](https://docs.stripe.com/auth.md)
- [Payment Intents](https://docs.stripe.com/payments.md)
- [Webhooks](https://docs.stripe.com/webhooks.md)
...

These files tend to be large — often hitting the 50 KB range. Stripe's has versioning warnings. Docker's links every concept guide. AWS covers dozens of services. The implicit audience is an AI agent that needs to find the right doc page for a specific task.

Pattern 2: The Company Brief (concise)

Some files are compact descriptions — who we are, what we do, a few key links. Shopify (861 bytes), Railway (868 bytes), and MongoDB (1.5 KB) take this approach.

# Shopify

> Shopify is a commerce platform that helps you > sell online and in person. Entrepreneurs, > retailers, and global brands use Shopify to > process sales, run stores, and grow their > businesses.

## Documentation

- [Shopify API docs](https://shopify.dev/docs/api)
- [Shopify CLI](https://shopify.dev/docs/api/shopify-cli)

These feel more like elevator pitches for AI. They answer the question "what is this site?" rather than trying to replace the docs. Clean, effective, low maintenance.

Pattern 3: The Hub File

Supabase and NVIDIA take a meta-approach: their top-level llms.txt links to other, more specific llms.txt files. Supabase has separate files for each SDK (JavaScript, Dart, Python, etc.). NVIDIA links to each country subdomain. This is the most architecturally sophisticated pattern — treating llms.txt as a routing layer.

# Supabase Docs

- [Guides](https://supabase.com/llms/guides.txt)
- [Reference (JavaScript)](https://supabase.com/llms/js.txt)
- [Reference (Dart)](https://supabase.com/llms/dart.txt)
- [Reference (Python)](https://supabase.com/llms/python.txt)

Pattern 4: The Kitchen Sink

Some files try to include everything. Mailchimp's announces "988 web pages across 19 categories." Auth0, Okta, and Salesforce dump what looks like auto-generated sitemaps into the file. These max out at 50 KB and feel more like SEO artifacts than thoughtfully curated AI resources.

The most effective files share a few properties: they lead with a clear description of what the site is, they link to .md versions of docs (not HTML), and they're organized by task rather than by URL hierarchy.

Size Distribution

File Sizes Range From 546 Bytes (Supabase) To 50 KB (Stripe, Vercel, AWS, And a Dozen Others That Clearly Hit a Generation Limit)

• Under 2 KB — 8 files. Concise company briefs. (Shopify, Railway, Supabase, Svelte, ClickUp, MongoDB)
• **2–