llms.txt: what it is, what it is not, and how to write one that earns its place

llms.txt is a markdown file at your domain root that tells an AI client what your site is and which pages matter. Jeremy Howard proposed it in 2024, and it has spread quickly through developer-tools companies — Anthropic, Stripe, and Cloudflare all publish one.

Two things are true at once, and skipping either produces bad advice. The format is genuinely useful and cheap to adopt. And no major AI provider has committed to reading it as a retrieval input. This guide treats it accordingly.

What the format is

llms.txt lives at https://example.com/llms.txt. It is markdown, and the structure is fixed enough to be parsed but loose enough to write by hand:

# Acme

> Acme is usage-based billing infrastructure for SaaS companies.
> Meter events, price them, and invoice — without building a billing system.

Acme handles metering, rating, invoicing, and revenue recognition.
Most customers integrate in a day using the HTTP API or one of six SDKs.

## Docs

- [Quickstart](https://acme.com/docs/quickstart): Send your first metered event in ten minutes
- [Pricing models](https://acme.com/docs/pricing-models): Tiered, volume, package, and hybrid schemes
- [API reference](https://acme.com/docs/api): Full HTTP surface with request and response schemas

## Product

- [Pricing](https://acme.com/pricing): Plans, limits, and overage rates
- [Security](https://acme.com/security): SOC 2 Type II, data residency, retention

## Optional

- [Changelog](https://acme.com/changelog): Weekly release notes

The parts, in order:

1. An H1 with the site or organisation name. Exactly one.
2. A blockquote with a one- or two-sentence summary. This is the highest-leverage text in the file — it is the sentence most likely to be lifted verbatim when something describes you.
3. Optional prose giving context that does not fit the blockquote. No headings here.
4. H2 sections grouping links. Each link is [Title](absolute-url): one-line summary.
5. An ## Optional section at the end for material a client can skip when it needs to be brief.

Some sites also publish llms-full.txt, which inlines the full text of the linked documents into a single file. It is a different artefact with a different use — a compact corpus to paste into a context window, rather than a map — and it is only worth maintaining if your docs are stable enough that it will not go stale.

The mistakes that make it useless

Dumping your sitemap into it. The value is curation. Two hundred links with no summaries is worse than eight links with good ones, because it hands the consumer the same “figure out what matters” problem it already had.

Marketing voice in the blockquote. “Acme empowers modern teams to unlock the future of revenue” tells a model nothing it can use to decide whether your page answers a question. Say what the product does, for whom, in concrete nouns. The test: could a competitor’s blockquote be swapped in without anyone noticing? If yes, rewrite it.

Relative URLs. The file may be read entirely out of context, with no base URL. Every link must be absolute.

Letting it rot. A file listing three deprecated endpoints is actively harmful — it is a confident, machine-readable statement of things that are no longer true. Either wire it into your release process or keep it small enough that reviewing it is trivial.

Serving it as HTML. It should return text/plain or text/markdown. A framework that wraps unknown routes in your app shell will silently break it. Check with curl -sI https://example.com/llms.txt.

What it is not

It is not robots.txt. robots.txt is access control that crawlers broadly honour; llms.txt is editorial guidance that consumers may ignore. Publishing llms.txt grants no permissions and revokes none. If you want to control what AI crawlers may fetch, that is a robots.txt and edge-rules problem — see robots.txt for AI crawlers.

It is not a ranking signal. Nobody has published evidence that llms.txt affects retrieval or citation frequency, and you should be suspicious of anyone selling it as one.

It is not a substitute for the pages themselves. A model that follows your links still has to parse what it finds. If the destination pages extract badly, a tidy map just gets it to bad content faster.

Writing yours

Start from the questions people actually ask about you, not from your site structure. If prospects ask “does it support usage-based pricing”, “how long does integration take”, and “what does it cost”, then the linked pages should answer those three questions and the summaries should say so.

Keep it to fifteen or twenty links. Write summaries that state what the reader will learn, not what the page is called. Then set a calendar reminder to reread it in a quarter.

You can see ours — it is short on purpose.

Knowing whether anyone reads it

This is the part that turns llms.txt from an act of faith into a measurable one. Requests to /llms.txt are ordinary HTTP requests, with a user agent attached. If you log them, you know precisely which operators fetch the file, how often, and whether they come back after you change it.

Since client-side analytics never see them, that measurement has to happen server-side or at whatever sits in front of your origin. Shiftrank serves the file at the edge and records every fetch — which is how we can say with confidence that ours gets read.