Writing Docs That AI Agents Can Read

Your documentation has a new kind of reader. A coding agent looks up an API call before it writes the code. A chat assistant with search reads your pricing page before it answers a customer. Neither one scrolls, and neither cares about your sidebar.

Most docs sites were built for people with browsers, not for agents.

The fix is mostly boring, cheap work. This post covers what to do, in the order that pays off fastest.

Why normal docs pages are hard for agents

A rendered HTML page carries navigation, scripts, banners and layout markup around the real content. An agent has to dig the text back out of all that. It is slow and imprecise, and every extra token costs time and money.

Context windows have grown, but they are still too small to hold a whole docs site. The same idea sits behind llms.txt, covered below: agents do best with concise, expert-level information gathered in one easy-to-reach place.

So the goal is simple. Give agents a clean version of each page, a short map of where things are, and pages that make sense without any surrounding context.

flowchart LR
    A[Agent needs an answer] --> B[Fetch /llms.txt]
    B --> C[Pick the relevant page]
    C --> D[Fetch page.md]
    A --> E[Request page URL with Accept: text/markdown]
    E --> D
    D --> F[Answer from clean Markdown]

Step 1: Publish a Markdown copy of every page

The llms.txt proposal suggests that every page an agent might need also exists as plain Markdown at the same URL, with .md added. So /docs/quickstart.html becomes /docs/quickstart.html.md, or /docs/quickstart.md if you replace the extension. URLs without a file name use index.md.

If you use a docs platform, you may already have this. Mintlify generates a .md version of every page, and the links in its llms.txt point to those. GitBook serves an llms.txt file for published docs. For self-hosted sites there are plugins, such as docusaurus-plugin-llms and vitepress-plugin-llms.

Also tell clients where the Markdown lives. The proposal recommends a link relation of rel="alternate" type="text/markdown", either as a <link> element or as an HTTP header:

Link: </docs/quickstart.md>; rel="alternate"; type="text/markdown"

The header form can be added in your web server or CDN config without touching a single page.

Step 2: Serve Markdown when an agent asks for it

The other route is content negotiation. An agent sends Accept: text/markdown, and your server replies with Markdown at the normal page URL. Browsers keep getting HTML.

curl https://example.com/docs/quickstart -H "Accept: text/markdown"

If your site is on Cloudflare, you can get this without changing your origin. Cloudflare's Markdown for Agents, announced in February 2026, converts HTML to Markdown at the edge for zones that turn it on. It is available on Pro, Business and Enterprise plans at no extra cost. You can switch it on for a whole zone under AI Crawl Control, or for specific subdomains and paths with a configuration rule.

The output is tidy. Navigation, footers and scripts are stripped, the title and description come from your meta tags, and any JSON-LD is kept at the end of the document. So good meta descriptions and structured data carry over.

A few things to know before you enable it:

• Only HTML is converted, and the limit is 6 MiB of decompressed HTML. Cloudflare raised it from 2 MiB in October 2026.

• Converted responses no longer carry token-count headers, so count tokens on your side if you need them.

• Converted responses include a Content-Signal header. If your origin doesn't set one, Cloudflare adds a default saying the content may be used for AI training, search and AI input. If your origin sets its own, Cloudflare keeps yours, so set it yourself if the default doesn't suit you.

If you run your own server, the same idea works. Look at the Accept header, return Markdown, and set Vary: Accept so caches keep the two versions apart.

Step 3: Add an llms.txt

llms.txt is a Markdown file that gives agents a short map of your docs. Jeremy Howard of Answer.AI proposed it in September 2024, and the proposal got a v2 update in August 2026.

Only one part is required: an H1 with the project name. After that, the format allows a blockquote summary, some free text, and H2 sections that hold lists of links with an optional note for each.

# Acme API

> Acme is a payments API for marketplaces. Amounts are in cents. Every request needs an API key.

Important notes:

- The v1 API is deprecated. Use v2 for new work.

## Docs

- [Quickstart](https://example.com/docs/quickstart.md): Create your first payment
- [Authentication](https://example.com/docs/auth.md): API keys and scopes
- [Errors](https://example.com/docs/errors.md): Every error code and what to do about it

## Optional

- [Changelog](https://example.com/docs/changelog.md)

By convention, a section named Optional marks links an agent can skip when it needs a shorter context.

A few tips:

• Link to the Markdown versions of your pages, not the HTML.

• Keep the file small enough to fit comfortably in context. The detail belongs behind the links.

• Write a short note for each link that says what the page is for.

• Put the file at the site root, or at /docs/llms.txt to cover only the docs. A file covers the URLs under its path, and agents should prefer the most specific one. Keep in mind that Lighthouse's llms.txt check looks at the domain root.

You will also see llms-full.txt, a single file holding the full text of the docs. It is a common companion and Mintlify generates it, but it is not part of the llms.txt proposal itself.

Does anyone actually read it?

Fair question. llms.txt is a proposal, not a ratified standard, and who reads it depends on who you ask.

Google Search does not. Its May 2026 guide to generative AI features says you don't need llms.txt to appear in AI Overviews or AI Mode. Google later added a clarification: it is fine to keep these files for other services that use them, and they neither help nor hurt your rankings, because Google Search ignores them.

On the other side, Chrome's Lighthouse added an experimental Agentic Browsing category in 2026 that includes an llms.txt check. The check is optional: a site with no file gets a Not Applicable result, not a failure. According to the llmstxt.org proposal, OpenAI, Anthropic and Google (for the Gemini API) all publish llms.txt files for their own developer docs.

The practical takeaway: don't expect it to move search rankings. Do expect it to help coding agents and developer tools that go looking for it. Treat it as a bonus once your Markdown pages exist.

Step 4: Write pages that make sense on their own

Format only gets you so far. An agent reading a .md page has no menus, no breadcrumbs and no memory of the previous page. What you write matters as much as how you publish it.

These rules are practical advice, not part of any spec:

• Make every page self-contained. Skip "as shown above" and "see the previous step". Repeat the one thing the reader needs.

• Put the answer first. A one-paragraph summary at the top helps humans and agents alike.

• Name your versions. Say which version of the API, SDK or runtime the page covers.

• Give complete examples. Include imports, setup and the expected output. Agents copy what they see.

• Say what is not supported. If a feature doesn't exist or was removed, write that down. Otherwise an agent may confidently invent it.

• Don't hide content. Text that lives only in an image, a video or a JavaScript widget disappears in a Markdown export. Put steps in plain text.

• Use plain headings. "Rotate an API key" beats "Keys, revisited".

• Fill in the description field. Platforms like Mintlify use each page's frontmatter description as the note in llms.txt, so keep it short and specific.

• List errors with causes and fixes. A table of error codes with a cause and a fix is the first thing an agent needs when something breaks.

Don't forget the repository

If your docs live next to your code, add an AGENTS.md. It is a plain Markdown file in the repo root that tells coding agents how to set up the project, run tests and follow your conventions. Think of it as a README for agents.

OpenAI contributed AGENTS.md to the Agentic AI Foundation, a Linux Foundation project, in December 2025, alongside Anthropic's Model Context Protocol and Block's goose.

Put in it what you would tell a new teammate: the install command, the test command, the lint rules, the pull request format, and anything agents should never touch. Keep it short and keep it current.

Test it

Try the test the llms.txt guidelines suggest: ask an agent questions about your product, giving it only your llms.txt as a starting point. If it can't find the answer, a stranger's agent won't either.

Then check the mechanics:

• Request a page with Accept: text/markdown and confirm the response type is text/markdown.

• Open the .md version of a few pages and look for anything missing, like tabs, code blocks or tables.

• Run Lighthouse's experimental Agentic Browsing audit. Chrome's docs say the category needs Chrome 150 or later.

A short checklist

• A Markdown version of every page

• A Link header or <link> element pointing to it

• Support for Accept: text/markdown

• An llms.txt with a summary and a curated list of links

• Self-contained pages with versions, complete examples and known limits

• An AGENTS.md in the repo

• A test with a real agent

Start with the Markdown pages. Everything else builds on them.

Further reading

• The /llms.txt file, v2

• Cloudflare: Markdown for Agents

• Google Search Central: Optimizing your website for generative AI features

• Chrome for Developers: Lighthouse llms.txt audit

• OpenAI: Agentic AI Foundation and AGENTS.md

• Mintlify: llms.txt