llms.txt Explained: Should Your Site Have One?

llms.txt is a plain Markdown file placed at /llms.txt on a website that gives AI agents a short summary of the site and a curated list of links to its most useful pages. Jeremy Howard proposed it on September 3, 2024, and published a revised v2 in August 2026. It’s a proposal, not a web standard. Google Search says it ignores the file, and no major AI company says its crawlers use third-party llms.txt files to decide what to cite.

So should you add one? My honest answer: it depends on who reads your site. If coding agents read your docs, yes, today. If you run a local business hoping for more AI Overview mentions, it won’t move anything, though it costs you almost nothing either.

I get asked about this file more than almost any other “AI SEO” tactic. Every week, honestly. So in this post I’ll walk through what it is, how I’d write one correctly, who has said what on the record, and the exact conditions where I’d spend 20 minutes on it and where I’d tell you to close the tab.

What Is llms.txt, in Plain Terms?

Think of it as a table of contents written for a language model. That’s it. A web page wraps its content in navigation, scripts and ads. An agent that wants the facts has to strip all of that away, and it only has so much context window to spend.

The llms.txt proposal answers that with two ideas. First, a Markdown file at the root (or at any subpath, such as /docs/llms.txt) that tells an agent what the site is and where the good material lives. Second, clean Markdown copies of important pages at the same URL with .md added, like page.html.md.

The v2 update also added a way for pages to point at their Markdown copy and their llms.txt file, using standard rel="alternate" and rel="describedby" link relations. That matters for discovery. Before v2, an agent had to guess whether a site even had the file.

Here’s the part most people miss. Howard’s own proposal says the file is mainly for inference, meaning an agent reads it on demand while helping a user. It isn’t a ranking signal for anyone, and it was never pitched as one.

What Does a Correct llms.txt File Look Like?

The format is strict. That’s deliberate. A program should be able to parse it with ordinary code, not guesswork. Per the spec, the sections come in this order:

1. An H1 with the name of the site or project. This is the only required part.
2. A blockquote with a short summary holding the key facts.
3. Optional paragraphs or lists with more detail (no headings allowed here).
4. Zero or more H2 sections, each a list of links written as name, optionally followed by a colon and a note.

An H2 called “Optional” is a convention for secondary links an agent can skip when it’s short on context. In v2 it no longer has special machine meaning, but it’s still a sensible way to separate core pages from nice-to-have ones.

Here’s an example for an invented plumbing company. Swap in your own facts and URLs.

# Example Plumbing Co

> Licensed plumbing company serving Austin, Texas since 2012. We handle emergency repairs, water heater installation and drain cleaning for homes and small businesses. We do not offer gas line work.

Service area: Austin, Round Rock and Pflugerville. Emergency calls are answered 24/7 by phone. Prices are quoted after an on-site visit.

## Services

- [Emergency plumbing](https://www.example.com/emergency-plumbing/): What counts as an emergency and response times
- [Water heater installation](https://www.example.com/water-heaters/): Tank and tankless options we install
- [Drain cleaning](https://www.example.com/drain-cleaning/): Methods we use and when a camera inspection is needed

## Company

- [About us](https://www.example.com/about/): License number, team and history
- [Contact](https://www.example.com/contact/): Phone, hours and booking form

## Optional

- [Blog](https://www.example.com/blog/): Maintenance tips and seasonal checklists

Two notes from my own drafts of these files. The blockquote carries most of the value, so put the facts an assistant gets wrong about you right there, like what you don’t offer. And the spec prefers links to Markdown versions of pages; plain HTML URLs still work, they’re just noisier for the agent.

Who Actually Reads llms.txt Today?

This is where most articles get vague. Not here: I’ll stick to what each company has put on the record, in its own documents or through its own staff, and I’ll tell you where I couldn’t find a statement at all.

Google Search: no. Google’s guide to optimizing for generative AI features, published in May 2026, lists llms.txt under things you can ignore. It says you don’t need machine readable files, AI text files, markup or Markdown to appear in Google Search, including its AI features, because Google Search itself doesn’t use them. A June 2026 note added that keeping the file for other services is fine and “will neither harm nor help” your Google visibility.

That matches what Google’s people said earlier. In April 2025, John Mueller wrote on Reddit that, as far as he knew, none of the AI services had said they use it, and compared it to the keywords meta tag. Gary Illyes said at Search Central Live in July 2025 that Google doesn’t support it.

Chrome Lighthouse: sort of. Here Google gets confusing, and I don’t blame anyone for being puzzled. Lighthouse added an llms.txt check in its agentic browsing audits. Read the details, though: a missing file is marked “not applicable”, and the check fails only when your server errors out while serving it. That’s a readiness check for browser agents, not a search signal.

OpenAI, Anthropic and Perplexity: they publish, they don’t promise. All three host llms.txt files for their own developer documentation, and their doc pages point readers to them. None of their crawler documentation that I read in September 2026 says their bots fetch your llms.txt or use it to choose sources. Their crawler pages talk about robots.txt only, and I separate what OpenAI actually documents from the folklore in getting cited in ChatGPT search.

Coding agents and tools: yes, and this is the real use. Documentation platforms such as Mintlify and GitBook generate the file automatically, and the proposal lists WordPress plugins (Yoast SEO, AIOSEO) and Wix as generators too. When a developer points an agent at a library’s docs, the file genuinely saves it work.

Is llms.txt the Same as robots.txt?

No. I’ve seen people mix them up, and it causes real mistakes. robots.txt is an access rule. It tells crawlers what they may fetch. llms.txt is a guide. It tells an agent what’s worth reading once it’s allowed in. Putting Disallow lines in llms.txt does nothing, and an llms.txt file can’t open doors robots.txt has closed.

If your goal is controlling AI crawlers, robots.txt is the tool, and each vendor publishes its own user-agent tokens. The ones below come from each company’s own crawler documentation as of September 2026. Crawler and rendering access as a whole belongs to a separate technical topic, which I cover in my guide to technical SEO for AI search, so I’ll keep this to the names.

One trap: user-triggered fetchers like ChatGPT-User and Perplexity-User are documented as not always following robots.txt rules, because a person asked for the page.

Should You Add One? The Conditions That Decide

The decision turns on four questions. Answer them honestly and, in my experience, the verdict mostly writes itself.

1. Do agents or developers read your content to get work done? API docs, SDKs, software help centers. If yes, the file has a proven audience.
2. Is your CMS already generating one? If Yoast, AIOSEO, Wix or your docs platform can switch it on, the cost drops to a checkbox and a review.
3. Do you expect a Google visibility gain? If that’s the only reason, stop. Google has told you in writing it ignores the file.
4. Will anyone keep it current? A stale llms.txt that lists retired services or dead URLs is worse than none, because an agent will trust it.

If you publish documentation, then add llms.txt and Markdown page versions now. This is the use case the proposal was built for.

If you’re a local or service business, then treat it as optional hygiene. Spend 20 minutes if your platform generates it, fix the summary, and move on to work that actually shows up in results.

If you’re choosing between llms.txt and anything else on your list, then do the other thing first. Content that answers real questions, clean indexing and accurate business info all beat a text file nobody is committed to reading.

When Is llms.txt Worth 20 Minutes, and When Is It Not?

Here’s my decision matrix. It’s the same one I walk through when a client asks whether to bother, usually right after they’ve read a LinkedIn post promising that one text file will get them cited everywhere.

The edge cases are worth a sentence each. A large ecommerce catalog shouldn’t try to list every product; link category hubs and policy pages instead. A multi-brand company can place separate files at subpaths, since v2 says a file covers the pages under its path. And if your server returns an error for /llms.txt, fix that even if you never publish one, because Lighthouse flags server errors.

What Should You Do Instead for AI Visibility?

If the real goal is showing up in AI answers, I wouldn’t start with this file. Not even close. Google’s guide says a page has to be indexed and eligible for a snippet to appear in its AI features. That’s ordinary SEO work, and it still decides most outcomes.

My take? Fix indexing, answer the real questions buyers ask, make business facts consistent everywhere, then measure. I cover the strategy side in GEO vs SEO, and how search engines understand your brand as a thing in entity SEO. If you’re wondering whether markup plays a bigger role than this file, my companion piece on schema markup for AI search covers what Google actually says about it.

And if you’d rather have someone audit the whole picture, that’s what our AI search optimization service does. You can also start with a free SEO audit.

Frequently Asked Questions

Does Google Use llms.txt?

No. Google’s AI optimization guide says Google Search doesn’t use llms.txt or other AI text files, and that having one will neither help nor harm your rankings or visibility in AI Overviews and AI Mode. Chrome’s Lighthouse checks the file for browser-agent readiness, but that’s a separate tool, not a search ranking factor.

Where Do I Put an llms.txt File?

At the root of your site, so it loads at https://yourdomain.com/llms.txt. The v2 spec also allows files at subpaths, such as /docs/llms.txt, which then cover only the pages under that path. Serve it as plain text or Markdown and check that it returns a normal 200 response.

What Is llms-full.txt?

It’s a common companion file that some documentation platforms generate, holding the full text of the docs in one Markdown file. It isn’t part of the core llms.txt spec. It makes sense for documentation sites; for a typical business site it adds maintenance with no clear reader.

Can llms.txt Block AI Crawlers?

No. It has no access rules at all. To block or allow AI crawlers, use robots.txt with each vendor’s documented user-agent token, such as GPTBot, ClaudeBot or Google-Extended. Keep in mind that user-triggered fetchers may not follow robots.txt.

Last updated: September 2026 by Mizanur Rahman