Do you need an llms.txt file?

You may have heard your site needs an llms.txt file to show up in AI answers. It's a small job, so it's tempting to just do it and move on.

Here's what the platforms have actually published, where the file makes sense, and what I'd do first. If you want to know where I'm coming from, my background is here.

What is an llms.txt file, and what is it meant to do?

The idea comes from Jeremy Howard, who published the llms.txt proposal in September 2024. It's a Markdown file at the root of your site that gives an AI agent a short, curated starting point: a title, a one-paragraph summary, then lists of links to the pages that matter.

The proposal's pitch is about context windows. A sitemap lists everything, which is often far too much for a model to read. An llms.txt file lists the few pages worth reading first, and the agent follows links only when it needs more.

What the file actually contains

According to the proposal, the format is strict enough to parse:

1. A single H1 with the site or project name (the only required part).
2. A blockquote summary.
3. Optional plain paragraphs or lists with extra context.
4. H2 sections holding link lists, each link with a name, URL and optional note.
5. An optional section for secondary links an agent can skip when space is tight.

It also suggests serving a Markdown version of each page by adding .md to the URL. That second part is more work than the file itself, and it's the part most WordPress plugins skip.

What it isn't

It isn't an access control. The proposal says it complements robots.txt, which governs access, and sitemap.xml, which lists pages. A file that lists a page doesn't stop a crawler fetching it, and a file that omits a page doesn't hide it.

What have Google, OpenAI and Anthropic actually said?

This is the question that matters, and the answer is thinner than the hype. I read each company's own documentation rather than someone's summary of it.

Google

Google's page on AI features and your website says AI Overviews and AI Mode have no requirements beyond normal Search eligibility: the page has to be indexed and eligible to show with a snippet. It also says you don't need to create new machine-readable files, AI-specific text files or special markup to appear. The page doesn't mention llms.txt at all.

Google's John Mueller has been more direct in public. In June 2025 he wrote on Bluesky that no AI system currently uses llms.txt, and he added that server logs show consumer chatbots fetching pages for training and grounding without ever requesting the file, as Search Engine Roundtable reported. That's one Google employee's observation from 2025, not a policy document, so weigh it accordingly.

OpenAI

OpenAI's bots documentation lists four user agents: OAI-SearchBot for ChatGPT search, GPTBot for training, ChatGPT-User for visits triggered by a user's request, and OAI-AdsBot for checking ads landing pages. You control them through robots.txt, and each setting is independent. You can allow search and block training. llms.txt appears on that page only as a link to OpenAI's own developer documentation index, not as something site owners should publish to influence ChatGPT.

Anthropic

Anthropic's help article on how its crawler works names ClaudeBot for training, Claude-User for user-directed visits and Claude-SearchBot for search. It says they follow standard robots.txt rules. It doesn't mention llms.txt.

A silence isn't proof of non-use. A platform could read the file and not say so. But if you're being asked to spend money on something, the burden is on the people selling it to show it works.

Where does llms.txt make sense, and where doesn't it?

My view is that llms.txt is a reasonable idea for one kind of site and a distraction for most others.

Sites where it's worth an hour

The proposal's own adoption list is mostly documentation. The page says documentation platforms generate the file automatically and that OpenAI, Anthropic and Google Gemini publish one for their developer docs. That fits the use case: a coding agent already on a docs site needs to find the right reference page without loading the whole thing. Mueller himself saw a narrow use of that kind, where the file helps an agent that's already on a site get around it.

If you run documentation, an API reference or a large knowledge base, a generated file costs almost nothing.

Sites where it's mostly noise

A service business with 20 to 60 pages has no context-window problem. Your sitemap is already short, and the pages worth reading are the ones in your main navigation. An llms.txt file for that site restates your menu in Markdown.

There's also a trust problem. Mueller has argued that it's safer for a system to rely on what's on the page than on a separate file that could say anything. A file you control, only for machines, is the easiest place to make claims nobody checks. That's a reason platforms may never weight it.

• Worth it: documentation sites, developer tools, large knowledge bases, anything an agent navigates to complete a task.
• Optional: mid-sized content sites, if your CMS can generate it automatically.
• Skip for now: local service sites and small business sites, until a platform documents a use for it.

What should you do instead of writing llms.txt?

The things platforms have documented are boring, and they're the things that decide whether you can be cited at all.

1. Check that the right crawlers can get in

Because OpenAI and Anthropic separate their search, user and training bots, a single blanket rule in robots.txt can block you from AI search when you only meant to opt out of training. Read your robots.txt and your CDN or firewall settings, and decide on purpose which bots you want. OpenAI notes that robots.txt changes for search can take about 24 hours to take effect, so test, wait, and test again.

2. Make sure the page is indexed and eligible for a snippet

Google's requirement for AI features is ordinary Search eligibility. If a page is noindexed, blocked or showing no snippet, it's out of the running before any AI file matters.

3. Put the answer in text, on the page

Keep important content in plain text rather than inside images or scripts, and make any structured data match what a visitor can see. That's Google's own advice on the same page.

4. Be realistic about the prize

In a study of 76 Australian sites, Uppercut found the median site got 0.64% of its sessions from AI assistants, measured in GA4 between August and October 2026, with the middle half of sites between 0.28% and 1.12%. That doesn't mean AI visibility is worthless. It means a file that isn't documented to do anything shouldn't come before the work that moves the other 99%.

For the practical side of getting named in answers, I've covered the full process in how to get a business cited in AI answers, so I won't repeat it here.

The times in that table are my rough guesses, not measurements. Your site's size will change them.

How would you test whether llms.txt does anything for your site?

If you do publish one, you can at least learn something from it. Most of the confident claims online come from people who never looked.

Read your own server logs

Mueller's point was that chatbots don't request the file. You can check that for your own site. Ask your host for access logs, filter for requests to /llms.txt, and look at the user agents. If OAI-SearchBot, ClaudeBot and the rest never request it, you have your answer for your domain.

Don't credit the file for a citation you'd have got anyway

AI answers change between runs, so a single before-and-after check proves nothing. If you add the file and a citation appears the next week, you can't separate the cause from the noise. The honest position is that nobody outside the platforms can show a lift from this file yet, and I'd treat any agency that claims one without a method with suspicion.

Where I come down

I'm happy to see a file like this on a docs site. I wouldn't put it on a client's priority list, and I'd say so in the first meeting. If you want a second opinion on where your own site's AI visibility work should start, that's the kind of conversation I have as an SEO consultant in Sydney, and you can get in touch to talk it through.