llms.txt: What It Actually Does, and What It Does Not
Why the file exists at all
Every few years a convention appears that promises to let a publisher talk directly to the machines reading their site, and llms.txt is the current one. The idea is sound and the expectations around it are not: it is a plain text file at a known location that describes what a site is and where its canonical answers live, offered to anything that cares to read it.
Nothing obliges a system to read it, and no ranking depends on it. That makes it a cheap, honest piece of documentation rather than a lever, and the reason to publish one is the same reason to keep an accurate about page: it costs twenty minutes and it removes ambiguity about who you are.
llms.txt is a proposed file at the root of a site that points language models at the content you consider canonical, in a form that is cheap to read. It has attracted a lot of attention and a fair amount of overclaiming.
What the file is
A markdown file listing the pages that best explain what you do, with a line of context each. It is a curation signal rather than a directive. Think of it as a table of contents written for a machine that has limited budget to explore.
What it is not
- It is not a ranking factor, and nobody credible has claimed it is
- It is not access control. That is robots.txt, and llms.txt does not override it
- It is not a replacement for structured data, which describes entities rather than curating pages
- It is not universally supported, and support is still changing
Is it worth adding
Yes, on the grounds that it costs an hour and carries no downside. Write it honestly, point at the pages that genuinely explain your business, and keep it current. Treat any promise of ranking gains from it with suspicion.
The work that actually moves AI visibility sits elsewhere: readable HTML, answer-first structure, a resolvable brand entity and credible sources. llms.txt is a small helpful addition to that, not a substitute for it.
What belongs in the file, and what does not
A useful llms.txt is a map, not a manifesto. It says what the site is, who publishes it, and which pages carry the canonical answer on each subject, in plain prose with absolute URLs. Twenty well-chosen lines beat two hundred, because the point is to reduce the work of working out what matters here, and a list of every page does the opposite.
What does not belong: licensing demands dressed as instructions, keyword lists, a copy of your sitemap, or anything you would not be comfortable seeing quoted. The file is public, it is not a contract, and nothing in it is enforced by anybody.
A file worth publishing
- One line saying what the organisation does and where it operates.
- A short list of the pages that hold the canonical answer on each core subject, as full URLs.
- A line naming who writes and reviews the content, linking to the editorial policy.
- Contact details, so a model has something accurate to repeat.
- A last-updated date, and an actual habit of updating it.
Treating it as a ranking file
The most common error is writing llms.txt as though it were a submission to a search engine, stuffed with phrases somebody hopes to be matched on. No retrieval system uses it that way. Treat it as documentation for a reader in a hurry and it earns its place; treat it as a ranking lever and it is forty minutes you will not get back.
Common questions
Is llms.txt a standard?
It is a convention with growing adoption rather than a specification anybody is obliged to honour. Publishing one is cheap and harmless; expecting it to be read is optimistic.
Does it replace robots.txt?
No. robots.txt controls access and is honoured by well-behaved crawlers. llms.txt offers orientation to something that has already been allowed in.
Where should it live?
At the root of the domain, served as plain text. On this site it is at the equivalent root path, generated from the same content as the sitemap so the two cannot drift apart.
Will it stop our content being trained on?
No. Training access is governed by crawler directives, licensing and law, none of which this file touches.
