AI Platform Optimization
llms.txt and crawler policy
The job here is specific: decide deliberately which agents may read your site, and tell them where the good material is.
What is meant by llms.txt and crawler policy?
Decide deliberately which agents may read your site, and tell them where the good material is. In practice that means a bot by bot decision separating training crawlers, retrieval crawlers and search crawlers, with the reasoning written down, and then the harder half: robots.txt rules that say what you mean, verified against server logs rather than hoped at.
Why this work exists
The work looks administrative from the outside. It decides how much of everything else actually lands.
This is one part of AI Platform Optimization, described separately because it is usually the first thing anybody needs.
The nearest piece of work to it is making the product data legible to an assistant that reads feeds rather than pages. Where that is already handled, this moves faster than the estimate; where it is not, the two get done together or neither of them holds.
What good looks like
Working well means the decisions downstream get made faster, because the argument about what is true has already been had.
The gap between that and where most accounts start is why this is scoped as llms.txt and crawler policy rather than as a line inside something larger. Deciding which assistant is worth building for and which is noise this year is the usual second step once it is in place.
When this is the right work
It is worth doing when:
- Somebody pasted a block list from a forum into robots.txt and nobody has reviewed it
- You want to allow citation but not training and are not sure that is even possible
- Server logs show agents you cannot identify taking a meaningful share of your bandwidth
If none of those is true, something else is probably the constraint, and the AI Platform Optimization overview maps the alternatives. The usual neighbours are tracking the answers month to month, because they move without warning and getting the brand’s own numbers into the sources assistants reach for.
What the work covers
Four items, each with an output:
- A bot by bot decision separating training crawlers, retrieval crawlers and search crawlers, with the reasoning written down. Boring, necessary, and the first casualty when a deadline moves.
- Robots.txt rules that say what you mean, verified against server logs rather than hoped at. One pass is rarely enough; the second is much faster.
- An llms.txt that is a useful map rather than a second sitemap. This is the piece a client can verify themselves, which matters more than it sounds.
- Asset access checking, because a rule that blocks a stylesheet directory renders your pages blank to anything that obeys it, scoped against what the business can actually act on rather than what is possible.
How it runs
Three stages. Most of the risk sits in the first one.
Where the judgement goes
What is left needs a decision rather than a fix. These get taken with whoever owns the consequence, not resolved quietly in a ticket.
Checking it held
Changes get re-checked after a fortnight, because a surprising number of them do not survive the next deployment or the next platform update.
Handing it over
Whatever is left running ends up with a named owner and a schedule, or it stops happening within two months.
What each part runs into
Taken one at a time, against real markets:
- A bot by bot decision separating training crawlers, retrieval crawlers and search crawlers, with the reasoning written down, where a Indore engagement contends with a language mix the content has to match rather than assume
- Robots.txt rules that say what you mean, verified against server logs rather than hoped at – working in London this means accounting for an enquiry path that runs through a messaging app rather than a form
- An llms.txt that is a useful map rather than a second sitemap, and in Los Angeles the constraint on it is a catchment smaller than the ambition on the website
- Asset access checking, because a rule that blocks a stylesheet directory renders your pages blank to anything that obeys it, harder in Toronto because of an incumbent whose position comes from presence rather than effort
The error worth naming
Blocking assets along with pages. It is the single most damaging robots.txt error and it is invisible until somebody renders the site as a crawler.
The damage is less in the wasted work than in the conclusion it leads to.
How it is measured
Fetches per agent in server logs, and whether the agents you intended to allow are actually arriving. Agreed up front and reported every period, with the raw data available rather than only a summary.
The definition is written down once and not quietly adjusted when it becomes inconvenient.
What you end up with
What gets handed over: a documented crawler policy, a corrected robots.txt, an llms.txt worth publishing, and a log check proving the rules behave as intended.
It is yours, in a format you can edit, whether or not the engagement continues.
How it differs by sector
By sector, the same scope produces different priorities:
- For Home and trade services, emergency intent, service-area geography, and a phone that has to be answered. and that shapes how far this can go.
- B2B services brings its own limit: a buying committee, a long cycle, and attribution that has to survive both.
- SaaS and technology, constrained by the fact that product-led companies where the buyer self-educates and the trial is the sales call.
The industry pages carry the rest of that detail. Home and trade services and B2B services are the two where this piece of work most often decides the outcome.
Common questions
Is llms.txt a standard?
A convention with growing adoption rather than a specification anybody must honour. Publishing one is cheap; expecting it to be read is optimistic.
Can we allow citation but block training?
Partly. Several providers separate their crawlers, so you can allow the retrieval agent and block the training one. It is a real choice, and it only works if you know which agent is which.
How much of our time does this take?
A few hours in the first fortnight, mostly answering questions about history that cannot be reconstructed from the data. After that, a monthly review. Worth a look before committing budget: Featured snippet or AI overview, then Core Web Vitals field data or Lighthouse lab score.
What do we actually get at the end?
A documented crawler policy, a corrected robots.txt, an llms.txt worth publishing, and a log check proving the rules behave as intended. Documented, so it survives whoever was in the room, and specific enough to act on without a follow-up conversation. For the mechanics, How do we measure SEO for a long sales cycle?. For what it costs to get wrong, Do we need separate pages for every city we serve?.
What if this is not our problem?
Then we say so. The signals on this page are the honest filter, and work sold against the wrong diagnosis wastes a quarter. Most of the hard parts here are covered in BreadcrumbList schema, with the exceptions in Call to action.
Ask about llms.txt and crawler policy
Tell us what you have tried already. That is usually the fastest route to a useful reply.
Replace the assumptions with measurements
Bring the numbers you have. We will tell you which ones are measuring the wrong thing before we suggest anything new.