Generative Engine Optimization
llms.txt and AI crawler access
One part of Generative Engine Optimization, described on its own: make sure the systems that build AI answers can reach, read and correctly attribute your content.
llms.txt and AI crawler access, defined
Make sure the systems that build AI answers can reach, read and correctly attribute your content. In practice that means auditing robots.txt, firewall and CDN rules for the retrieval user agents, in server logs, and then the harder half: writing and maintaining llms.txt and llms-full.txt with the pages that matter.
Why this work exists
Everyone agrees this matters. Far fewer put a name against it.
Within Generative Engine Optimization this is the item with the clearest cause and effect, which is why it is scoped on its own.
The nearest piece of work to it is getting the facts about pricing and coverage right in the sources models actually read. Where that is already handled, this moves faster than the estimate; where it is not, the two get done together or neither of them holds.
What good looks like
The useful version ends with a decision somebody can act on this month, not a report somebody files.
The gap between that and where most accounts start is why this is scoped as llms.txt and AI crawler access rather than as a line inside something larger. Restructuring the pages so a model can lift a correct answer out of them is the usual second step once it is in place.
When this is the right work
Three observations mean this work is due:
- A CDN bot rule is silently returning 403 to assistant crawlers
- Your content only exists after JavaScript runs
- No decision has ever been made about AI crawler access, so the default is whatever the host chose
If none of those is true, something else is probably the constraint, and the Generative Engine Optimization overview maps the alternatives. The usual neighbours are checking that the retrieval crawlers are not being turned away at the edge and earning mentions on the handful of sources assistants quote in this category.
What the work covers
The pieces of work inside it:
- Auditing robots.txt, firewall and CDN rules for the retrieval user agents, in server logs. One pass is rarely enough; the second is much faster.
- Writing and maintaining llms.txt and llms-full.txt with the pages that matter, the step where disagreements surface, which is why it goes early rather than late.
- Checking that rendered content is in the HTML rather than only in client-side JavaScript. This is where the theory meets the account and usually loses.
- Setting a deliberate policy on which crawlers are allowed and documenting the reason. Slow, unglamorous, and the reason the rest holds up.
How it runs
Three stages, in this order, for the reason each one gives.
Deciding what not to do
A list of things deliberately left alone, with the reason for each. It saves the same conversation happening four times over the following year.
The second look
A fortnight after the first pass, the same ground gets covered again with fresh eyes. The second pass finds a different class of problem from the first.
Bringing in whoever owns the system
Developers, agency partners or whoever actually controls the platform get involved before recommendations are written, not after they are rejected.
What each part runs into
Where each part tends to meet resistance:
- Auditing robots.txt, firewall and CDN rules for the retrieval user agents, in server logs, harder in Surat because of a reporting line that averages away the market in question
- Writing and maintaining llms.txt and llms-full.txt with the pages that matter – in Sharjah, complicated by the fact that a market where buyers qualify by area rather than by city
- Checking that rendered content is in the HTML rather than only in client-side JavaScript, which in New York runs into a competitive set that changes street by street
- Setting a deliberate policy on which crawlers are allowed and documenting the reason: the Austin version of this has to deal with a decision made by somebody who never visits the site
The trap
Assuming robots.txt is the whole story. Most blocking happens at the CDN or WAF layer and never appears in robots.txt at all.
The damage is less in the wasted work than in the conclusion it leads to.
How it is measured
Successful fetch rate per retrieval user agent in server logs. That is what gets reported, and it is deliberately narrow enough to move within the engagement.
Reported with its own caveats attached, rather than cleaned up for the deck.
What you end up with
What arrives at the end of it: a verified access report per crawler, a published llms.txt, and a written access policy you can defend to a board.
Written once, for the person who has to do something about it.
How it differs by sector
Three sectors where this work looks materially different:
- B2B services, constrained by the fact that a buying committee, a long cycle, and attribution that has to survive both.
- Ecommerce and retail: Margin per order decides everything, and the product feed is usually the weakest asset. so this work has to account for that before anything else.
- Home and trade services, where emergency intent, service-area geography, and a phone that has to be answered. which moves this work earlier in the plan.
The industry pages carry the rest of that detail. B2B services and Ecommerce and retail are the two where this piece of work most often decides the outcome.
Common questions
Does llms.txt actually do anything?
It is a proposed convention, not a standard anyone is required to honour, and support varies. It costs almost nothing to publish and it serves as a clean, machine-readable map of what you consider canonical, which also helps human reviewers. Treat it as useful housekeeping rather than as a ranking mechanism.
Should AI crawlers be blocked?
That is a business decision with a real trade-off: blocking protects content from training use and also removes you from the answers those systems generate. The usual sensible position is to allow the retrieval agents that drive citations and referral traffic, and to decide separately about agents used purely for training.
How long does this take?
The diagnostic part is usually two to three weeks. What follows depends on what the diagnosis finds, and we scope the second phase after the first rather than quoting both upfront.
How much of our time does this take?
A few hours in the first fortnight, mostly answering questions about history that cannot be reconstructed from the data. After that, a monthly review. Read Noindex or Robots.txt disallow next if the problem is already defined, and Performance Max or Standard Search if it is not.
What do we actually get at the end?
A verified access report per crawler, a published llms.txt, and a written access policy you can defend to a board. Documented, so it survives whoever was in the room, and specific enough to act on without a follow-up conversation. The short version is here. Should we block AI crawlers? and Should we use a website builder or WordPress? are the long ones.
Ask about llms.txt and AI crawler access
Write what you would say on a call. The reply will be the same thing we would have said.
Find out whether this is worth doing at all
Most accounts have one thing holding them back. The useful first conversation is about finding which one.