Which AI Crawlers Should You Allow, and What Happens When You Block Them
Why this decision keeps being made badly
The question arrives as a values question, usually prompted by a news story about training data, and it gets answered by pasting a block list from a forum into robots.txt. The result is frequently the opposite of what was intended: the crawler that would have cited you is blocked, the one collecting training data is not on the list, and nobody checks for a year.
Separating the bots by what they are for turns a values question into an operational one with different answers for different parts of it. A publisher may well want to block training and allow retrieval. Very few businesses, on reflection, want to block the system that decides whether they get named in an answer.
A plain explanation of the main AI crawlers, what each one is used for, and how to decide which ones get access.
The crawlers worth knowing
GPTBot gathers content for OpenAI. OAI-SearchBot supports search features. ClaudeBot and anthropic-ai serve Anthropic’s products. PerplexityBot supports Perplexity’s answers. Google-Extended controls whether your content is used for Gemini and AI training, separately from normal Google crawling.
They are not interchangeable, and they do not all obey the same conventions. Treating them as one group is how sites accidentally remove themselves from the surfaces they wanted to appear in.
Deciding access
If your content is the product you sell, blocking makes commercial sense. If your content exists to win customers for something else, blocking removes you from the answers your buyers read.
The decision can be partial. Allow retrieval of your public marketing content and block the sections you monetise, such as a paid research archive.
Check what your file actually says
Most robots files were written years ago and inherited through migrations. Read yours line by line before deciding anything, then test the result rather than assuming the syntax did what you intended.
Separating the three kinds of bot
The decision gets easier once you stop treating ‘AI crawler’ as one category. There are training crawlers, which collect content to build models; retrieval crawlers, which fetch a page because a user just asked something and may cite you in the answer; and search crawlers belonging to the same companies, which feed ordinary search indexes. Blocking the second to protest the first is the most common self-inflicted wound in this area.
A workable default for most businesses is to allow retrieval, allow search, and make a deliberate decision about training based on how much of your value is the content itself. A publisher whose revenue is the writing has a genuine case for blocking training. An agency or a manufacturer, whose content exists to be found, usually does not.
Writing the rules so they say what you mean
- List the user agents you actually intend to address, by their documented names, rather than copying a block list from a forum.
- Group them by purpose: training, retrieval, search.
- Allow retrieval and search agents in full, including your assets, or they render your pages blank.
- Decide on training agents and record the reason in a comment in the file, so the next person knows it was a choice.
- Re-read the file after any platform migration, because platforms ship their own and overwrite yours.
Blocking assets along with pages
A rule that excludes a directory of stylesheets or scripts applies to every crawler that obeys the file, and the result is a renderer that sees an unstyled, often empty page. It is the single most damaging robots.txt error and it is invisible until somebody renders the page as a crawler rather than as a visitor.
Common questions
Does robots.txt stop training?
It stops compliant crawlers from fetching. It does nothing about content already collected, about non-compliant crawlers, or about content reproduced on other sites.
Will allowing AI crawlers hurt our search rankings?
No. They are separate systems, and the search crawlers were already allowed.
What about noindex instead?
Different mechanism, different effect: noindex keeps a page out of an index but the crawler still reads it. For training access, the crawl directive is the relevant control.
Should the rules be the same across all our sites?
Only if the content has the same value. A documentation site and a paywalled publication deserve different answers.
