Technical SEO
robots.txt
robots.
What is robots.txt?
Robots.txt is a file at a site’s root that tells compliant crawlers which paths they may fetch. It controls crawling, not indexing, and it is advisory: well-behaved agents honour it and others ignore it.
Why it comes up
It is the first place to check when pages are not being crawled, and the first place a misconfiguration can make an entire site invisible.
In practice
Written as user-agent blocks with allow and disallow rules, conventionally with a sitemap reference. Separate blocks let different agents, including AI crawlers, be treated differently.
The misconception
It does not keep pages out of the index and it does not keep content private. A disallowed URL can still appear in results, and the file itself is publicly readable.
The term is used loosely in the industry, so it is worth pinning down before agreeing to work.
Where this comes up in the work
It is part of Answer Engine Optimization, and more specifically of FAQ and Q and A schema systems. Across the rest of the work it shows up in:
- Analytics and Measurement, because working out why two reports disagree about the same number depends on it
- Brand and Creative, in which it shows up as measuring branded search instead of reporting impressions
- Generative Engine Optimization, where it decides earning mentions on the handful of sources assistants quote in this category
- Answer Engine Optimization, since finding where one strong page would beat the eleven thin ones it replaces is only as good as this is
What to check
Two questions that establish whether this is actually in order on a given site:
- Whether it is actually true on the site rather than true in the documentation
- Whether anybody has measured it, or whether it is being taken on trust
More in Technical SEO
Common questions
Is robots.txt expensive to get right?
It depends entirely on where the problem originates. Establishing that is the first piece of work.
How urgent is robots.txt?
It comes before work that assumes it is already handled, and after anything actively blocking indexation.
What is the evidence for robots.txt mattering?
The symptom is usually downstream: it is the first place to check when pages are not being crawled, and the first place a misconfiguration can make an entire site invisible, showing up as something else entirely in the reporting. Read Manufacturing and industrial next if the problem is already defined, and Marketplaces and platforms if it is not.
What is robots.txt often confused with?
It does not keep pages out of the index and it does not keep content private. A disallowed URL can still appear in results, and the file itself is publicly readable. The cheapest version of this is in Analytics and Measurement. The one that lasts is in Answer Engine Optimization.
Does robots.txt affect rankings directly?
It is the first place to check when pages are not being crawled, and the first place a misconfiguration can make an entire site invisible. Whether that becomes a ranking change depends on what else is true about the site, which is why it is worth measuring rather than assuming. Two pages carry the detail this one summarises: Featured snippet or AI overview, and then Core Web Vitals field data or Lighthouse lab score.
How is robots.txt audited?
Written as user-agent blocks with allow and disallow rules, conventionally with a sitemap reference. Separate blocks let different agents, including AI crawlers, be treated differently. Anybody with access can establish it in an afternoon. The specifics live in Is programmatic SEO safe?. Is SEO still worth it in 2026? is the context that makes them make sense.
Start with an honest read on the gaps
We will tell you if the answer is that you do not need us yet. That happens often enough to be worth saying out loud.