Technical SEO
Log file analysis
Log file analysis is the examination of server access logs to see what crawlers actually requested, when, and with what response.
What is log file analysis?
Log file analysis is the examination of server access logs to see what crawlers actually requested, when, and with what response. It is the only source that shows real crawler behaviour rather than a simulation of it.
What it changes
It settles questions that nothing else can: which pages are crawled, how often, which return errors to crawlers only, and where crawl budget actually goes.
The mechanics
Logs are filtered by verified user agent, then aggregated by directory, status code and frequency, and compared against the commercial value of each section.
A frequent error
A crawler simulation is not a substitute. It shows what a crawler could do, not what the real one did, and the two routinely differ.
Most disagreements about this turn out to be disagreements about the definition.
Where this comes up in the work
It is part of AI Platform Optimization, and more specifically of Google Gemini optimization. Across the rest of the work it shows up in:
- Lifecycle and Email, which cannot be done properly without it
- Analytics and Measurement, where getting it wrong shows up in deleting the metrics that have never changed a decision
- Brand and Creative: it is the reason writing for the person who has to defend the choice internally is scoped the way it is
- Generative Engine Optimization, where it is the difference between restructuring the pages so a model can lift a correct answer out of them working and appearing to
What to check
Two questions that establish whether this is actually in order on a given site:
- Whether what the report says about it matches what a crawl of the site says
- Whether it was reviewed since the last platform change
Common questions
Who should be responsible for log file analysis?
Usually whoever owns brand and creative, because writing for the person who has to defend the choice internally depends on it. In practice it falls between teams often enough that naming an owner is half the fix.
What does fixing log file analysis involve?
The diagnosis is cheap. The fix ranges from an afternoon to a project depending on how deep it sits in the build, and we scope the second after doing the first.
Should log file analysis come before other work?
Before anything that depends on it, which is the test rather than a general ranking. It settles questions that nothing else can: which pages are crawled, how often, which return errors to crawlers only, and where crawl budget actually goes.
How would we know log file analysis is a problem?
By measuring it rather than by argument. Logs are filtered by verified user agent, then aggregated by directory, status code and frequency, and compared against the commercial value of each section. Nonprofit and NGO is written for the practitioner, Real estate and property for whoever is paying.
Is log file analysis the same as anything else?
A crawler simulation is not a substitute. It shows what a crawler could do, not what the real one did, and the two routinely differ. The distinction matters because the two are fixed differently. Two of these come up often enough to be worth reading first: App Store Optimization and Content and Editorial.
Will fixing log file analysis move rankings?
It settles questions that nothing else can: which pages are crawled, how often, which return errors to crawlers only, and where crawl budget actually goes. Whether that becomes a ranking change depends on what else is true about the site, which is why it is worth measuring rather than assuming. First-party data or Third-party data goes one level down from here, and GA4 or Matomo two.
Find out whether this is worth doing at all
One call, no sequence of follow-up emails afterwards. If the timing is wrong we will leave you to it.