Technical SEO
Crawl budget and log analysis
This is the narrowest useful unit of Technical SEO: find out what search engines actually spend their crawl on, which on large sites is rarely your important pages.
Crawl budget and log analysis, defined
Find out what search engines actually spend their crawl on, which on large sites is rarely your important pages. In practice that means server log analysis by user agent, status code and directory, and then the harder half: identifying the parameter, facet and pagination traps consuming the crawl.
Why this work exists
It is the sort of work that never makes a slide, and shows up in every result that does.
Within Technical SEO this is the item with the clearest cause and effect, which is why it is scoped on its own.
The nearest piece of work to it is cutting the long tasks that make a loaded page unresponsive. Where that is already handled, this moves faster than the estimate; where it is not, the two get done together or neither of them holds.
What good looks like
The test is whether somebody outside the team would reach the same conclusion from the same evidence. If they would not, this is not finished.
The gap between that and where most accounts start is why this is scoped as crawl budget and log analysis rather than as a line inside something larger. Ending the contradiction between canonical and noindex signals is the usual second step once it is in place.
When this is the right work
The tells:
- Important pages take weeks to be indexed after publication
- The site has far more indexed URLs than real pages
- Nobody has looked at a server log
If none of those is true, something else is probably the constraint, and the Technical SEO overview maps the alternatives. The usual neighbours are keeping redirects to single hops rather than chains and building a quality gate so thin pages never go live indexable.
What the work covers
What is in scope:
- Server log analysis by user agent, status code and directory, scoped against what the business can actually act on rather than what is possible.
- Identifying the parameter, facet and pagination traps consuming the crawl, revisited rather than finished, because the inputs keep moving.
- Measuring crawl allocation against commercial value by section, done before anything downstream is committed to.
- Fixing the rules so the crawl reaches pages that matter and stops reaching those that do not. The output here is what everything after it is built from.
How it runs
Three stages. Most of the risk sits in the first one.
Testing on the awkward cases
Whatever gets built is checked against the messy real examples rather than the clean ones, because the clean ones were never going to be the problem.
Writing it down for the next person
The reasoning gets recorded alongside the result, so a decision taken in March can still be defended in September by somebody who was not there.
Deciding what not to do
A list of things deliberately left alone, with the reason for each. It saves the same conversation happening four times over the following year.
What each part runs into
What each part actually contends with:
- Server log analysis by user agent, status code and directory – working in Noida this means accounting for a buying pattern that resolves faster than the reporting cycle
- Identifying the parameter, facet and pagination traps consuming the crawl, and in Chennai the constraint on it is a sector mix that pulls the brief in several directions
- Measuring crawl allocation against commercial value by section, harder in Chandigarh because of a seasonal peak that arrives before the content is indexed
- Fixing the rules so the crawl reaches pages that matter and stops reaching those that do not – in Manchester, complicated by the fact that a reporting line that averages away the market in question
The avoidable version of this
Guessing at crawl behaviour from a crawler simulation. Only the logs show what actually happened.
It is worth stating because it is the version of this work that gets sold most often, and it fails in a way that looks like the discipline not working rather than the execution being wrong.
How it is measured
Share of crawl requests hitting commercially valuable URLs, and time from publication to indexing. That is what gets reported, and it is deliberately narrow enough to move within the engagement.
It is worth tracking alongside one commercial number, so a technical win that sells nothing is visible as such.
What you end up with
Delivered as a log-based crawl report showing where budget goes by section, with the specific rules changed to redirect it.
Scoped so the first action can be taken the same week it arrives.
How it differs by sector
By sector, the same scope produces different priorities:
- Healthcare and clinics: the thing that changes the scope here is that content that affects health decisions is held to a higher standard, and advertising is restricted.
- For Education and training, admission cycles, a parent paying for a student’s decision, and course-level demand. and that shapes how far this can go.
- Automotive brings its own limit: Local inventory, model-level demand, and a buyer who arrives having already decided.
The industry pages carry the rest of that detail. Healthcare and clinics and Education and training are the two where this piece of work most often decides the outcome.
Common questions
How do you know whether crawl budget is a real problem?
By comparing indexed URL count against the number of pages that should exist, and by looking at how long a new page takes to be indexed. On a site under a few thousand pages it is rarely the constraint. Above that, and particularly with facets or parameters, it usually is, and the logs settle it either way.
What wastes crawl budget most often?
Faceted navigation that generates combinations without limit, session or tracking parameters creating duplicate URLs, calendar and pagination that goes on forever, and internal links to redirects and 404s. These are mechanical problems with mechanical fixes, and they are usually larger than people expect.
Does this work at our size?
The diagnosis does, at any size. Whether the work that follows is worth the money depends on what an improvement in share of crawl requests hitting commercially valuable URLs, and time from publication to indexing is actually worth to the business, which is the first thing to settle.
Can this be scoped on its own?
Yes, where the diagnosis says it is the constraint. More often it runs inside a Technical SEO engagement, because the adjacent work is what makes the result hold. The scoping conversation settles which. Both London and Los Angeles assume this page, so they are the natural next two.
How long does this take?
The diagnostic part is usually two to three weeks. What follows depends on what the diagnosis finds, and we scope the second phase after the first rather than quoting both upfront. Finance and fintech is the one people underestimate. Healthcare and clinics is the one they over-invest in.
Ask about crawl budget and log analysis
Whatever context you have is enough to start. The first reply is a read on the problem, not a proposal.
Ask us the question you cannot get answered
We will tell you if the answer is that you do not need us yet. That happens often enough to be worth saying out loud.