Technical SEO

How a search engine actually processes your page

A page has to clear several hurdles before anyone sees it in a result. Each hurdle can trip it up on its own, which is what makes diagnosis interesting. Never crawled? Then it can’t be indexed. Crawled but rendered as an empty shell? Google has nothing to understand. Understood, but surrounded by five near-twins? Then it’s a coin toss which twin Google keeps, and the odds aren’t in your favour.

1. Discover. A link, sitemap entry or redirect tips the bot off.
2. Crawl. Bot asks your server for the page; HTML and assets come back.
3. Render. Scripts execute, roughly like Chrome on a phone.
4. Index. It decides if the page deserves a slot, and which version counts as canonical.
5. Rank. For each query, it sorts the indexed pages it thinks will help most.
6. Serve. It shows a blue link, a rich result or a cited source inside an AI answer.

Most SEO chatter is about ranking and serving, the last two boxes, where keywords and links and content quality get argued over. Our patch is mostly the first four. Less exciting at dinner parties, sure. But if the water stops flowing at box two, nobody cares how shiny the tap is.

Discovery: how a crawler learns your page exists

New URLs reach a crawler three ways. It follows links off pages it already trusts, it reads whatever XML sitemap you’ve submitted, and it chases redirects from retired addresses. Links win that contest easily. When an important page links somewhere, it’s vouching for the destination.

That’s why an orphan page (one nothing links to) gets visited so rarely and ranks so poorly, even if it’s sitting right there in your sitemap. A sitemap entry basically says “this exists.” A link says “this matters.” Google listens harder to the second one.

Crawling: fetching the page

Googlebot knocks; your server had better reply quickly, with an honest status code. A 200 hands the page over. A 301 points to its new home, permanently. 404? Not found. 410 is the deliberate “we deleted it.” Keep throwing 5xx errors and Google politely backs off, visiting less often until things calm down.

There’s a size cap as well, and it surprises people. Google’s own documentation says Googlebot reads the first 2 MB of a supported file like an HTML page; whatever comes after that point simply isn’t used for indexing. Most pages never get near it. Pages stuffed with giant inline scripts, enormous JSON blobs or a few thousand product tiles sometimes do, and if your best paragraph lives at the very bottom of that file, it may never be read at all.

Rendering: turning code into a page

Lots of modern sites are built mostly out of JavaScript. Google can execute it, though rendering is pricier than reading plain HTML, so it often happens in a second wave, occasionally after a lag. Plenty of other crawlers (many AI ones included) don’t run JavaScript whatsoever. So if your prices, descriptions or internal links only show up once scripts fire, some bots are staring at a blank page.

Our JavaScript rendering page digs into this properly, and the trade-offs are laid out side by side in server-side vs client-side rendering.

Indexing: deciding what to keep

A crawl isn’t a promise. Google is picky about storage. Ten URLs showing near-identical stuff usually collapse into one, and the other nine vanish from results. Thin, broken or copied pages can be fetched and then dumped. You’ll see this in Search Console as “Crawled, currently not indexed” or “Duplicate, Google chose different canonical than user.” Honestly, those two lines turn up in nearly every audit we open.

Ranking and serving

Here’s where relevance, content, links and user behaviour take over. Technical work still nudges things, though. Mobile usability, page experience and schema all affect what your listing looks like, and whether it gets bonus bits such as price and stock in product snippets. AI tools quoting sources tend to grab clean, parseable pages first. Messy ones get skipped.

The 12 areas of technical SEO, explained simply

People talk about “doing technical SEO” as if it were one job. It’s closer to twelve jobs that keep bumping into each other. For each one below you’ll get a plain definition, the mistakes we keep running into, and a picture of what healthy looks like. Where we’ve got a dedicated page, there’s a link if you want the long version.

1. Crawlability and robots.txt

Type /robots.txt after your domain and you’ll see it: a tiny robots.txt text file naming folders that bots are asked to skip. Think of it as a “staff only” sign on a door, not a lock. Googlebot respects the sign; scrapers walk straight through.

What goes wrong: The horror story everyone in SEO has heard: launch day, somebody copies the staging robots.txt across, and its single line “Disallow: /” tells every bot to stay out of the entire site. Big brands have done it. Quieter blunders are more common, like blocking the folder with your CSS and JS (so Google can’t render anything properly) or blocking a parameter that happens to power your best category pages.

What good looks like: A file you can read in thirty seconds. Junk gets blocked (site search, cart, checkout, runaway filter combos), the sitemap URL sits at the bottom, and a human eyeballs it whenever code ships.

2. Crawl budget and crawl efficiency

Crawl budget is roughly how many of your URLs a search engine can and will fetch over a stretch of time. Running a 50-page brochure site? You can skip this section, honestly. But an online shop with 200,000 products, plus filters spawning millions of URL variants, lives or dies by it.

What goes wrong: Faceted navigation, nearly always. Colour, size, brand, price, sort order: multiply those together and you get an almost endless pile of URLs showing the same shoes in a slightly different order. Meanwhile Googlebot is busy with “red, size 9, cheapest first, page 14” while yesterday’s new collection sits there unvisited.

What good looks like: Filter pages people genuinely search for (“women’s running shoes”, say) get crawled, indexed and linked like proper landing pages. The worthless combinations aren’t linked at all, or they’re tamed with parameter rules or robots.txt. Pull up the server logs and most of Googlebot’s hits land on pages that make money. Our crawl budget optimisation page goes deeper.

3. Indexation control

You decide which URLs deserve a spot in Google, rather than letting the CMS decide for you. The levers are meta robots tags, the X-Robots-Tag header (handy for PDFs), canonical tags and good old redirects.

What goes wrong: Mostly index bloat. WordPress tag archives breed. So do author pages, flimsy city pages, print versions and URLs with session IDs, until half of what Google holds for you is clutter. That drags down quality signals and spreads ranking power thin. The reverse happens as well, where a money page is still carrying a noindex tag some developer added in testing and forgot.

What good looks like: Search Console’s indexed count lands close to the number of pages you’d actually like to rank, and every one of them earns its place. Here’s how we check that on our indexation control page.

4. Site architecture and internal linking

Ever been lost in a badly signposted hospital? Bad site architecture does that to crawlers. Good structure is the opposite: reception points you to cardiology, cardiology has a sign for ward 4, done in three turns.

What goes wrong: Key pages buried six or seven clicks down. Menus built purely in JavaScript that bots struggle to follow. Category pages linking out to every product while no product ever links sideways to its cousins. Then there’s anchor text reading “click here,” which tells Google precisely nothing about where it leads.

What good looks like: Money pages are three clicks from home, tops. Related pages link to one another inside the body copy, rather than only via the footer. Hub pages pull a topic together and point down to each detailed page beneath, and breadcrumbs tell people and bots alike exactly where they’re standing.

Our site architecture service handles URL structure, hub design and the internal link plan. Torn between folders and subdomains? Have a look at subdirectory vs subdomain.

5. JavaScript rendering

Build a site in React, Vue or Angular and the server may ship an almost empty HTML file, leaving the visitor’s browser to assemble everything. That’s client-side rendering. Can it rank? Yes. Is it riskier at every single step? Also yes.

What goes wrong: Content that only appears after someone clicks or scrolls. “Links” that are really buttons with click handlers instead of proper <a href> tags, so crawlers have nowhere to go. Titles and canonicals that switch after load. And the sneaky one: missing pages answering 200 OK because the app does its own routing, which Google then labels as soft 404s.

What good looks like: Copy and links arrive baked into the server’s HTML, thanks to server-side rendering, static generation or pre-rendering. JavaScript then layers interactivity on top (developers call that hydration). Want a two-minute test? Right-click, choose View Page Source rather than Inspect, and Ctrl+F for a sentence from your main copy. No match means some crawlers can’t see it either.

6. Page speed and Core Web Vitals

We covered the three Core Web Vitals in the table above. Improving them almost never comes down to one magic switch; it’s usually a dozen small delays you shave off one by one.

Why LCP is slow, usually: a massive uncompressed hero image, that same image set to lazy-load even though it’s the first thing anyone sees, a sluggish server, or CSS and fonts that are render-blocking.

Why INP is poor, usually: bloated JavaScript clogging the main thread. Every chat bubble, heatmap tool and ad script adds weight, and a tap sets off long tasks that make the screen freeze for half a second.

Why CLS is bad, usually: ad slots and photos with no width or height set. Or a late web font that reflows every line, or that cookie banner shoved in at the top once the page has settled.

Lab versus field data confuses loads of teams. Our explainer on field data vs Lighthouse shows how a 98 in testing can still fail in Search Console. Want hands-on help? That’s our Core Web Vitals service.

7. Mobile-first indexing

Google crawls and indexes with a smartphone bot now. Put simply, your mobile site is the site Google grades. If the phone version hides copy, drops internal links or leaves out schema that the desktop version has, Google only ever sees the cut-down edition.

What goes wrong: Tabs and accordions whose content only loads via JavaScript when tapped. Separate m-dot sites that never got half the pages. Mobile menus with whole sections chopped out.

What good looks like: One responsive site. Phones and desktops get identical copy, links, metadata and structured data; just the layout shifts around.

8. Structured data

Structured data is a bit of code, usually JSON-LD, that labels what’s on a page. This is a product. Here’s its price. That’s the company behind the site, and this person wrote the article. Search engines and AI models have less guessing to do when it’s there.

A word of warning: half the schema guides online are out of date. Google restricted FAQ rich results to government and health sites in August 2023, then stopped showing them in Search altogether in May 2026. HowTo rich results went in 2023. During 2025 Google retired a fresh batch, Course Info, Claim Review, Estimated Salary, Learning Video, Special Announcement and Vehicle Listing among them. So if an agency is still promising FAQ dropdowns under your listing, ask them some pointed questions.

Still worth doing, though? Yes. Product, review, organisation, article, breadcrumb, local business and video markup still drive visible features. It also helps search engines tie your brand to a knowledge graph entity, and tidy, accurate markup makes your facts easier for machines to pick up and reuse. More on our schema implementation page and in the schema markup glossary entry.

9. International SEO and hreflang

Selling into several countries or languages? Then Google needs telling which version belongs to whom, and that’s the job of hreflang. It says, in effect, “/en-gb/ is for British readers, /en-in/ is for India, /de/ is German, and they’re all the same page in different clothes.”

What goes wrong: Hreflang breaks if you look at it funny. Tags point at URLs that redirect or 404. Return links go missing, so page A names page B and B never names A back. Codes get mangled (“en-UK” isn’t a thing; it’s “en-GB”). And IP-based auto-redirects can mean Googlebot, which mostly crawls from American addresses, never once lays eyes on your other editions.

What good looks like: Every edition names all its siblings (itself too) and there’s an x-default catching everyone else. We want the CMS spitting those tags out automatically, because hand-editing 4,000 rows in Excel never ends well. As for visitors, offer them a little country picker rather than bouncing them around by IP. Still picking a setup? Read subfolders vs country domains and our international SEO entry.

10. Status codes, redirects and HTTPS

Every URL answers with a status code, and you’d be amazed how often they’re wrong. Chain four redirects together and you waste crawl time while bleeding a little signal on each hop. Create a loop and crawling stops dead. Leave a discontinued product returning 200 with “out of stock” on it and it’ll eat crawl budget for years.

HTTPS is simply expected now. Ranking-wise it’s a featherweight signal; Chrome’s red “Not secure” label does the real damage. For us the job is boring but strict: one address per page (https, www or not, slash or not), with every other spelling bouncing there in one hop.

Redirect or canonical, which one? We wrote up 301 vs canonical for exactly that headache.

11. Site migrations

We call it a migration whenever something big changes underneath a site: a new domain, a new CMS, HTTP to HTTPS, a fresh URL scheme, a redesign that swaps the templates. It’s the moment technical SEO earns its fee, because one botched launch can erase years of organic growth inside a week.

What goes wrong: There’s no redirect map, or there’s a lazy one that dumps every old URL on the homepage (Google treats that much like a 404). Copy gets trimmed in the redesign because it “looked busy.” Internal links keep pointing at old addresses. The staging noindex sneaks onto production. And somebody, somewhere, picks Black Friday week for go-live.

What good looks like: Before anyone touches a template, we crawl the old site top to bottom. Every URL that has traffic, backlinks or rankings gets matched to its exact new twin, and the staging build gets crawled with that same tool so the two lists can be diffed. Launch lands in a sleepy week. After that we’re checking crawl errors, coverage and rankings every single day for a minimum of six weeks. Our migration planning service runs exactly that playbook.

12. Log file analysis and monitoring

Your server writes down every request it gets, Googlebot, Bingbot and AI crawler visits included. Log file analysis reads those records, which means you see what bots really did instead of what a tool estimates.

Logs answer things nothing else can. Which folders does Googlebot hammer, and which does it ignore? How many hours pass between publishing and the first crawl? What share of hits get wasted on redirects, errors and parameter junk? Has GPTBot or ClaudeBot shown up yet, and which pages did they grab?

Then comes monitoring, which never really ends. Code ships, things snap. We set alerts that fire within 24 hours if robots.txt is edited, a noindex lands on an important template, server errors jump or Core Web Vitals begin to slip.

Noindex, canonical, 301 or robots.txt: which one do you need?

We see these four mixed up constantly, and picking the wrong one can hurt. Each answers its own question. Robots.txt is about crawling. Noindex is about indexing. A canonical tag is a hint about which duplicate you’d prefer, while a redirect physically moves people and signals somewhere new.

Here’s a combo we untangle a lot: noindex plus a robots.txt block on one URL. Blocked bots can’t open the page, so they never spot the noindex, and that URL lingers in results for months. Just as messy: a canonical aimed at a page that’s noindexed, or one that redirects elsewhere. Send signals that contradict each other and Google shrugs, ignores your hints and makes its own call.

Three worked examples

Advice in the abstract only gets you so far. So here are three situations, written as composite scenarios stitched together from patterns that show up in audit after audit. Names are invented. The problems, and the fixes, are real and depressingly common.

Example 1: an online store drowning in filter pages

The situation. A fashion retailer carries about 8,000 products, yet Search Console lists more than 1.2 million discovered URLs, most marked “Discovered, currently not indexed.” New collections take three or four weeks to appear in Google.

What the audit finds. Each filter on each category page spits out a crawlable link, and colour, size, brand, material, price band plus four sort orders can all stack. Server logs show more than 70% of Googlebot’s requests going to combinations no human ever searches. Product pages, meanwhile, get crawled about once a month.

The fix. The team finds roughly 300 filter combos that match real demand, things like “black leather boots” or “linen shirts for men”, and turns them into proper landing pages with their own titles, intros and internal links. Every other combination stops being a normal anchor link. Sort parameters get blocked in robots.txt, canonicals mop up the rest, and the XML sitemap is rebuilt to list indexable URLs only.

What changes. Over the next two months the discovered-URL count drops hard, Googlebot drifts back towards products, and new collections start appearing within days rather than weeks. Those fresh filter pages begin pulling in long-tail traffic the shop never had. Our ecommerce SEO page covers this in more depth, and the faceted navigation entry explains the core idea.

Example 2: a SaaS company whose blog was invisible

The situation. A software firm rebuilds its marketing site in a JavaScript framework. It looks fantastic. Six months on, though, the blog that used to feed a steady trickle of sign-ups has lost most of its organic traffic, and nobody on the team can say why.

What the audit finds. Everything renders in the browser. Fetch the raw HTML of any post and you get a title tag, one empty div and a fat script bundle. Googlebot renders the pages eventually, slowly and not every time. Links inside the articles are buttons wired to click events rather than real links, which leaves related posts stranded from each other, and several AI crawlers see nothing whatsoever.

The fix. Developers flip the blog and the main landing pages over to server-side rendering, using an option the framework already shipped with. Body links become proper anchor tags again. Canonical and meta tags move into the server response so they exist before a single script runs.

What changes. Raw and rendered HTML finally match. Indexing recovers over the following weeks and the blog creeps back for its old terms. Bonus: Largest Contentful Paint gets quicker too, since the browser isn’t waiting on a giant script to assemble the page. Sound familiar? Have a look at SEO for SaaS and technology companies.

Example 3: a publisher’s redesign that went wrong

The situation. A regional news-and-guides site ships a redesign with new URLs, dropping dates from the paths and renaming categories. A fortnight later organic traffic is down by nearly half.

What the audit finds. Redirects exist for the top 500 articles only. The remaining 14,000 old URLs now throw 404s, and plenty of them had backlinks from other sites. Older articles still link internally to the old addresses, so the site is marching its own readers, and crawlers, into dead ends.

The fix. Every old URL gets pulled together from a pre-launch crawl, analytics, Search Console and backlink tools. From that list the team writes a rule-based redirect map pairing each old address with its new match. Internal links get rewritten in bulk inside the database to point straight at new URLs, and the handful of articles with no new home get a deliberate 410.

What changes. It isn’t instant, since Google has thousands of URLs to recrawl. But the slide stops within days, and over the next couple of months most of the lost traffic finds its way back. The moral? A redirect map isn’t a nice extra. It basically is the migration. Our migration planning service exists so you never have to learn this one the hard way.

How to prioritise technical fixes

Run any crawler over a real site and it’ll throw up 400 “issues.” Your developers have maybe four days this sprint. Most of those warnings are noise anyway, so we sort everything with two blunt questions. What will fixing it do for traffic or revenue? And how painful is it to build?

Our running order rarely changes. Blockers first, anything stopping crawling or indexing, since those cap every other gain. Template problems next, where one change mends hundreds of URLs. Fiddly page-by-page polish? That waits.

Context matters too. A missing canonical on a five-page site is trivial. The same issue on a 100,000-page store can be the biggest problem on the list. Treat tool severity labels as suggestions and always ask, “How many pages, and how much revenue, does this touch?”

Free tools you can use today

A surprising amount can be diagnosed without spending a rupee. These are the free tools we’d point a friend towards, along with what each is genuinely good at and where it’ll mislead you.

Tools report symptoms, though. Screaming Frog will happily tell you 3,000 pages share a title; it can’t tell you those 3,000 pages shouldn’t exist in the first place. That call needs a person who understands your business, and it’s usually where the real value is.

Working with your developers

Here’s the uncomfortable truth: technical SEO lives or dies in the dev backlog. An audit nobody builds is an expensive PDF. A few habits decide whether an audit ships or gathers dust.

Speak their language. “Google wants this” means nothing to an engineer. What they need is the problem, evidence they can reproduce, the exact change and a way to test it. We write tickets that drop straight into Jira, Linear or whatever board you use.

Respect the platform. Shopify forces every product under /products/ and every category under /collections/, Webflow has its quirks, and headless setups have a whole different set. So a recommendation your stack can’t take is pointless; we hunt for the next-best route instead. Haven’t picked a platform yet? Our WordPress vs Shopify breakdown might help, as might site builder vs CMS.

Fix templates, not pages. Nearly every technical issue sits in a template. Change the product template once and every product benefits, so we group findings by template and your team touches as few files as possible.

Add guard rails. The best fix is one nobody can undo by accident. A couple of automated checks in your deploy pipeline can block any release that slaps noindex on the homepage, edits robots.txt or breaks canonicals. It’s an afternoon’s work. It can save you months.

If a rebuild is coming regardless, bake SEO in from the first sprint. Retrofitting it later costs a small fortune by comparison. Our website design and development team builds with these rules baked in.