Conversion
A/B test
An A/B test splits traffic between two or more variants and compares a defined outcome, with the sample size and duration calculated before the test starts so the result can be interpreted.
What is a/b test?
An A/B test splits traffic between two or more variants and compares a defined outcome, with the sample size and duration calculated before the test starts so the result can be interpreted.
What turns on it
It is the only way to know whether a change helped, and it is routinely run in ways that cannot produce a trustworthy answer.
The mechanics
Run until the pre-calculated sample is reached and for at least one full business cycle, then analysed including segment-level results and documented either way.
A frequent error
Stopping when the result looks good is the dominant failure mode. Repeated peeking at an uncontrolled threshold produces false positives at a rate that makes the programme worse than no testing.
Most disagreements about this turn out to be disagreements about the definition.
Where this comes up in the work
It is part of AI Automation and Workflows, and more specifically of Lead routing and enrichment. Across the rest of the work it shows up in:
- Paid Social and Meta Ads, where it is the difference between reading the comments because they are free message testing working and appearing to
- Conversion Rate Optimization, where it quietly limits how far testing the thank-you page because the journey does not end at submit can go
- Website Design and Development, where it sets the ceiling on deciding which animations earn their load cost
- AI Automation and Workflows, because deciding what a human has to approve and what can simply happen depends on it
What to check
Two questions that establish whether this is actually in order on a given site:
- Whether anybody has measured it, or whether it is being taken on trust
- Whether the person who set it up is still the person who understands it
Common questions
Whose job is a/b test?
Usually whoever owns Digital PR and Link Building, because keeping the research asset current enough to stay citable depends on it. In practice it falls between teams often enough that naming an owner is half the fix.
Is a/b test expensive to get right?
It depends entirely on where the problem originates. Establishing that is the first piece of work.
How urgent is a/b test?
High if it is actually broken, low if it is merely imperfect. The check tells you which.
How would we know a/b test is a problem?
By measuring it rather than by argument. Run until the pre-calculated sample is reached and for at least one full business cycle, then analysed including segment-level results and documented either way. What this leaves out is in Answer Engine Optimization for SaaS and technology, and some of it in App Store Optimization for SaaS and technology.
What is a/b test often confused with?
Stopping when the result looks good is the dominant failure mode. Repeated peeking at an uncontrolled threshold produces false positives at a rate that makes the programme worse than no testing. Agentic Commerce in Birmingham explains the signal. AI Automation and Workflows in Birmingham explains the noise around it.
Does a/b test affect rankings directly?
Not on its own, and claims that it does are usually overselling. It is the only way to know whether a change helped, and it is routinely run in ways that cannot produce a trustworthy answer. Both Faceted navigation control and Marketplace optimisation assume this page, so they are the natural next two.
Start with an honest read on the gaps
We will tell you if the answer is that you do not need us yet. That happens often enough to be worth saying out loud.