Conversion
Statistical significance
Statistical significance is the probability that an observed difference is not due to chance alone, conventionally reported against a 95 percent threshold.
What is statistical significance?
Statistical significance is the probability that an observed difference is not due to chance alone, conventionally reported against a 95 percent threshold. It describes the risk of a false positive, not the size or the value of the effect.
Why anybody cares
A significant result on a trivial effect is not worth shipping, and a non-significant result on a large effect may simply mean the sample was too small.
What the work looks like
Interpreted alongside the confidence interval and the minimum detectable effect, both of which say more about what the test can and cannot tell you than the p-value does.
What it is not
Significance reached mid-test is not a result. The figure fluctuates most when samples are small, which is exactly when teams tend to look at it.
It is a small idea with a disproportionate effect on how a site performs.
Where this comes up in the work
It is part of Technical SEO, and more specifically of Crawl budget and log analysis. Across the rest of the work it shows up in:
- Google Ads Management, where it decides making sure the platform is optimising toward something the business actually earns
- Paid Social and Meta Ads, since knowing when a campaign is dying and when it is just a bad Tuesday is only as good as this is
- Conversion Rate Optimization, in which it is the first thing checked before working out which trust signals this audience actually reads
- Website Design and Development, which cannot be done properly without it
What to check
Two questions that establish whether this is actually in order on a given site:
- Whether it was reviewed since the last platform change
- Whether the thing being reported is the thing you think is being reported
Common questions
What tells you whether statistical significance is in order?
Interpreted alongside the confidence interval and the minimum detectable effect, both of which say more about what the test can and cannot tell you than the p-value does. The check is cheap; the fix is where the work sits.
Who should be responsible for statistical significance?
It needs one named owner rather than a shared responsibility. The natural home is Conversion Rate Optimization, since making the measurement trustworthy, because everything downstream rests on it is where the consequence lands.
What does fixing statistical significance involve?
The diagnosis is cheap. The fix ranges from an afternoon to a project depending on how deep it sits in the build, and we scope the second after doing the first. Field data is where this gets expensive. First-party data is where it gets slow.
Should statistical significance come before other work?
It comes before work that assumes it is already handled, and after anything actively blocking indexation.
How do you prove statistical significance is worth fixing?
A significant result on a trivial effect is not worth shipping, and a non-significant result on a large effect may simply mean the sample was too small. The evidence is specific to the site, which is why the check comes before the recommendation. Brand and Creative for SaaS and technology answers what, Content and Editorial for SaaS and technology answers how much.
Is statistical significance the same as anything else?
Significance reached mid-test is not a result. The figure fluctuates most when samples are small, which is exactly when teams tend to look at it. The distinction matters because the two are fixed differently. AI Platform Optimization in Birmingham is the page to read before the kickoff call; Analytics and Measurement in Birmingham is the one to read after it.
Take the guesswork out of the next quarter
We will tell you if the answer is that you do not need us yet. That happens often enough to be worth saying out loud.