Conversion
Sample size
Sample size is the number of observations a test needs to detect an effect of a given size at a chosen confidence level.
What is sample size?
Sample size is the number of observations a test needs to detect an effect of a given size at a chosen confidence level. It is calculated from the baseline conversion rate and the minimum effect worth acting on.
Why it comes up
It determines whether a test is possible at all, and calculating it first prevents running tests that could never have concluded.
How it works in practice
Calculated before the test starts, which also fixes the duration. Low-traffic sites can still test large changes, and cannot test small ones.
The usual mistake
Running a test and seeing how it goes is not a method. Without a pre-calculated sample, there is no point at which the result becomes trustworthy.
That is the whole of it. Anything more specific depends on the site in front of you.
Where this comes up in the work
It is part of Agentic Commerce, and more specifically of Agent traffic attribution. Across the rest of the work it shows up in:
- Digital PR and Link Building: it is the reason understanding what the link profile actually looks like, including the old parts is scoped the way it is
- Google Ads Management, where it is the difference between running search on profit rather than on revenue efficiency working and appearing to
- Paid Social and Meta Ads, where it quietly limits how far running the holdout that tells you what the platform will not can go
- Conversion Rate Optimization, where it sets the ceiling on removing a reassurance that was never reassuring anyone
What to check
Two questions that establish whether this is actually in order on a given site:
- Whether there is a named owner, or whether it falls between two teams
- Whether the last audit of it predates the current site
Common questions
Does sample size affect rankings directly?
It determines whether a test is possible at all, and calculating it first prevents running tests that could never have concluded. The honest framing is that it removes a constraint rather than adding a boost.
How is sample size audited?
Calculated before the test starts, which also fixes the duration. Low-traffic sites can still test large changes, and cannot test small ones. The check is cheap; the fix is where the work sits.
Who owns sample size in a business?
Whoever is accountable for getting the pages into the Bing index, because Copilot cannot cite what it cannot find. Problems with this tend to persist because nobody is. Neither Hosted site builder or Self-hosted CMS nor Smart bidding or Manual bidding is optional once this is running at any size.
How much work is sample size?
Less than people expect to check and sometimes more than they expect to fix. Calculated before the test starts, which also fixes the duration. Most of the hard parts here are covered in Are backlinks still important?, with the exceptions in Do AI assistants send any traffic?.
Where does sample size sit in the priority order?
Before anything that depends on it, which is the test rather than a general ranking. It determines whether a test is possible at all, and calculating it first prevents running tests that could never have concluded. font-display goes through this properly, and Form abandonment shows where it stops working.
What is the evidence for sample size mattering?
The symptom is usually downstream: it determines whether a test is possible at all, and calculating it first prevents running tests that could never have concluded, showing up as something else entirely in the reporting. For the decision, Conversion Rate Optimization for SaaS and technology. For the implementation, Digital PR and Link Building for SaaS and technology.
Get the two or three things that matter
No pitch deck. A short call, a look at the data you already have, and a view on whether this is worth doing at all.