English
How many visitors each variant needs before the test can say anything, and how many days that is at your traffic. Set the smallest lift worth detecting, the confidence and the power, and read the count with the formula underneath.
Last updated
Coddy teaches you by writing real code in your browser - interactive lessons, instant feedback, and AI help when you get stuck.
A sample size is a promise you make to yourself before the test starts: "I will look at the result when this many people have seen it, and not before." Without it, every A/B test drifts into the same story. The first week looks promising, the second week the lift shrinks, someone asks whether it is significant yet, and the test ends the day the bar turns green. This page gives you the number to write in the plan, so the end date is a fact rather than a mood.
The calculation is the standard two-proportion sample size, the one behind Evan Miller's calculator and Optimizely's, and it depends on four things: the baseline rate, the lift you want to be able to detect, the confidence level, and the power. Power is the one people skip. At 80% power, a real lift of the size you named is caught eight times in ten; a test with half the sample has far less than half the power, which is why an underpowered test is not "a rougher answer" but usually no answer at all.
The curve under the result is the part worth remembering. The visitors needed grow with the square of one over the lift: halve the lift you want to see and you need about four times the traffic. A site that can confirm a 30% lift in two weeks needs about nine times as long to confirm a 10% one. If the number of days comes out in months, that is the calculator telling you to test something bigger, or to move up the funnel to a metric more people reach, before it is telling you to wait.
Use the rate of the page or step the test will change, measured over the same kind of visitors the test will see. Do not use a site-wide average when you are testing one funnel step.
Ask what lift would make you ship the change. That is your minimum detectable effect. 10% relative is a common choice for a mature page; a redesign might justify 20% or more.
95% confidence and 80% power are the conventions. Raise confidence when a false winner is expensive to roll back; raise power when a missed winner is expensive to leave on the table.
All variants together, counting only the visitors who will reach the test. The result turns into days and whole weeks.
The calculator gives you the sample size per variant. Put it in the test plan with the date it implies, and do not read results before it. The peeking simulator tab shows why.
Visitors per variant at 95% confidence and 80% power, two-sided. Multiply by two for a two-arm test.
| Baseline rate | +5% lift | +10% lift | +20% lift | +50% lift |
|---|---|---|---|---|
| 1% | about 637,000 | about 163,000 | about 42,700 | about 7,800 |
| 3% | about 208,000 | about 53,200 | about 13,900 | about 2,500 |
| 5% | about 122,000 | about 31,200 | about 8,200 | about 1,500 |
| 10% | about 57,800 | about 14,800 | about 3,800 | about 690 |
| 20% | about 25,600 | about 6,500 | about 1,700 | about 290 |
Baseline 5%, minimum detectable effect 10% (target 5.5%), 95% confidence, 80% power, two-sided. Result: about 31,000 visitors per variant, 62,000 in total.
At 1,000 visitors a day that is 62 days, so nine whole weeks. Many teams would call that too long and either test a bolder change or accept that this page needs a lift of 20% or more to be testable in a fortnight.
Baseline 3%, minimum detectable effect 20% (target 3.6%). Result: about 13,900 visitors per variant.
Lower baselines need more visitors for the same relative lift, because there are fewer conversions to count. A 20% relative lift here is only 0.6 percentage points, and the test has to tell 3.0% from 3.6%.
Baseline 5%, lift 10%, four variants including the control. The alpha per comparison becomes 0.05 / 3, and the per-variant sample grows by about a third; the total is four times the per-variant figure.
Adding variants is the fastest way to turn a two-week test into a two-month one. Test one strong idea against the control; run the next idea afterwards.
