Can I A/B Test? The Traffic Reality Check

Most small sites cannot A/B test, and nobody tells them

Every A/B testing guide is written by a company with millions of visitors. Microsoft's experimentation team, Google, Booking.com and Airbnb run thousands of tests a year, and their advice is right for them: test everything, trust the data, ship what wins. A startup with 8,000 visitors a month reads the same advice, builds the same dashboard, and then cannot understand why every test ends at 87% confidence with a lift that shrinks every week. It is not doing it wrong. It is doing something that does not work at that size.

This page does the arithmetic the guides skip. At 8,000 visitors a month converting at 3%, confirming a 10% lift takes about 106,000 visitors: thirteen months of every visitor you have, with the page frozen the whole time. In four weeks the smallest lift you could confirm is about 40%. Those are not tuning problems. Ronny Kohavi, who ran experimentation at Microsoft and Airbnb, puts the practical floor for detecting single-digit lifts at a few hundred thousand users; below that, he suggests that A/B testing may be premature and user research the better spend.

We are writing this from experience. Coddy's own dashboard ran about twenty named experiments, ten of them pricing tests on single countries, each on a small slice of traffic, and reported "Keep running" whenever the confidence sat under 95%. Several were marked successful and shipped. Looking back with this calculator, some of those tests could not have detected the lifts they were credited with. The honest version of that dashboard is this page: what can this traffic answer, how long would it take, and how often would a winner be a false alarm?

What decides whether you can test

• Conversions per month, not visitors, is the number that limits you. A page with 8,000 visitors at 3% has 240 conversions a month to split between two arms. The z-test needs hundreds per arm before it says anything.
• The lift you would act on sets the bill. Be honest about it: if you would ship a change for a 3% lift, you need a test that can see 3%, and almost no small site can.
• Time is a cost, not a fix. A test that takes a quarter freezes the page for a quarter, spans seasons and campaigns, and ends in a different business than it started in.
• How often your ideas work changes what a winner means. If one idea in ten really moves the metric, then at 95% confidence and 80% power more than a third of your "significant" winners are false alarms. The base rate is not a technicality; it is most of the answer.
• Below the line there are better methods. Talk to five users, watch session recordings, ship the change and compare before and after with wide error bars, or test on a metric more visitors reach, like a click rather than a purchase.

How to run the reality check

1. 1

   Enter the monthly visitors who would see the test

   Not the whole site. If the test is on the pricing page, it is the pricing page's visitors. If it is a checkout step, it is the people who reach that step.

2. 2

   Enter the current conversion rate of that step

   The calculator shows the conversions a month this implies, which is the real constraint.

3. 3

   Name the lift you would act on

   The relative lift that would make you ship the change. Most people say 10% and mean it; be honest if yours is smaller.

4. 4

   Read the verdict and the ladder

   Go means under a month. Stretch means under a quarter, with the page frozen the whole time. Stop means the test cannot answer the question at your traffic. The ladder shows what bigger lifts would cost instead.

5. 5

   Pick a base rate and read the false positive risk

   Choose how often your ideas really work. The tile shows what share of significant results would be false alarms at that rate, which is the number to remember the next time a test comes out green.

Months to confirm a 10% lift at 95% confidence, 80% power

Two arms, every visitor entering the test. Read your row and your column.

Three sites, three verdicts

Low-traffic testing mistakes

• Running the test anyway and reading it at 85%. The confidence level you set is a promise about how often you accept being fooled. Reading below it after the fact is the same as never having set one.
• Testing tiny changes. Button colours and word swaps produce lifts of a few percent at best, which small sites cannot detect. If you must test, test something that could plausibly move the number by a third.
• Testing at the bottom of the funnel. Purchases are rare; clicks are common. A test on the click that leads to the purchase has ten times the events and can finish.
• Letting a test run for six months. Seasons, campaigns and product changes all leak into it, and the two arms end up measuring different eras.
• Trusting a winner because it was significant. With few real winners among your ideas, a significant result is more often a false alarm than the p-value suggests. The false positive risk tile puts a number on it.
• Treating "not significant" as "the idea failed". At low traffic almost nothing is significant. The idea might be fine; the test could not see it.

Reality check FAQ