English
Type the visitors and conversions of each variant. You get the verdict in one sentence, the p-value with its working, the interval, the Bayesian chance B is better, a check that the traffic split is not broken, and the smallest lift this test could ever have seen.
Last updated
Coddy teaches you by writing real code in your browser - interactive lessons, instant feedback, and AI help when you get stuck.
Every A/B test calculator prints a p-value. Most stop there, and that is how founders end up staring at 87% confidence wondering what it means. This one answers the questions that actually decide what you do next. Is the difference bigger than chance would produce? How big could the true lift be, at worst and at best? If the test came out "not significant", could it ever have found anything at this sample size? And is the traffic split itself healthy, or are you comparing two groups that were never equal?
It runs the standard two-proportion z-test, the same one used by Optimizely's classic engine, VWO's frequentist mode and every statistics textbook, and shows each line of the arithmetic: the pooled rate, the standard error, the z-score and the tail area. Beside it sits the Bayesian answer, the probability that B's true rate is above A's, because "there is a 91% chance B is better" is the sentence most people were hoping the p-value would give them, and it is a different number.
We built it after looking at our own dashboard. Coddy has run about twenty named experiments, ten of them per-country pricing tests on small slices of traffic, and our internal tool showed a confidence bar with the words "Keep running" under 95%. That is optional stopping with a friendly face. The three tabs beside this one, sample size, the reality check and the peeking simulator, exist because a calculator that only reports significance is a calculator that helps you fool yourself faster.
Visitors are the people who were bucketed into that variant, not page views. Conversions are the ones who did the thing you are measuring: signed up, paid, clicked. Both must be counted the same way for both arms.
95% two-sided is the default and the honest choice when B could be better or worse. One-sided is only for when a worse B would be ignored in exactly the same way as no change, which is rarer than people think.
The banner says what the numbers support. The relative lift tile shows the interval: the true effect probably sits anywhere inside it, and the low end is the number to plan with, not the point estimate.
If the split is flagged, fix the test before reading anything else. If the smallest detectable lift is bigger than your observed lift, the test was underpowered: go to the sample size tab and see how many visitors it would take.
Copy result gives you the summary for a message or a document. Copy link puts the four numbers in the URL, so whoever opens it sees the same calculation.
What each tile means, in one line.
| Tile | What it is | How to read it |
|---|---|---|
| Relative lift | (rate B minus rate A) divided by rate A | The headline. The interval beside it is the range the true lift probably sits in. |
| p-value | Chance of a gap this big if A and B were identical | Under 0.05 at 95% confidence means significant. It is not the chance the result is wrong. |
| Confidence | 1 minus p, as a percentage | The number most tools print. 87% is not almost 95%; it is a coin the wrong side of the line. |
| Chance B is better | Bayesian probability B's true rate is above A's | The sentence people want. 91% means B is probably better, not that it is 91% better. |
| Smallest lift this test could see | The relative lift detectable at 80% power with these arm sizes | If your observed lift is below it, "not significant" means "could not tell". |
| Traffic split | Observed share of visitors in each arm against the plan | Flagged when the split is off by more than chance allows. Fix the test before reading results. |
A: 10,000 visitors, 500 conversions (5.00%). B: 10,000 visitors, 560 conversions (5.60%). Relative lift +12%, p = 0.058.
Not significant at 95%, and the interval runs from about minus 0.4% to plus 24%. The honest reading is "probably a small positive, could be nothing". The smallest lift this test could detect is about 17%, so a 12% effect was always going to land in the grey zone. It needs roughly twice the traffic, not another week of hoping.
A: 5,000 visitors, 100 conversions. B: 4,300 visitors, 120 conversions. Planned split 50/50.
B looks like a clear winner at +40%. The calculator refuses to say so: a 5,000 to 4,300 split has a chance of under one in a million of happening by accident at 50/50. Something is removing users from B before they are counted, often a redirect that fails for one browser, or a bot filter that treats the variant differently. Whatever it is, the two groups are not comparable and the lift is meaningless.
A: 400 visitors, 12 conversions (3.0%). B: 400 visitors, 19 conversions (4.75%). Relative lift +58%, p = 0.20.
A 58% lift sounds enormous, and the p-value is about 0.19. With 12 and 19 conversions the calculator flags low data: the numbers swing so much at this size that a single extra signup moves the rate by a quarter of a point. At this traffic the smallest lift the test could confirm in a month is over 100%. That is not a reason to run it longer; it is a reason to read the reality check tab.
