A/B Test Calculator
Check whether variant B really beats variant A — conversion rates, uplift, a two-proportion z-test with p-value and verdict, plus the confidence interval. Then plan how many visitors you need before you start.
Is B significantly better than A?
Variant A (control)
Variant B (challenger)
Confidence interval for the difference
—
Show your work
How many visitors do you need?
Size the test before it starts: pick the smallest lift worth detecting (MDE), and this plans the sample for 80–95% power at your α.
—
How it works
The significance test
This tool uses the two-proportion z-test with a pooled standard error — the same math behind the well-known chi-squared A/B calculators (a chi-squared test on the pooled 2×2 table gives the identical two-sided p-value).
The p-value comes from the standard normal CDF Φ, computed with the Abramowitz–Stegun 7.1.26 approximation (error below 1.5×10−7).
Worked example
A: 100 conversions / 1,000 visitors = 10.00%. B: 130 / 1,000 = 13.00%.
Pooled p̄ = 230 / 2,000 = 11.50%
SE = √(0.1150 × 0.8850 × (1/1000 + 1/1000)) = 0.01427
z = (0.1300 − 0.1000) / 0.01427 = 2.103
p (two-sided) = 0.0355 → significant at α = 0.05, not significant at α = 0.01
95% CI of the difference: +0.21 to +5.79 percentage points
In plain words: B converts about 30% relatively better — roughly 300 extra conversions per 10,000 visitors.
The sample-size formula
where δ = |p2 − p1| is the MDE, p̄ = (p1 + p2)/2, and zβ is the normal quantile of the power (0.84 for 80%). Example: baseline 10%, +20% relative MDE (target 12%), α = 0.05, power 80% → 3,841 visitors per variant (7,682 total).
z-score quick reference
| z | Two-sided p | One-sided p | Usual meaning |
|---|---|---|---|
| 1.645 | 0.10 | 0.05 | 90% confidence |
| 1.960 | 0.05 | 0.025 | 95% confidence (α = 0.05) |
| 2.576 | 0.01 | 0.005 | 99% confidence (α = 0.01) |
| 3.291 | 0.001 | 0.0005 | 99.9% confidence |
z values rounded to 3 decimals; p-values are the exact normal-tail areas.
Assumptions (read before trusting a result)
- Each visitor appears in exactly one variant, assigned at random.
- Outcomes are independent — one visitor's conversion does not influence another's.
- Samples are large enough for the normal approximation (rule of thumb: at least a few dozen conversions per variant).
- Nothing changes mid-test: same traffic mix, same pages, same tracking.
Why early stopping lies to you
A p-value is only honest at the pre-planned sample size. If you check daily and stop the first time p < 0.05, you are running many tests, not one — the true false-positive rate can climb far above 5%. Plan the sample size first, run to completion, then check once.
Frequently asked questions
What does an A/B test calculator do?
It compares the conversion rates of two variants, A and B. You enter visitors and conversions for each, and it returns each conversion rate, the absolute and relative uplift, a z-test p-value, a significant-or-not verdict at your chosen alpha, and a confidence interval for the difference.
How is statistical significance calculated here?
With a two-proportion z-test. Both variants are pooled into one overall rate p-bar = (xA + xB) / (nA + nB), the standard error is SE = sqrt(p-bar(1 - p-bar)(1/nA + 1/nB)), and z = (pB - pA) / SE. The p-value is the two-tailed area under the standard normal curve beyond |z|.
What p-value counts as significant?
Compare it with your significance level alpha, chosen before the test. At alpha = 0.05, a p-value below 0.05 is significant (95% confidence); at alpha = 0.01 you need p below 0.01 (99% confidence). A smaller p-value is stronger evidence against no difference. It is not the probability that B is better.
Should I use a one-sided or two-sided test?
Two-sided is the safe default: it detects B beating A or A beating B. A one-sided test (B is better) has more power in that direction, but it is only valid if you committed to the direction before seeing any data and would genuinely ignore a win by A.
What is minimum detectable effect (MDE)?
The smallest lift worth detecting, fixed before the test. Example: a 10% baseline conversion rate with a 20% relative MDE means you size the test to reliably catch a move from 10% to 12%. Smaller MDEs need much larger samples.
How many visitors does my A/B test need?
It depends on the baseline rate, the MDE, alpha, and power. As a rule of thumb, a 10% baseline detecting a 20% relative lift at 95% confidence and 80% power needs about 3,841 visitors per variant. Use the sample-size planner tab with your own numbers.
Why should I not check results every day (peeking)?
Every peek is another chance for a false positive. Checking daily and stopping the first time p dips below 0.05 can push the real false-positive rate far above 5%. Fix the sample size in advance with the planner, run the test to completion, then look at the result once.
What happens when a variant has zero conversions?
The test still runs. If both variants are at zero, the z-score is 0 and the p-value is 1, correctly saying there is no evidence of any difference. If only one variant is at zero, the math still works as long as the other variant has data, though tiny counts deserve skepticism and a larger sample.
What is the difference between absolute and relative uplift?
Absolute uplift is the difference in percentage points: 13% minus 10% is 3 points. Relative uplift divides by the baseline: 3 points on a 10% baseline is a 30% relative lift. Relative lift sounds bigger, so report both and nobody is misled.
Can I use this for email subject lines or signup buttons?
Yes. Any binary outcome works: opened or not, clicked or not, signed up or not. Enter recipients or visitors as the visitor count and the desired action as conversions for each variant.
What assumptions does the z-test make?
Each visitor is counted once, outcomes are independent, traffic is randomly split between A and B, and samples are large enough for the normal approximation. Do not reuse the same visitors across both variants, and do not change the page halfway through the test.
How does this compare to Evan Miller's A/B testing tools?
The significance math is equivalent: his chi-squared test on the pooled 2x2 table and this z-test produce the same two-sided p-value. He reports Wilson confidence intervals per variant; this tool reports a confidence interval for the difference B minus A. His sample-size formula uses the baseline rate in the null standard error while this tool uses the pooled average rate, so the two answers typically differ by a few percent (larger at very low baseline rates).
Free tool by NRTechStudio