Map Extract
Free tool

Did version B really win? A/B test calculator

You sent two subject lines or ran two versions of a page, and one did better. Enter the visitors (or sends) and conversions for each, and the calculator tells you whether the gap is likely to be real or could easily be chance, with the uplift, a p-value and a range for the true difference. A planner underneath works out how many visitors a test needs before you start.

Runs in your browser. Nothing you enter is uploaded.

Checking a result

  1. Enter the two versions

    Visitors or sends, and conversions, for A and for B. The examples load a subject line test, a landing page and a test that is too early to call.

  2. Choose the confidence level

    90%, 95% or 99%. 95% is the usual choice; 99% asks for stronger evidence before declaring a winner.

  3. Read the verdict

    The figures show whether one variant wins at that confidence, B's uplift over A, the p-value and both conversion rates. A bar shows the likely range of the difference; if it crosses zero, no difference has been shown.

  4. Check the warnings

    If the split between A and B is far from even, the page warns of a sample ratio mismatch, which usually means the test was set up wrongly.

  5. Plan the next test

    Under Before you start, enter your current conversion rate, the smallest uplift worth finding and daily visitors to get the visitors needed per variant, at 80% power, and the days it will take.

Example: two versions of an enquiry page

A kitchen fitter tested a new version of the quote request page against the old one for a month. Version A had 1,200 visitors and 54 enquiries (4.5%), version B 1,180 visitors and 79 enquiries (6.7%).

At 95% confidence B wins, with a 48.8% relative uplift and a p-value of 0.0198. The true difference probably lies between about 0.3 and 4.0 percentage points, so B is better but the gain could be modest. The planner showed that a test designed to detect a 20% uplift from 4.5% would need 9,118 visitors per variant, about 63 days at 300 a day.

Input
A: 1,200 visitors, 54 conversions   B: 1,180 visitors, 79 conversions   Confidence: 95%
Planner: current rate 4.5%, smallest uplift 20%, 300 visitors a day
Result
Variant B wins at 95% confidence   Uplift of B over A: +48.8%   p-value: 0.0198   Rates: 4.5% and 6.7%
The true difference most likely lies between 0.35 and 4.04 percentage points.

9,118 visitors per variant (18,236 in all) at 95% confidence and 80% power: about 63 days at 300 a day, rounded up to whole weeks.

What the numbers mean

Uplift
How much better B converted than A, relative to A. From 4.5% to 6.7% is an uplift of about 49%.
p-value
How often a gap at least this big would appear by chance if A and B were really the same. A p-value below 0.05 is significant at 95% confidence.
Confidence interval
The range the true difference in conversion rate probably falls in. If it includes zero, the test has not shown that either version is better.
Power
The chance a test detects a real difference of the size you planned for. The planner uses 80%.
Sample ratio mismatch
A visitor split much more uneven than chance would produce, usually a sign that tracking or randomisation went wrong.

Small lists and small sites

With a few hundred sends, only big differences show up as significant; the too-early-to-call example has 420 and 415 visitors, and a 30% uplift is still well within chance. That is not a reason to stop testing, but a reason to test bold changes, such as a different offer or a different opening line, rather than a single word, and to decide the sample size before you look at the results.

Questions people ask

How do I know if my A/B test is statistically significant?

Enter the visitors and conversions for both versions here. If the p-value is below 1 minus your confidence level (0.05 at 95%), and the interval for the difference does not cross zero, the result is significant: a gap that large would rarely happen by chance if the versions performed the same.

How many visitors do I need for an A/B test?

It depends on your current conversion rate, the smallest improvement you care about and the confidence you want. Lower rates and smaller improvements need far more visitors. Use the planner on this page before starting, and run the test until it reaches that number rather than stopping when it first looks good.

Can I use this for cold email subject line tests?

Yes. Use sends as visitors and opens or replies as conversions. Opens are less reliable than they used to be because some mail apps load images automatically, so replies or meetings booked make a sturdier measure, though you will need more sends to see a difference.

Why should I not stop a test as soon as B is ahead?

Checking repeatedly and stopping at the first significant result greatly increases the chance of a false winner. Decide the sample size in advance with the planner, let the test run its full course, preferably in whole weeks so every weekday is included, and then read the result once.

Where your data goes

The statistics are calculated in your browser. Visitor and conversion numbers are not uploaded or stored.

Checked against the tool on

Top