How to calculate sample size for an A/B test: the formula, a worked example, how statistical power and MDE change the number, and a free calculator.
By Sergey BruhPublished 10 min read
A free course for PMs and marketers: short lessons, real product data and exercises with instant feedback.
Start the free course10 min read
Peeking in A/B testing turned a 5% false-positive rate into 27.6% in 100,000 simulated tests. When to stop an A/B test, and how sequential testing helps.
9 min read
How long to run an A/B test: divide the required sample by eligible daily traffic, round up to whole weeks, and plan for novelty effects and sales.
9 min read
What minimum detectable effect (MDE) means, relative vs absolute, and how to set it from business value and your traffic. With a worked example.
Here is how to calculate sample size for an A/B test: take four inputs, the baseline rate of your metric, the minimum detectable effect (MDE), the significance level (usually 5%) and the statistical power (usually 80%), and plug them into the two-proportion formula below. The result is the number of users you need per variant; divide the total by your daily traffic to get the test duration. For a 4% conversion rate and a 10% relative lift, that is 39,473 users per group. If you'd rather skip the arithmetic, our free A/B test sample size calculator does it in your browser.
CatChow, an online pet-food shop, has redesigned its checkout page and wants to test it against the old one. Planning the A/B test sample size takes five steps:
The next two sections do steps 4 and 5 by hand, so you can see what a calculator is doing for you.
For comparing two proportions, the number of users you need in each group is approximately:
Here is the baseline rate, is the rate you hope to see in the variant, and the values come from the normal distribution: for a two-sided 5% test and for 80% power.
You don't need to memorise it. Any decent online calculator or a line of Python does this. But it pays to understand what is in it: the difference sits in the denominator squared. That single fact explains most of the surprises below.
Inputs:
Step by step:
The exact figure is 39,473 users per group, or about 79,000 in total.
CatChow's checkout sees about 3,000 visitors a day, split 50/50. So the test needs days. Round up to whole weeks: run it for 4 weeks (28 days), so that every weekday is represented equally. Shopping on a Sunday evening looks nothing like shopping on a Tuesday morning. More on this in how long to run an A/B test.
A quick sanity check: a handy rule of thumb for 80% power and is , where is the absolute difference. Here that gives , close to the exact result. If your calculator gives a number ten times bigger or smaller, you probably mixed up relative and absolute MDE.
Once you understand the formula, let a tool do the arithmetic. The free A/B test sample size calculator takes your baseline and MDE (relative or in percentage points), with and power under the advanced settings, and returns users per variant, the total sample and, if you enter daily traffic, how many days the test needs. Nothing you type leaves your browser. If your traffic is fixed and you want to know the smallest lift it can detect, run the question backwards with the MDE calculator.
The current value of the metric in the control group. Take it from recent data, not from a good month you remember fondly. The baseline matters because rare events are harder to measure: detecting a change in a 2% conversion rate takes far more users than detecting the same relative change in a 10% rate.
The smallest effect that would be worth acting on. It is a business decision, not a statistical one. Ask: "If the redesign lifts conversion by X, would we ship it and would it pay for itself?" The post on the minimum detectable effect covers how to pick it.
State the MDE clearly as relative or absolute. A 10% relative lift on a 4% baseline means going from 4.0% to 4.4%, which is an absolute change of 0.4 percentage points. Mixing these two up is the most common source of wildly wrong sample sizes.
The false-positive rate you accept: the chance of declaring a winner when there is no real difference. The convention is , tested two-sided.
Statistical power () is the probability that the test detects the effect if it really is as large as your MDE. The convention is 80%, meaning you accept a 20% chance of missing a real effect of that size. For decisions that are expensive to get wrong, teams use 90%.
Power is the input people most often leave at the default without thinking. An underpowered test doesn't just "miss sometimes": when it comes back "not significant", you can't tell whether the change did nothing or the test was too small to see it. Higher power costs users: for CatChow's 10% relative MDE, moving from 80% to 90% power raises the sample from 39,473 to 52,842 per group, about a third more.
Keeping the 4% baseline, and 80% power, and changing only the MDE:
Halving the MDE roughly quadruples the sample. That is the squared difference at work.
Other levers, all for the 10% relative MDE:
For a metric like average order value (AOV) the logic is the same, but the variance comes from the data's standard deviation instead of :
If CatChow's AOV is about 40 dollars with a standard deviation of 25 dollars and you want to detect a 2-dollar change, then users per group. Be careful: revenue data is usually skewed, with a few very large orders inflating . Estimate the standard deviation from real recent orders, and consider capping extreme values before you plan.
At CatChow's Monday stand-up, someone suggests skipping all this: "Let's show the new checkout to half the visitors for a week and look at the numbers on Friday."
It sounds reasonable, and it is how many teams burn a sprint. A week of traffic might be enough to see a huge effect, but checkout changes rarely produce huge effects. If the redesign lifts conversion by a modest 5–10%, a one-week test will most likely come back "not significant". The team then concludes the redesign "doesn't work" and moves on, when in fact the test simply never had the resolution to see the effect.
Sample size planning answers one question before launch: how much data do we need so that a real effect of the size we care about has a good chance of showing up?
Picking the MDE to fit the traffic. Working backwards ("we have two weeks, so the MDE is whatever two weeks can detect") is fine as long as you say it out loud: "This test can only catch a lift of 20% or more." If nobody believes the change could do that, don't run the test, or pick a more sensitive metric.
Peeking and stopping early. Checking the dashboard every day and stopping the moment inflates the false-positive rate far above 5%. The sample size is a commitment: decide it up front and analyse once, or use a method designed for continuous monitoring (sequential testing).
Stopping mid-week. Even if you hit the target on day 23, finish the full week. Otherwise the result over-represents some days of the week.
Randomising by one unit and counting another. If you randomise users but count sessions, the observations aren't independent and the real sample is smaller than it looks. Plan the sample in the same unit you randomise.
Forgetting about multiple variants and metrics. Three variants against control, or ten "primary" metrics, mean more chances for a false positive. Either adjust (for example, a Bonferroni correction) and plan for a larger sample, or pick one primary metric in advance.
Ignoring a sample ratio mismatch. If you planned a 50/50 split and got 52/48 on large numbers, something in the assignment is broken. Fix that before trusting any result.
There is no universal number. It depends on your baseline rate and the smallest effect you care about. With a 4% conversion rate, detecting a 10% relative lift at 80% power needs about 39,500 users per group, while detecting a 30% lift needs under 5,000. Calculate it for your own inputs.
Divide the total required sample by the daily traffic entering the test, then round up to full weeks. Most product tests should run at least one full week, and two is a safer minimum when behaviour differs between weekdays and weekends.
Test bolder changes (a larger MDE), use a metric higher up the funnel with a higher baseline (for example, add-to-cart instead of purchase), reduce variance by capping outliers, or accept that this decision will be made on judgement rather than a test. Running an underpowered test and reading its result as "no effect" is the worst option.
80% is the common default. Use 90% when missing a real effect would be costly, for example when a "no effect" result would kill a strategic project. Expect about a third more users for the same MDE.
Sample size planning clicks once you have done it a few times with real numbers. The free lesson Power and sample size walks through CatChow's checkout test step by step, with quizzes and a practice task checked by AI. It is part of the free course Statistics for Product Managers and Marketers.
