# How to calculate sample size for an A/B test: formula, example and calculator

> How to calculate sample size for an A/B test: the formula, a worked example, how statistical power and MDE change the number, and a free calculator.

- Source: https://productovi.com/blog/ab-test-sample-size
- By: Sergey Bruh
- Published: 2026-09-27
- Topics: Sample size, Statistical power, A/B test design

Here is how to calculate sample size for an A/B test: take four inputs, the **baseline rate** of your metric, the **minimum detectable effect (MDE)**, the **significance level** (usually 5%) and the **statistical power** (usually 80%), and plug them into the two-proportion formula below. The result is the number of users you need per variant; divide the total by your daily traffic to get the test duration. For a 4% conversion rate and a 10% relative lift, that is **39,473 users per group**. If you'd rather skip the arithmetic, our free [A/B test sample size calculator](/tools/ab-test-sample-size) does it in your browser.

## How to calculate sample size in five steps

CatChow, an online pet-food shop, has redesigned its checkout page and wants to test it against the old one. Planning the A/B test sample size takes five steps:

1. **Find the baseline.** The current conversion rate in the control group, from the last 4–8 weeks of data. For CatChow's checkout it is 4%.
2. **Choose the MDE.** The smallest lift worth shipping. CatChow picks a 10% relative lift, from 4.0% to 4.4%.
3. **Set the significance level and power.** The defaults are $\alpha = 0.05$ (two-sided) and 80% power.
4. **Plug the numbers into the formula** (or a calculator) to get users per group.
5. **Convert users into days** with the traffic that actually enters the test, and round up to whole weeks.

The next two sections do steps 4 and 5 by hand, so you can see what a calculator is doing for you.

![Four inputs (baseline rate, MDE, significance level, power) give 39,473 users per group; at 3,000 visitors a day that rounds up to a four-week test](/blog/ab-test-sample-size/diagram.png)

## The sample size formula for conversion rates

For comparing two proportions, the number of users you need **in each group** is approximately:

$$n = \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \cdot \big(p_1(1-p_1) + p_2(1-p_2)\big)}{(p_2 - p_1)^2}$$

Here $p_1$ is the baseline rate, $p_2$ is the rate you hope to see in the variant, and the $z$ values come from the normal distribution: $z_{0.975} \approx 1.96$ for a two-sided 5% test and $z_{0.80} \approx 0.84$ for 80% power.

You don't need to memorise it. Any decent online calculator or a line of Python does this. But it pays to understand what is in it: the difference $p_2 - p_1$ sits in the denominator **squared**. That single fact explains most of the surprises below.

## Worked example: CatChow's checkout redesign

Inputs:

- baseline conversion $p_1 = 0.04$;
- MDE: 10% relative lift, so $p_2 = 0.044$;
- $\alpha = 0.05$ two-sided, power 80%.

Step by step:

1. $(1.96 + 0.84)^2 \approx 7.85$.
2. Variance terms: $0.04 \times 0.96 = 0.0384$ and $0.044 \times 0.956 = 0.0421$; together $0.0805$.
3. Squared difference: $(0.044 - 0.04)^2 = 0.004^2 = 0.000016$.
4. $n \approx 7.85 \times 0.0805 / 0.000016 \approx 39{,}500$.

The exact figure is **39,473 users per group**, or about 79,000 in total.

CatChow's checkout sees about 3,000 visitors a day, split 50/50. So the test needs $78{,}946 / 3{,}000 \approx 26.3$ days. Round **up to whole weeks**: run it for **4 weeks** (28 days), so that every weekday is represented equally. Shopping on a Sunday evening looks nothing like shopping on a Tuesday morning. More on this in [how long to run an A/B test](/blog/how-long-to-run-an-ab-test).

A quick sanity check: a handy rule of thumb for 80% power and $\alpha = 0.05$ is $n \approx 16 \cdot p(1-p) / \delta^2$, where $\delta$ is the absolute difference. Here that gives $16 \times 0.0384 / 0.000016 = 38{,}400$, close to the exact result. If your calculator gives a number ten times bigger or smaller, you probably mixed up relative and absolute MDE.

## Use an A/B test sample size calculator

Once you understand the formula, let a tool do the arithmetic. The free [A/B test sample size calculator](/tools/ab-test-sample-size) takes your baseline and MDE (relative or in percentage points), with $\alpha$ and power under the advanced settings, and returns users per variant, the total sample and, if you enter daily traffic, how many days the test needs. Nothing you type leaves your browser. If your traffic is fixed and you want to know the smallest lift it can detect, run the question backwards with the [MDE calculator](/tools/mde-calculator).

## Choosing the inputs

### Baseline rate

The current value of the metric in the control group. Take it from recent data, not from a good month you remember fondly. The baseline matters because rare events are harder to measure: detecting a change in a 2% conversion rate takes far more users than detecting the same relative change in a 10% rate.

### Minimum detectable effect (MDE)

The smallest effect that would be worth acting on. It is a **business decision**, not a statistical one. Ask: "If the redesign lifts conversion by X, would we ship it and would it pay for itself?" The post on the [minimum detectable effect](/blog/minimum-detectable-effect) covers how to pick it.

State the MDE clearly as relative or absolute. A 10% **relative** lift on a 4% baseline means going from 4.0% to 4.4%, which is an **absolute** change of 0.4 percentage points. Mixing these two up is the most common source of wildly wrong sample sizes.

### Significance level ($\alpha$)

The false-positive rate you accept: the chance of declaring a winner when there is no real difference. The convention is $\alpha = 0.05$, tested two-sided.

## What is statistical power?

Statistical power ($1 - \beta$) is the probability that the test detects the effect if it really is as large as your MDE. The convention is 80%, meaning you accept a 20% chance of missing a real effect of that size. For decisions that are expensive to get wrong, teams use 90%.

Power is the input people most often leave at the default without thinking. An underpowered test doesn't just "miss sometimes": when it comes back "not significant", you can't tell whether the change did nothing or the test was too small to see it. Higher power costs users: for CatChow's 10% relative MDE, moving from 80% to 90% power raises the sample from 39,473 to **52,842 per group**, about a third more.

## How the inputs move the number

Keeping the 4% baseline, $\alpha = 0.05$ and 80% power, and changing only the MDE:

| Relative MDE | Target rate | Users per group | Days at 3,000 visitors/day |
| ------------ | ----------- | --------------- | -------------------------- |
| 5%           | 4.2%        | 154,302         | 103                        |
| 10%          | 4.4%        | 39,473          | 27                         |
| 20%          | 4.8%        | 10,314          | 7                          |
| 30%          | 5.2%        | 4,780           | 4                          |

Halving the MDE roughly **quadruples** the sample. That is the squared difference at work.

Other levers, all for the 10% relative MDE:

- **Power 90%** instead of 80%: 52,842 per group (about a third more).
- **$\alpha = 0.01$** instead of 0.05: 58,734 per group.
- **Baseline 2%** instead of 4%: 80,679 per group. **Baseline 10%**: 14,749 per group.

## Sample size for average order value and other continuous metrics

For a metric like average order value (AOV) the logic is the same, but the variance comes from the data's standard deviation $\sigma$ instead of $p(1-p)$:

$$n = \frac{2 \, (z_{1-\alpha/2} + z_{1-\beta})^2 \, \sigma^2}{\delta^2}$$

If CatChow's AOV is about 40 dollars with a standard deviation of 25 dollars and you want to detect a 2-dollar change, then $n = 2 \times 7.85 \times 625 / 4 \approx 2{,}453$ users per group. Be careful: revenue data is usually skewed, with a few very large orders inflating $\sigma$. Estimate the standard deviation from real recent orders, and consider capping extreme values before you plan.

## Why not just run the test for a week?

At CatChow's Monday stand-up, someone suggests skipping all this: "Let's show the new checkout to half the visitors for a week and look at the numbers on Friday."

It sounds reasonable, and it is how many teams burn a sprint. A week of traffic might be enough to see a huge effect, but checkout changes rarely produce huge effects. If the redesign lifts conversion by a modest 5–10%, a one-week test will most likely come back "not significant". The team then concludes the redesign "doesn't work" and moves on, when in fact the test simply never had the resolution to see the effect.

Sample size planning answers one question before launch: **how much data do we need so that a real effect of the size we care about has a good chance of showing up?**

## Common mistakes

**Picking the MDE to fit the traffic.** Working backwards ("we have two weeks, so the MDE is whatever two weeks can detect") is fine as long as you say it out loud: "This test can only catch a lift of 20% or more." If nobody believes the change could do that, don't run the test, or pick a more sensitive metric.

**Peeking and stopping early.** Checking the dashboard every day and stopping the moment $p < 0.05$ inflates the false-positive rate far above 5%. The sample size is a commitment: decide it up front and analyse once, or use a method designed for continuous monitoring ([sequential testing](/blog/ab-test-peeking)).

**Stopping mid-week.** Even if you hit the target on day 23, finish the full week. Otherwise the result over-represents some days of the week.

**Randomising by one unit and counting another.** If you randomise users but count sessions, the observations aren't independent and the real sample is smaller than it looks. Plan the sample in the same unit you randomise.

**Forgetting about multiple variants and metrics.** Three variants against control, or ten "primary" metrics, mean more chances for a false positive. Either adjust $\alpha$ (for example, a [Bonferroni correction](/blog/abn-test-multiple-comparisons)) and plan for a larger sample, or pick one primary metric in advance.

**Ignoring a sample ratio mismatch.** If you planned a 50/50 split and got 52/48 on large numbers, something in the assignment is broken. Fix that before trusting any result.

## Key takeaways

- To calculate A/B test sample size you need four inputs: baseline, MDE, $\alpha$ and statistical power.
- The MDE is a business choice: the smallest lift worth shipping.
- Sample size grows with the square of the precision you want: halve the MDE, and you need about four times the users.
- Convert users to days with your real traffic and round up to whole weeks.
- Decide the sample size before launch, and don't stop early just because the p-value dipped below 0.05.

## FAQ

### What is a good sample size for an A/B test?

There is no universal number. It depends on your baseline rate and the smallest effect you care about. With a 4% conversion rate, detecting a 10% relative lift at 80% power needs about 39,500 users per group, while detecting a 30% lift needs under 5,000. Calculate it for your own inputs.

### How long should an A/B test run?

Divide the total required sample by the daily traffic entering the test, then round up to full weeks. Most product tests should run at least one full week, and two is a safer minimum when behaviour differs between weekdays and weekends.

### What if I don't have enough traffic?

Test bolder changes (a larger MDE), use a metric higher up the funnel with a higher baseline (for example, add-to-cart instead of purchase), reduce variance by capping outliers, or accept that this decision will be made on judgement rather than a test. Running an underpowered test and reading its result as "no effect" is the worst option.

### Should I use 80% or 90% power?

80% is the common default. Use 90% when missing a real effect would be costly, for example when a "no effect" result would kill a strategic project. Expect about a third more users for the same MDE.

## Learn it hands-on

Sample size planning clicks once you have done it a few times with real numbers. The free lesson [Power and sample size](/courses/statistics-for-pm-and-marketers/power-and-sample-size/theory) walks through CatChow's checkout test step by step, with quizzes and a practice task checked by AI. It is part of the free course _Statistics for Product Managers and Marketers_.

## Related lessons

- [Power analysis and sample size](https://productovi.com/courses/statistics-for-pm-and-marketers/power-and-sample-size.md)
- [Conversion rates and the two-proportion z-test](https://productovi.com/courses/statistics-for-pm-and-marketers/conversion-rate-tests.md)
- [Pitfalls: peeking, multiple testing, novelty effect](https://productovi.com/courses/statistics-for-pm-and-marketers/ab-testing-pitfalls.md)