# How big and how long

> Sample size and duration for a product test.

- Course: [Product Analytics: From Data to Decisions](https://productovi.com/courses/product-analytics.md)
- Module: Did the change work?
- Lesson 18/32, ~31 min
- Source: https://productovi.com/courses/product-analytics/test-size

## “A week should do it”

**Alex Tutor:** The paywall test at Halves has a plan, and Maya wants your view on how long to run it.

**Maya:** The test plan for the trial-first paywall is signed. Let's run it for a week and read it next Monday.

**Alex Tutor:** A week might be enough, or it might be a coin toss. The length isn't picked; it's computed from four inputs and one fact about traffic.

**Alex Tutor:** **Baseline:** how often the primary metric happens today. Last quarter, 4.0% of first-time paywall viewers paid within 14 days. **MDE**, the minimum detectable effect: the smallest change worth finding. Our plan says +1 pp, from 4% to 5%.

**Alex Tutor:** **Significance level α = 5%**, two-sided: the risk of a false win when the paywall does nothing. **Power = 80%**: the chance of catching a real +1 pp. The theory is in [Power and sample size](/courses/statistics-for-pm-and-marketers/power-and-sample-size).

![A fine-meshed net holds a small fish, while a net with wide holes lets the same kind of fish slip through.](https://productovi.com/courses/product-analytics/inline/test-size-illustration-2.webp)

*With enough users, a small real effect is hard to miss.*

**Sofia:** And the fact about traffic?

**Alex Tutor:** How many new users enter the test each week. We'll get there.

> **Exercise:** Match each input of the size calculation to what it means for the paywall test.

## Users per variant

**Alex Tutor:** For a conversion metric, a standard two-proportion formula gives the users needed **per variant**. 1.96 is the z-value for α = 5% two-sided and 0.84 for 80% power, so the first factor is (1.96 + 0.84)² = 7.84.

$$
n = \frac{(z_{\alpha/2}+z_{\beta})^2\,\left[p_1(1-p_1)+p_2(1-p_2)\right]}{(p_2-p_1)^2}
$$

*Users per variant for comparing two conversion rates: p₁ is the baseline, p₂ = p₁ + MDE. Round up.*

**Sofia:** So p₁ = 0.04 and p₂ = 0.05. Everything else is in the formula.

**Alex Tutor:** Right. Online calculators, including [ours](/tools/ab-test-sample-size), use slightly different versions of this formula, so expect answers a few users apart. Anything close is the same answer.

> **Exercise:** How many users **per variant** does the paywall test need? Baseline 4.0%, MDE +1 pp, α = 5% two-sided, power 80%. Round up to a whole user.

## From users to weeks

**Alex Tutor:** Now the traffic. Only users who see the paywall **for the first time** enter the test, about **3,400 a week** across both variants.

**Sofia:** 13,470 ÷ 3,400 = 3.96 weeks. So 28 days, almost exactly.

**Alex Tutor:** Round up to **whole weeks** anyway: 4. Halves has a weekly rhythm. Rent is split at weekends, trips start on Fridays. A test that stops on a Wednesday over-represents some days.

![A wheel with seven spokes rolls along a road, leaving an orange mark each time it completes a full turn.](https://productovi.com/courses/product-analytics/inline/test-size-illustration-3.webp)

*Run a test for whole weeks, so every day of the week counts equally.*

**Alex Tutor:** And one more thing: the metric is “paid within 14 days”. The last user who joins on day 28 needs 14 more days before we can count them.

**Maya:** So when do I actually get the answer?

> **Exercise:** The Halves paywall test needs 6,735 users per variant (13,470 in total), and its primary metric is “paid within 14 days”. How many **calendar weeks** pass from the test's launch to the readout? Enrollment runs in whole weeks at 3,400 new users a week, and every user needs a full 14-day window.

**Alex Tutor:** Six weeks assumes 3,400 a week. Traffic dips in quiet months, so check what that does to the calendar.

> **Exercise:** In a quiet month only **2,000** first-time paywall viewers arrive per week. The Halves paywall test still needs 13,470 users and the 14-day window. How many calendar weeks from launch to readout?

## The price of a smaller effect

**Theo:** Six weeks to see +1 pp. Why not aim for +0.5 pp? Every bit of revenue counts.

![Baseline, MDE, significance and power give 6,735 users per variant; at 3,400 a week that is 4 weeks of enrollment and a readout after 6 weeks.](https://productovi.com/courses/product-analytics/diagrams/m5-l2-test-length.png)

**Sofia:** Half the effect, so twice the users?

**Alex Tutor:** More. The effect is squared in the denominator: halve the MDE and you need roughly **four times** the users. Work it out for 4.0% → 4.5%.

![A large rock is easy to see under a small lamp, while a tiny pebble beside it needs a huge floodlight.](https://productovi.com/courses/product-analytics/inline/test-size-illustration-4.webp)

*Halve the effect you want to detect and you need about four times the users.*

> **Exercise:** The Halves paywall test has a 4.0% baseline, α = 5% two-sided and 80% power. Now the MDE is **+0.5 pp** (4.0% → 4.5%). How many users **per variant**? Round up.

**Maya:** Sixteen weeks is a whole season. No.

**Alex Tutor:** That's the trade. The MDE comes from what's worth shipping and what the traffic allows, not from wishes. If the effect you care about is too small to find, test a bolder change.

**Sofia:** And the baseline? Does it matter which users it comes from?

**Alex Tutor:** A lot. Take it from the same people who'll enter the test: first-time paywall viewers, not all users. At an 8% baseline, the same +1 pp would need 12,192 per variant, nearly twice as many.

> **Exercise:** The Halves paywall test is sized at 6,735 users per variant: baseline 4.0%, MDE +1 pp, α = 5%, power 80%, a 50/50 split. Which change **reduces** the number of users it needs?

**Theo:** So a bolder paywall aiming for +1.5 pp would be quicker. How much quicker, exactly?

> **Exercise:** How many users **per variant** would the Halves paywall test need with a 4.0% baseline and an MDE of **+1.5 pp** (4.0% → 5.5%), at α = 5% two-sided and 80% power? Round up.

## Explaining it to Maya

**Maya:** Fine, but what's actually wrong with one week? We'd have 1,700 users per variant. That's a lot of people.

**Alex Tutor:** It's a lot of people and not much evidence. With 1,700 per variant, a real +1 pp would show up only about **29%** of the time. Most likely we'd read “not significant” and wrongly call the paywall useless.

**Alex Tutor:** There's a second way to cut a test short: keep checking it and stop when it looks good.

> **Exercise:** Maya checks the paywall test every morning and stops it on the first day the primary metric's interval is above zero. Her risk of a false win is still 5%.

**Alex Tutor:** Picture that week-one readout landing on Maya's desk as “not significant”.

> **Exercise:** The Halves paywall test (baseline 4.0%, MDE +1 pp) was sized at 6,735 users per variant. Suppose Maya, the product lead, ran it for one week anyway (1,700 per variant) and got “not significant”. Which reading is **right**?

**Sofia:** And if the first week happens to look great, stopping then is the peeking trap.

> **Exercise:** Maya asks: “Why can't we just run the paywall test for one week?” Answer her in 3–5 sentences, with the numbers from this lesson and at least one honest way to get an answer sooner.

**Alex Tutor:** Six weeks it is. Write the date of the readout into the plan now, so nobody is tempted to read it sooner.

- Previous lesson: [A/B test plan: hypothesis, metrics, guardrails](https://productovi.com/courses/product-analytics/test-design.md)
- Next lesson: [Who is in the test: triggers and cluster randomization](https://productovi.com/courses/product-analytics/who-is-in-the-test.md)

---
Productovi Learning. Free, no sign-up needed to read.