# A/B test plan: hypothesis, metrics, guardrails

> Write the A/B test plan before launch: a hypothesis, one primary metric, guardrail metrics and a decision rule.

- Course: [Product Analytics: From Data to Decisions](https://productovi.com/courses/product-analytics.md)
- Module: Did the change work?
- Lesson 17/32, ~27 min
- Source: https://productovi.com/courses/product-analytics/test-design

## Why an A/B test needs a plan

**Alex Tutor:** Halves is about to test a new paywall, and you'll help the product team write the A/B test plan.

**Maya:** The new Halves Plus paywall is ready: it leads with a **7-day free trial** instead of the price. I want it live on Monday for half of the users. Then we'll see if it works.

**Theo:** I'll watch trials. If more people start a trial, it works.

**Sofia:** And I'd add revenue, retention and paywall views. One dashboard with everything, and we'll see what moves.

**Alex Tutor:** That's how a test ends in an argument. Watch twenty numbers and a few will move by chance; then everyone quotes the one that suits them.

![A wall of small dials with a few needles twitching, while several people each point eagerly at a different one.](https://productovi.com/courses/product-analytics/inline/test-design-illustration-2.webp)

*Watch enough numbers and some will move by chance alone.*

**Alex Tutor:** So we decide what “works” means **before** the first user sees the new paywall. Four things on one page, the **A/B test plan**: a hypothesis, one primary metric, guardrail metrics and a **decision rule**.

> **Exercise:** Why write the plan before launch rather than after the data comes in? Select all that apply.

## How to write an A/B test hypothesis

**Alex Tutor:** A hypothesis names the change, who it affects, what should move and why. Our A/B test hypothesis: “If the paywall leads with a 7-day trial, more users who reach it will **pay within 14 days**, because trying Plus first makes paying feel less risky.”

**Sofia:** Why 14 days? The trial is only 7.

**Alex Tutor:** The first charge after a trial comes on day 8 at the earliest. Fourteen days leaves room for it, and gives every user the same window, whether they joined the test on Monday or Sunday.

**Maya:** Can the hypothesis just be “users will like the new paywall more”?

**Alex Tutor:** No data could ever prove that wrong, so it can't lose. A useful hypothesis names a number that could come out flat or worse.

> **Exercise:** Which of these is a testable hypothesis for the Halves paywall?

## Choosing one primary metric

**Alex Tutor:** The **primary metric** is the one number the decision hangs on: the share of users in the test who pay within 14 days of their first paywall view.

**Theo:** Why not trials? They move faster, and I'd see a result in days.

**Alex Tutor:** A trial is a step, not the goal. A paywall can double trials and still leave us with fewer payers if most trials quietly lapse. Trials become a **secondary metric**: we read them to explain the result, not to decide.

![Many people pass a free-entry turnstile into a shop, but only a few reach the till and leave holding a receipt.](https://productovi.com/courses/product-analytics/inline/test-design-illustration-3.webp)

*A trial is a step; the decision hangs on how many end up paying.*

**Sofia:** What if we make both of them primary, to be safe?

**Alex Tutor:** Then you've got two chances to win by luck. At a 5% false-positive rate each, two independent metrics give about 1 − 0.95² ≈ 9.8% chance that at least one “wins” by accident. More on that in [A/B testing pitfalls](/courses/statistics-for-pm-and-marketers/ab-testing-pitfalls).

> **Exercise:** Match each role in the paywall test plan to its metric.

**Theo:** Fine, trials are secondary. But revenue per user, paid conversion and trials as three primaries? That covers everything.

> **Exercise:** In the Halves paywall test, Theo, the growth marketer, also wants revenue per user as a primary metric, next to paid conversion and trials. With **three** independent primary metrics, each read at a 5% false-positive rate, what's the chance that at least one “wins” by accident? In %, one decimal.

## Guardrail metrics: what must not break

**Alex Tutor:** A **guardrail metric** is one we don't try to improve, only protect. A pushier paywall might annoy people who just want to split the rent, and they'd stop using Halves.

**Alex Tutor:** So: the share of test users who add an expense in days 8–14. We accept a drop of **at most 3 percentage points**, and we write that margin down now.

**Maya:** Why 3 and not 1?

**Alex Tutor:** It's a business call: the smallest loss in usage we'd trade for more payers. The number matters less than the timing. After the test, any drop looks “small” to someone who wants to ship.

**Sofia:** And we check the guardrail with an interval too?

**Alex Tutor:** Yes. We want its whole 95% interval for treatment − control above −3 pp. “The drop wasn't significant” isn't enough: a small test can miss a real drop.

![A car speeds along a mountain road towards a flag while a sturdy rail runs along the cliff edge.](https://productovi.com/courses/product-analytics/inline/test-design-illustration-4.webp)

*A guardrail metric isn't there to win, only to show nothing important broke.*

> **Exercise:** In the Halves paywall test, the guardrail (share of users who still add an expense in days 8–14, margin −3 pp) came out “not statistically significant”. That's enough to conclude the new paywall did no harm.

**Alex Tutor:** So the check is the lower end of the interval against the margin. Try it on two results.

> **Exercise:** Guardrail margin: −3 pp. Which result **passes** the guardrail?

## The decision rule: ship, don't ship or inconclusive

**Alex Tutor:** Last part: the decision rule, what we do for every possible result. We report the effect as **treatment − control** in percentage points, with its 95% interval, and compare it with the smallest effect worth shipping, **+1 pp**.

> **Test plan: trial-first paywall**
>
> - **Hypothesis:** leading with a 7-day trial raises the share of paywall viewers who pay within 14 days.
> - **Who is in the test:** users at their first `paywall_viewed`, split 50/50.
> - **Primary metric:** paid within 14 days of the first paywall view.
> - **Secondary:** trials started. **Guardrail:** added an expense in days 8–14, margin −3 pp.
> - **Smallest effect worth shipping:** +1 pp.
> - **Ship** if the primary's interval is entirely above 0 and the guardrail's interval is above −3 pp.
> - **Don't ship** if the guardrail's interval reaches below −3 pp, or the primary's interval includes 0 and stays below +1 pp (any effect is too small to matter).
> - **Otherwise:** inconclusive; keep the current paywall and decide on a larger test.

![Ship when the primary's interval is entirely above 0, don't ship when it includes 0 and stays below +1 pp, and call it inconclusive when it spans both.](https://productovi.com/courses/product-analytics/diagrams/m5-l1-decision-rule.png)

**Maya:** And if after a week it's clearly winning, we stop early?

**Alex Tutor:** No. We read it once, at the planned size. Checking daily and stopping on a good day inflates false wins. How big and how long is the next thing we'll work out.

**Sofia:** So “not significant” lands in “inconclusive” unless the interval also rules out +1 pp.

> **Exercise:** The Halves trial-first paywall test ends (smallest effect worth shipping: +1 pp). Primary metric, treatment − control: **+0.4 pp**, 95% interval **−0.5 to +1.3 pp**. The guardrail passed. What does the decision rule say?

**Alex Tutor:** Now the case Maya will find hardest: a clear win on payers, with the guardrail looking shaky.

> **Exercise:** The Halves trial-first paywall test ends. Primary: **+1.8 pp**, interval **+0.6 to +3.0 pp**. Guardrail: **−2.1 pp**, interval **−3.8 to −0.4 pp** (margin −3 pp). What does the decision rule say?

## Your turn

**Alex Tutor:** Theo can keep watching trials, as long as nobody votes with them. Now a different change to practise on.

**Maya:** Next quarter we want to try **invite-first onboarding**: new users are asked to invite a flatmate or partner before adding their first expense.

**Sofia:** It might grow groups, or it might scare people away before they see any value.

> **Exercise:** Write a test plan for invite-first onboarding at Halves (new users are asked to invite a flatmate or partner before adding their first expense) in 4–7 lines: a hypothesis, one primary metric with its window, one or two guardrails with a margin, and a decision rule that covers ship, don't ship and inconclusive.

**Alex Tutor:** One page, dated, before launch. After that the data can surprise us, but it can't be argued into saying what we'd like.

- Previous lesson: [Case: retention analysis by segment](https://productovi.com/courses/product-analytics/retention-by-segment.md)
- Next lesson: [How big and how long](https://productovi.com/courses/product-analytics/test-size.md)

---
Productovi Learning. Free, no sign-up needed to read.