# Price testing case: price or paywall?

> Price testing in practice: read a price test and a paywall before-and-after honestly, then recommend what Halves Plus should do next.

- Course: [Product Analytics: From Data to Decisions](https://productovi.com/courses/product-analytics.md)
- Module: Are we making money on it?
- Lesson 28/32, ~43 min
- Source: https://productovi.com/courses/product-analytics/price-or-paywall

## Two ideas, one quarter

**Alex Tutor:** Next quarter's plan at Halves has room for one idea, and Maya wants you to pick it with data. Today's case is price testing.

**Maya:** Two proposals for next quarter. Theo wants Halves Plus prices up 20%. The design team wants to show the paywall at a different moment. We have room for one. Which?

![A person at a fork chooses between two paths: one leads to a big blank price tag, the other to a redesigned shop entrance.](https://productovi.com/courses/product-analytics/inline/price-or-paywall-illustration-2.webp)

*Two ideas, room for one: which has the better evidence?*

**Alex Tutor:** Let's see what evidence each one has. For price we have a real pricing experiment: in June new paywall viewers were randomly shown either today's prices or prices 20% higher.

**Alex Tutor:** The plan, written before launch: primary metric, **first-payment revenue per paywall viewer**. Guardrails: conversion to paid and trial starts. Decision rule: raise prices only if the whole 95% interval of the difference is above zero.

**Sofia:** Revenue per viewer, not conversion, because a higher price is supposed to lower conversion a bit.

**Alex Tutor:** Right. Make sure the reasoning holds before we open the data: which metric could judge a price test on its own?

> **Exercise:** In June Halves randomly showed new paywall viewers either today's Halves Plus prices or prices 20% higher. Why is **first-payment revenue per paywall viewer** the right primary metric for this price test?

**Dataset (CSV):** [Halves Plus price test, June 2026](https://productovi.com/datasets/product-analytics/price-or-paywall/halves-price-test.csv)

Randomized by user; read on 5 July, when every trial had ended.

- `arm`: control (current prices) or higher_price
- `monthly_price`: Monthly plan price shown, USD
- `yearly_price`: Yearly plan price shown, USD (paid upfront)
- `paywall_viewers`: New paywall viewers randomized into the arm, 1–21 June 2026
- `trials_started`: Viewers who started the 7-day trial
- `paid_monthly`: Viewers whose first payment was a monthly plan (by 5 July)
- `paid_yearly`: Viewers whose first payment was a yearly plan (by 5 July)

> **Exercise:** What was the primary metric, **first-payment revenue per paywall viewer**, in the **higher_price** arm? Round to cents.

## Reading the price test: effect and interval

**Alex Tutor:** Control earned \$1.2923 per viewer, the higher price \$1.3286. The effect, higher price minus control, is **+\$0.036** per viewer, +2.8%. Its 95% interval runs from **−\$0.30 to +\$0.37**.

**Sofia:** It's positive, though. Could we call it almost significant?

**Alex Tutor:** No such thing. The interval holds a 23% loss and a 29% gain. Yearly plans arrive as \$40 lumps, so revenue per viewer is noisy, and 3,000 viewers per arm can't tell a 3% effect from zero.

![A scale weighs a stream of small coins with the occasional heavy bar dropping in, its needle swinging widely.](https://productovi.com/courses/product-analytics/inline/price-or-paywall-illustration-3.webp)

*Big yearly payments make revenue per viewer noisy, so the range is wide.*

**Alex Tutor:** So the decision rule isn't met. And "not shown" isn't "no effect": the test simply lacked the power. [Confidence intervals](/courses/statistics-for-pm-and-marketers/confidence-intervals) covers the maths.

> **Exercise:** In Halves's June price test (Halves Plus prices 20% higher vs today's, about 3,000 paywall viewers per arm), the effect on first-payment revenue per viewer (higher price − control) was +\$0.036, 95% interval −\$0.30 to +\$0.37. The rule was to raise prices only if the whole interval is above zero. Which reading is right?

**Alex Tutor:** Theo will say “just let it run longer”. Here are two plans; which one would you back?

> **Exercise:** Halves's June price test (Halves Plus prices 20% higher, about 3,000 paywall viewers per arm) is inconclusive: the effect on revenue per viewer is +\$0.036, 95% interval −\$0.30 to +\$0.37. Which next step is **sound**?

## How much conversion can we lose?

**Theo:** Conversion to paid fell from 8.67% to 7.63%. That's the price doing its job, right?

**Alex Tutor:** Up to a point. With every price 20% higher, revenue per viewer holds as long as conversion stays above control's rate divided by 1.2. That's the **break-even conversion**.

**Sofia:** That assumes the same share of yearly plans in both arms.

**Alex Tutor:** Good catch. It was close: 28% of control's payers chose yearly, 27% of the higher-price arm's. Work out the break-even rate.

> **Exercise:** In Halves's June price test, the control arm converted 261 of 3,012 paywall viewers to paid. With every price 20% higher and the same plan mix, what is the **break-even conversion to paid**, below which revenue per viewer falls?

**Alex Tutor:** Conversion to paid has two steps inside it: starting a trial, then paying when it ends. Check the second one in the higher-price arm.

> **Exercise:** Use the price test table. What was **trial-to-paid conversion** in the **higher_price** arm (monthly and yearly payers together)? Every trial had ended when the test was read.

## What the test can't see yet

**Alex Tutor:** 7.63% is above the 7.22% break-even, but the conversion gap's interval, −2.42 to +0.35 points, reaches below it. And the primary metric stops at the first payment.

**Sofia:** The first monthly renewals at \$5.99 only come in July. If more of those people cancel, the lifetime gain shrinks.

![A person waits at a letterbox while a line of envelopes is still on its way along the road.](https://productovi.com/courses/product-analytics/inline/price-or-paywall-illustration-4.webp)

*The first renewals at the new price haven't arrived; the test can't see them yet.*

**Maya:** And the paywall idea? Since the redesign, paywall-to-trial went from 13.4% to 16.8%. That's a quarter more trials.

**Alex Tutor:** That's a before-and-after during trip season. Within every trigger the rate stayed flat or fell; the currency trigger grew from 12% to 39% of viewers. At April's mix, June converts at 12.7%.

> **Exercise:** Maya, the Halves product lead, must pick one idea for next quarter. A 20% Halves Plus price rise has a randomized June price test that measured first-payment revenue per paywall viewer. A paywall redesign shipped in May, and paywall-to-trial then rose from 13.4% in April to 16.8% in June, during trip season. Which statements about the evidence are correct? Select all that apply.

**Alex Tutor:** One more trap before we design anything, about what even a perfect price test would show.

> **Exercise:** Halves's June price test (Halves Plus at \$5.99 a month and \$47.99 a year instead of \$4.99 and \$39.99) measured first-payment revenue per paywall viewer. If it had shown first-payment revenue per viewer up, with the whole 95% interval above zero, Halves could be confident the higher price raises revenue over a subscriber’s first year.

## Designing a paywall A/B test

**Sofia:** So neither idea has proof. Price has a weak test, the paywall has none.

**Alex Tutor:** Then part of the answer is how to get proof. Before we write to Maya, sketch the paywall test: who's in it, one primary metric, and the rule you'd decide by.

**Theo:** Keep it short. I'd like my price rise back on the table by autumn.

> **Exercise:** Halves's design team wants to show the Halves Plus paywall at a different moment; its only evidence so far is a before-and-after comparison. Sketch an A/B test for this paywall change in 3–5 sentences: who is enrolled, the one primary metric, guardrails, and the decision rule.

## The recommendation

**Alex Tutor:** Now the memo. Maya wants a choice for the quarter, the numbers behind it, what we don't know, and the next step.

**Maya:** One page, please. I'll forward it to the CEO as it is.

> **Exercise:** Write Maya a recommendation: raise the Halves Plus price, change the paywall, or neither yet?

**Alex Tutor:** Pricing decisions are rare and expensive to undo. Read the effect with its interval, name what the test couldn't see, and never let a before-and-after chart stand in for a test.

- Previous lesson: [CAC, payback and ROI](https://productovi.com/courses/product-analytics/cac-payback-roi.md)
- Next lesson: [Forecasting active users](https://productovi.com/courses/product-analytics/forecasting.md)

---
Productovi Learning. Free, no sign-up needed to read.