# Volume, quality and Simpson's paradox

> Tell how much from how well, and catch Simpson's paradox: a per-group ratio that rose only because the mix of groups changed.

- Course: [Product Analytics: From Data to Decisions](https://productovi.com/courses/product-analytics.md)
- Module: Which numbers should we steer by?
- Lesson 6/32, ~30 min
- Source: https://productovi.com/courses/product-analytics/volume-vs-quality-metrics

## Volume vs quality metrics: six tiles and a bigger budget

**Alex Tutor:** Board week at Halves: Maya and Theo both want a read on the dashboard from you, their product teammate. One of its tiles hides a Simpson's paradox.

**Maya:** Board meeting on Thursday. I want to show which of my six dashboard tiles prove the product is improving. And Theo wants to double the ad budget for September.

**Theo:** Which will push every tile up. You're welcome.

**Alex Tutor:** Not every tile. Put one question to each number: would it still move if the product didn't change at all? Say Theo doubles ad spend, or a holiday sends half the country on trips.

**Alex Tutor:** If the answer is yes, it's a **volume metric**: how much. Users, groups, expenses, revenue totals. A **quality metric** says how well Halves works per user or per group: week-4 retention, the share of new groups that get a second member, expenses per active group.

![On one side a big crowd streams in through an open door; on the other, a small group happily splits a receipt at a table.](https://productovi.com/courses/product-analytics/inline/volume-vs-quality-metrics-illustration-2.webp)

*Volume says how many came; quality says how well the product works for each.*

**Sofia:** So volume can be bought, and quality has to be built?

**Alex Tutor:** Mostly. The board needs both; they answer different questions. More ad money brings more people, not an easier way to split a bill. A volume tile sold as proof of a better product is a **vanity metric**.

> **Exercise:** Theo doubles ad spend in September and the product doesn't change. Which of Maya's tiles would you expect to **rise from the extra spend alone**? Select all that apply.

**Alex Tutor:** Now put the test to work on a slide. Theo drafted two lines about September for the board.

> **Exercise:** Ad spend doubled in September and sign-ups doubled with it. Which line belongs on the board slide?

## Where the two meet in the tree

**Sofia:** Our north star, weekly active groups, is a count. Does that make it a volume metric?

**Alex Tutor:** It's a count built from both kinds. One branch of the tree: newly active groups = new groups × share that get a second member × share that add a shared expense in week 1.

**Alex Tutor:** The first factor is volume, Theo's territory. The other two are quality, the product team's. Last week that was 1,000 × 40% × 47.5% = 190.

**Theo:** So if I double the first factor, the branch doubles.

**Alex Tutor:** Only if the newcomers behave like today's users. Cheaper audiences rarely do.

> **Exercise:** Theo doubles new groups from 1,000 to 2,000 a week. The extra groups are less committed, so the share that gets a second member falls from 40% to **30%**; the share that adds a shared expense in week 1 stays at **47.5%**. How many **newly active groups** does the branch give?

## A quality metric that rose on its own

**Maya:** Fine, but I do have a quality number for the board: expenses per active group, **11.0** in May and **12.6** in July. The new expense screen shipped in June.

**Alex Tutor:** Then check it with the test. Something besides the product changed between May and July: summer. Here's the ratio split by group type.

**Dataset (CSV):** [Halves active groups and expenses, May and July 2026](https://productovi.com/datasets/product-analytics/volume-vs-quality-metrics/halves-mix-shift-may-july.csv)

Expenses added by people (no recurring bills, no staff), by group type.

- `month`: Calendar month
- `group_type`: flat, couple or trip
- `active_groups`: Groups with at least one expense added by a person that month
- `expenses`: Expenses added by people that month

**Sofia:** I'll rebuild July first: all July expenses over all July active groups.

**Alex Tutor:** Yes: sum `expenses` for July, sum `active_groups`, divide. If you land on Maya's number, we open the types.

> **Exercise:** Using the dataset **Halves active groups and expenses, May and July 2026**, compute **expenses per active group for July**, all group types together. Round to one decimal.

**Sofia:** 12.6, same as Maya. Now per type. Flats: 3,200 ÷ 400 = 8.0 in May, 2,888 ÷ 380 = 7.6 in July. Couples: 12.0, then 11.6. Trips: 20.0, then 19.0. Hold on.

**Maya:** Every type went down and the total went up? The export must be broken.

**Theo:** Don't look at me. My budget was flat all summer.

**Alex Tutor:** The export is fine, and so is Theo's budget. Before I explain, pick the reason yourself.

> **Exercise:** Flats, couples and trips each added fewer expenses per group in July than in May, yet the overall figure rose from 11.0 to 12.6. What explains it?

## Simpson's paradox: every type fell, the total rose

**Alex Tutor:** It's who is inside the average. In May trip groups were 100 of 800, 12.5%. In July, 340 of 1,000: 34%. A trip group adds about 20 expenses a month, a flat about 8.

**Alex Tutor:** The overall ratio weights each type by its number of groups. Summer moved weight onto the busiest type, so the average climbed while every type slipped. That's **mix shift**: a quality metric passing on a change in volume.

![Receipt piles stand beside a house, a pair of rings and a suitcase; the suitcase pile is tallest and many more suitcases roll in.](https://productovi.com/courses/product-analytics/inline/volume-vs-quality-metrics-illustration-3.webp)

*Each group type slipped, but many more trip groups pulled the average up.*

**Alex Tutor:** This pattern has a name: **Simpson's paradox**. Every segment moves one way and the total moves the other. Flats, couples and trips each added fewer expenses per group, yet the overall ratio rose from 11.0 to 12.6. Mix shift is the cause; the paradox is the symptom you see.

![Flats, couples and trips each added fewer expenses per group in July, yet the overall ratio rose from 11.0 to 12.6 because trips grew to 34% of groups.](https://productovi.com/courses/product-analytics/diagrams/m2-l2-simpson-paradox.png)

**Sofia:** So how do I show what the product did, with summer taken out?

**Alex Tutor:** Freeze the weights. Take July's rate for each type and apply it to May's number of groups of that type. Same people-mix, new behavior.

![A hand clamps the weights on a balance in place while the items in its pans are swapped for new ones.](https://productovi.com/courses/product-analytics/inline/volume-vs-quality-metrics-illustration-4.webp)

*Keep May's mix, apply July's behaviour, and the product's own change shows.*

$$
\text{July at May's mix} = \frac{\sum_{t} \text{rate}_{\text{Jul}}(t) \times \text{groups}_{\text{May}}(t)}{\sum_{t} \text{groups}_{\text{May}}(t)}
$$

*For each group type t: July's expenses per group, weighted by May's number of groups of that type.*

> **Exercise:** Freeze the mix: apply **July's** expenses per group for each type to **May's** number of groups of that type. What is expenses per active group at May's mix? Round to two decimals.

**Alex Tutor:** 10.53 against May's 11.0, about −4.3%. With the mix frozen, groups added slightly fewer expenses, not more. Freezing the mix is how you see through Simpson's paradox.

**Alex Tutor:** The freeze works in the other direction too: keep **July's** mix and ask what May's behavior would have given. Try it.

> **Exercise:** Apply **May's** expenses per group for each type (flat 8.0, couple 12.0, trip 20.0) to **July's** number of groups (380, 280, 340). What is expenses per active group at July's mix? Round to one decimal.

## Two charts for Thursday

**Maya:** So which chart goes to the board?

**Alex Tutor:** The overall line, 11.0 → 12.6, would fool them: it invites credit for the expense screen. Show expenses per group by type instead, with 10.53 at May's mix next to it.

**Alex Tutor:** Keep the volume story too, because it's true: total expenses grew from 8,800 to 12,596. Summer brought more groups. What isn't true is that each group uses Halves more.

**Alex Tutor:** A ratio also lies when its denominator shrinks: if the quietest groups drop out, expenses per group rises while the total falls. Put the counts next to every ratio.

**Alex Tutor:** Check that last trap on numbers of your own before Maya's note.

> **Exercise:** In one month Halves had 1,000 active groups and 10,000 expenses; the next month, 700 active groups and 8,400 expenses. **True or false:** expenses per active group rose from 10 to 12, so groups used Halves more.

**Maya:** Then write me the note that goes under the by-type chart.

> **Exercise:** Write the 3–5 sentence note under Maya's by-type chart for the board: did Halves get better at its job between May and July? Use numbers from the dataset **Halves active groups and expenses, May and July 2026**, and keep the volume claim and the quality claim apart.

> **Key ideas**
>
> - **Volume metrics** say how much (users, groups, expenses, revenue totals); **quality metrics** say how well the product works per user or per group (retention, second-member share, expenses per group).
> - The test: would the number still move if the product didn't change, say with doubled ad spend or a holiday? Volume metrics do; a quality metric that does is telling you about the mix.
> - Many counts combine both: newly active groups = new groups (volume) × second-member share × week-1 expense share (quality). Doubling the first factor doesn't double the branch if the quality factors drop.
> - An overall ratio is a weighted average of its segments. When the weights change it moves even if no segment does: **mix shift**. When every segment moves one way and the total the other, that's **Simpson's paradox**. Check by segment, or freeze the mix (July's rates at May's mix: 10.53 vs 11.0).
> - A ratio can also rise because its denominator shrank. Show the counts next to it, and keep the volume claim and the quality claim apart.

**Alex Tutor:** Summer grew the volume and dressed it up as quality. The product's own answer sits inside each group type, and there it was a small dip.

- Previous lesson: [The north star metric and its KPI tree](https://productovi.com/courses/product-analytics/north-star.md)
- Next lesson: [DAU, WAU, MAU and stickiness](https://productovi.com/courses/product-analytics/active-users.md)

---
Productovi Learning. Free, no sign-up needed to read.