# Selection bias: users who do X retain better

> A selection bias example: correlation, self-selection, a confounding variable and what a feature's users can and can't tell you.

- Course: [Product Analytics: From Data to Decisions](https://productovi.com/courses/product-analytics.md)
- Module: What can we learn without a test?
- Lesson 23/32, ~30 min
- Source: https://productovi.com/courses/product-analytics/correlation-and-selection

## Correlation or cause: a slide that wants to be true

**Alex Tutor:** Maya has spotted a promising pattern in Halves data and brings it to you before it goes into a pitch: do users of one feature really retain better because of it?

**Maya:** Look what I found for the Plus pitch. New users who **scan a receipt** in their first week are active in week 4 at **49%**. Everyone else: **27%**. Let's push scanning in onboarding and lift retention for everyone.

**Theo:** I'd put it on the store page too: “Scanners stay longer.”

**Alex Tutor:** The numbers are right. The risky part is the jump from “people who scan retain better” to “scanning **makes** people retain”. The first is a correlation. The second is a claim about cause.

![People carrying open umbrellas walk along a rain-soaked street under grey clouds.](https://productovi.com/courses/product-analytics/inline/correlation-and-selection-illustration-2.webp)

*Umbrellas don't cause rain: seen together isn't the same as cause.*

**Alex Tutor:** Before any data, sort the two readings. Which one do Maya's two numbers support on their own?

> **Exercise:** Halves new users who scanned a receipt in their first week were 49% active in week 4, everyone else 27%. Which sentence do they **support as they stand**?

**Sofia:** What else could make scanners stay?

**Alex Tutor:** Ask who scans. Free users get three scans a month; you scan when you're the one paying for dinners and groceries. Here are April–May sign-ups who added at least one expense in week 1, split by how many they added.

**Dataset (CSV):** [Receipt scanning and week-4 retention](https://productovi.com/datasets/product-analytics/correlation-and-selection/halves-receipt-scan.csv)

New Halves users from April–May 2026 who added at least one expense in days 0–6.

- `week1_expenses`: Expenses added in days 0–6: 1, 2_to_4 or 5_plus
- `scanned_receipt_week1`: Scanned at least one receipt in days 0–6: yes or no
- `users`: Users in this row
- `active_days_22_28`: Of them, opened the app on at least one of days 22–28

> **Exercise:** Maya, the Halves product lead, wants to tell the Plus pitch that receipt scanners retain far better. Check her gap: week-4 retention of scanners minus that of non-scanners, all buckets together. Percentage points, two decimals.

## Self-selection bias: who ends up scanning

**Alex Tutor:** Now look at the rows, not the total. Where do the scanners sit?

**Sofia:** Mostly in `5_plus`: 303 of 493. Among non-scanners, `5_plus` is only 397 of 2,707.

**Alex Tutor:** So scanners were already the busiest users in week 1. That's **selection bias**, here **self-selection**: users chose to scan, and the two groups differ because of who ended up in each, not only because of the scanning.

![At a café table one friend with a thick stack of receipts scans them with a phone, while the others, holding one receipt each, chat.](https://productovi.com/courses/product-analytics/inline/correlation-and-selection-illustration-3.webp)

*Scanners were already the group's busiest payers before they scanned.*

**Sofia:** Heavy users retain better anyway, whether they scan or not.

**Alex Tutor:** Right, and the slide compares a group that's mostly heavy users with a group that's mostly light ones. Check the share yourself.

> **Exercise:** What share of **scanners** are in the `5_plus` bucket? Percent, two decimals. (Among non-scanners it's 14.67%.)

## The confounding variable: compare equally busy users

**Alex Tutor:** So compare scanners with non-scanners who were equally busy: inside each bucket. Week-1 activity is the **confounding variable** here: it drives both scanning and retention.

![Week-1 activity drives both scanning and week-4 retention; among equally busy users the scanners' 22-point lead shrinks to about +2.5 pp.](https://productovi.com/courses/product-analytics/diagrams/m6-l3-confounder.png)

**Maya:** Start with the heavy users, that's where most scanners are.

**Sofia:** Same activity, different receipts. If scanning itself mattered a lot, the gap should survive inside each bucket.

> **Exercise:** In the `5_plus` bucket, what is the week-4 retention gap, scanners minus non-scanners? Percentage points, two decimals.

**Alex Tutor:** Two to three points in every bucket, not 22. If we give scanners the non-scanners' mix of buckets, their retention is **29.9%** against 27.4%: about **+2.5 pp**.

**Alex Tutor:** Now turn it around: give the non-scanners the scanners' activity and see how much of the gap is left.

> **Exercise:** In the Halves receipt-scanning data, re-weight to remove the difference in week-1 activity: what would **non-scanners'** week-4 retention be at the **scanners' mix** of buckets (11.56% in `1`, 26.98% in `2_to_4`, 61.46% in `5_plus`), keeping the non-scanners' own rates? Percent, one decimal.

**Maya:** So scanning does nothing?

**Alex Tutor:** Not that either. Some gap remains, but we split by one thing only. Scanners may also be the group's treasurer, travel more or be on a Plus trial. Observational data can't pull those apart from scanning itself.

**Sofia:** So it can say “scanners retain better, mostly because they're busier”, but not “scanning makes people stay”.

> **Exercise:** Which statements does the receipt-scanning data support? Select all that apply.

## How to avoid selection bias: test what you'd ship

**Theo:** Then how do we ever know if pushing scanning is worth it?

**Alex Tutor:** Test the thing you'd actually ship: an onboarding prompt that suggests scanning. Randomize who gets it, and compare **everyone assigned**, scanners or not. Comparing only those who scanned would bring the selection right back.

![A hand flips a coin above two identical doors, sending each arriving newcomer through one or the other.](https://productovi.com/courses/product-analytics/inline/correlation-and-selection-illustration-4.webp)

*Randomize what you'd ship, then compare everyone assigned to it.*

**Sofia:** And the answer might be much smaller than 22 points.

**Alex Tutor:** Likely. A prompt only nudges some people to scan, so its effect on everyone assigned is smaller still. Size the test for that, not for the slide.

**Alex Tutor:** Put a rough number on “smaller still” before anyone sizes the test.

> **Exercise:** Halves plans a randomized test of an onboarding prompt that suggests scanning a receipt. Suppose the prompt moves **20%** of assigned users into scanning, and scanning truly adds **2.5 pp** of week-4 retention for them. Nobody else is affected. What effect should the test expect across **everyone assigned**? Percentage points, one decimal.

> **Exercise:** What is the best way to learn whether **promoting** receipt scanning raises week-4 retention?

**Maya:** Fine. But I still want the slide. Help me word it so it's true.

**Alex Tutor:** Say what the data shows, name the selection, and state what you'd test. That's a stronger slide than a claim someone can knock down with one question.

**Sofia:** And keep the 49% and 27%, just not the arrow between them.

> **Exercise:** Rewrite the slide claim of Maya, the Halves product lead, “Scanning receipts boosts retention by 22 points” in 2–4 sentences, so that it is true to the data and useful for the Plus pitch.

- Previous lesson: [Traffic mix and seasonality](https://productovi.com/courses/product-analytics/traffic-and-seasons.md)
- Next lesson: [Case: the version or the traffic?](https://productovi.com/courses/product-analytics/did-the-version-help.md)

---
Productovi Learning. Free, no sign-up needed to read.