# Course recap: choosing the right test

> A decision guide that ties the whole course together.

- Course: [Statistics for Product Managers and Marketers](https://productovi.com/courses/statistics-for-pm-and-marketers.md)
- Module: Modelling and segmentation
- Lesson 24/24, ~40 min
- Source: https://productovi.com/courses/statistics-for-pm-and-marketers/choosing-the-right-test

## One map, six modules

**Alex Tutor:** Last lesson. After a whole course of answering data questions for CatChow's product team, it's time to look back.

**Sofia:** Alex, I've learned twelve different methods over five modules and I can't remember which one goes where anymore.

**Alex Tutor:** That's exactly what today is for. Every method we covered answers one of a handful of questions. Once you know the question, the method mostly picks itself.

![A cat at the start of a branching trail on a large map, each path ending at a different small tool icon, one path highlighted.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/choosing-the-right-test-illustration-2.webp)

*Know the question first, and the fork in the path usually points to the method.*

**Sofia:** So it's not really twelve separate things to memorise?

**Alex Tutor:** Two things matter most: how many groups or variables you're comparing, and whether your data meets a parametric test's assumptions — roughly normal, no extreme outliers, decent sample size.

> **The two questions behind every test**
>
> 1) What am I comparing — one variable's typical value, two groups, three or more groups, or a relationship between variables? 2) Do my data meet parametric assumptions, or do I need a rank-based, non-parametric alternative?

> **Exercise:** CatChow compares time-on-page for 15 users in variant A vs 15 in variant B. The data are heavily skewed with a couple of extreme outliers. Which test fits best?

## Two groups and beyond

**Alex Tutor:** For two independent groups: t-test if the data are roughly normal with a decent sample size, Mann-Whitney if they're skewed, small, or full of outliers.

**Sofia:** And for the same users measured before and after a change?

**Alex Tutor:** That's paired data — paired t-test or its non-parametric twin, the Wilcoxon signed-rank test. Same normal-vs-skewed logic, just for differences instead of raw values.

**Sofia:** Three or more groups — that's where ANOVA comes back?

**Alex Tutor:** Right: one-way ANOVA (parametric) or Kruskal-Wallis (non-parametric) for independent groups; repeated-measures ANOVA or Friedman for the same subjects measured repeatedly.

> **Exercise:** CatChow measures the same 40 users' satisfaction score before and after a redesigned onboarding flow. Which test is appropriate?

## A/B tests: proportions and intervals

**Alex Tutor:** For a single A/B test on a conversion rate, you're comparing two proportions, not two means — that calls for a two-proportion z-test or a chi-square test.

**Sofia:** And if I'm comparing conversion across 3 landing page variants at once?

**Alex Tutor:** Same family: a chi-square test on the 3-by-2 table of variant versus converted/not-converted, not ANOVA — ANOVA compares means of a continuous variable, and conversion is a proportion.

**Sofia:** So 'is it a mean or a proportion' is another question I should be asking upfront?

**Alex Tutor:** Exactly. And whatever the test, wrap the result in a confidence interval, and plan the sample size before launch using baseline, MDE and power.

**Sofia:** Give me a quick number to check I've got the CI formula right.

$$
\text{margin of error} \approx 1.96 \times SE
$$

*The 95% margin of error: about 1.96 standard errors on each side of the estimate.*

**Alex Tutor:** Here's a small one to compute by hand.

> **Exercise:** CatChow's conversion rate estimate has a standard error of 1.2 percentage points. Using margin of error ≈ 1.96 × SE, what is the approximate 95% margin of error?

**Alex Tutor:** Now turn that margin into an interval. Say the estimate itself is 12.0%.

> **Exercise:** The estimated conversion rate is 12.0% with a standard error of 1.2 percentage points, so the 95% margin of error is about 2.35 points. What is the upper bound of the 95% confidence interval?

## Relationships, models and segments

**Alex Tutor:** When you want to know how two variables move together, that's correlation. When you want to predict one from another, that's regression.

**Sofia:** And for a yes/no outcome, like 'will this user churn'?

**Alex Tutor:** Logistic regression: it predicts the probability of that yes/no outcome from your predictors.

**Sofia:** Different from linear regression because the outcome is bounded between 0 and 1?

**Alex Tutor:** Exactly that. Try applying it to a concrete case.

> **Exercise:** You want to predict the probability that a specific customer churns next month from their order history. Which method fits?

**Alex Tutor:** When you have no target variable at all, just customers you want to group, that's this module: clustering.

**Sofia:** And if it's survey questions I want to group instead of customers?

**Alex Tutor:** Factor analysis — same spirit as clustering, but grouping correlated variables into hidden themes instead of grouping observations into segments.

**Sofia:** So clustering needs a decision about k, unless I use the hierarchical version?

> **Exercise:** If you don't know in advance how many customer segments exist, hierarchical clustering is more appropriate than k-means.

## The full map

**Alex Tutor:** Let's put it all on one map. I'll give you several scenarios, and you match each to its method.

> **CatChow's method map**
>
> Two groups, normal data -> t-test. Two groups, skewed/outliers -> Mann-Whitney. 3+ groups, normal -> ANOVA. Before/after, same users -> paired t-test or Wilcoxon. Relationship between two metrics -> correlation or regression. No labels, just group similar things -> clustering or factor analysis.

**Sofia:** This is basically the cheat sheet I've wanted since module 2.

**Alex Tutor:** Keep it open, then — you'll want it for what's next.

**Sofia:** Ready when you are.

> **Exercise:** Match each situation to the method that fits it.

## Choosing the test isn't the whole job

**Alex Tutor:** One warning before we finish: picking the right test on paper doesn't protect you from the practical traps we covered in module 3.

**Sofia:** Peeking at results early, running twenty metrics and reporting the one that's significant...

**Alex Tutor:** Exactly those. Decide the test and the primary metric *before* looking at data, and always check practical significance, not just p < 0.05.

![Left: a cat pins up a target before shooting. Right, greyed out: a cat paints a target around an arrow that has already landed.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/choosing-the-right-test-illustration-3.webp)

*Pin up the target before you shoot: choose the test and primary metric before you look at the results.*

**Sofia:** So the process around the test matters as much as the test itself.

**Alex Tutor:** Exactly. Start with the most common trap: a tiny effect with a tiny p.

> **Exercise:** An A/B test on 2 million users shows conversion rising from 5.0% to 5.1%, p = 0.01. Which reading is **correct**?

**Alex Tutor:** And a few more warning signs to spot.

> **Exercise:** Which of these are signs you should not trust a 'statistically significant' result at face value? Select all that apply.

**Alex Tutor:** Let's cement the whole process — not just which test, but the order you do things in.

**Sofia:** I have a feeling 'pick the test first, look at data second' is about to become a strict rule.

**Alex Tutor:** It already is one. Order these steps and you'll never method-shop again.

**Sofia:** Okay, let me think through the sequence.

> **Exercise:** Put the steps of a disciplined test-selection process in the right order.

## Wrapping up the course

**Alex Tutor:** That's the whole course in one map: how many groups, what shape the data has, and what question you're really asking.

![A cat in a graduation cap holds a compass on a hilltop, looking down at a cosy cat-food shop with a winding path to it.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/choosing-the-right-test-illustration-4.webp)

*Course complete: you now have a compass for CatChow's next data question.*

**Sofia:** Six months ago I would've just eyeballed two averages and called it a day.

**Alex Tutor:** And now you know why that's not enough.

**Sofia:** So what's left before I graduate this course?

**Alex Tutor:** Explain your test-selection process to a colleague — that's the next task.

> **Exercise:** In 2–3 sentences, explain how you'd decide between a t-test and a Mann-Whitney test for comparing average order value between two CatChow markets.

**Alex Tutor:** Last one: leadership wants a single, durable guide the whole company can use, not another one-off analysis.

**Sofia:** No pressure, just writing the company's statistics constitution.

**Alex Tutor:** You have everything you need for it already — every method, every pitfall, all six modules.

**Sofia:** Alright. Let's finish this course properly.

> **Exercise:** Write your method-selection memo.

**Alex Tutor:** That's a wrap. You started this course eyeballing averages, and now you can defend a test-selection decision in front of anyone. Well done.

- Previous lesson: [Factor analysis: surveys and NPS drivers](https://productovi.com/courses/statistics-for-pm-and-marketers/factor-analysis.md)

---
Productovi Learning. Free, no sign-up needed to read.