# Two-sample t-test

> The two-sample t-test (Student's t-test): comparing the means of two groups and deciding whether the gap is bigger than chance.

- Course: [Statistics for Product Managers and Marketers](https://productovi.com/courses/statistics-for-pm-and-marketers.md)
- Module: Comparing two groups
- Lesson 6/24, ~39 min
- Source: https://productovi.com/courses/statistics-for-pm-and-marketers/t-test

## A widget, and a question

**Alex Tutor:** A new question has landed on CatChow's product team: the checkout team added an upsell widget ('add a scratching post?') to half of new sessions last week. They want to know: does it change the average order value?

![Two identical checkout screens side by side; only the right one shows a small pop-up with a scratching post, and a cat's paw hovers over it.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/t-test-illustration-2.webp)

**Sofia:** Easy — control averaged $42.50, variant averaged $52.50. Widget wins, ship it everywhere!

**Alex Tutor:** Whoa, hold the confetti. Both numbers came from samples of only 10 orders each. Remember last lesson: sample means wobble. We need a way to ask 'is \$10 bigger than the wobble, or is it the wobble?'

**Sofia:** So we can't just trust whichever number is bigger?

**Alex Tutor:** Not on its own. That's exactly what the **two-sample t-test** (the t-test for independent samples) answers: is the gap between two group means bigger than sampling noise would plausibly produce? Here's the full dataset.

**Dataset (CSV):** [Checkout widget A/B test, one week](https://productovi.com/datasets/statistics-for-pm-and-marketers/t-test/catchow-widget-test.csv)

20 orders: 10 sessions without the upsell widget (control) and 10 with it (variant). Synthetic data made for this course.

- `order_id`: Order number
- `group`: control (no widget) or variant (widget shown)
- `order_value`: Order value, USD

**Sofia:** That's independent samples, right — the two groups are different customers, not the same person before and after?

**Alex Tutor:** Right. **Independent samples** means nothing links a specific control order to a specific variant order. If it were the same customers measured twice, we'd need a *paired* test instead — that's a later lesson.

**Alex Tutor:** Time for a quick check on when this test applies at all.

> **Exercise:** Which scenario calls for the classic (independent-samples) t-test?

## How to calculate the t-statistic

**Alex Tutor:** The idea: take the difference between the two means, then divide by how much that difference would naturally wobble from sample to sample — its standard error.

![Two jiggling bowls of kibble with an orange arrow spanning the gap between them, much wider than the jiggle, while a cat measures with a tape.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/t-test-illustration-3.webp)

*Is the gap bigger than the usual wobble?*

$$
t = \frac{\bar{x}_2 - \bar{x}_1}{SE_{\bar{x}_2-\bar{x}_1}}
$$

*Student's t-statistic: the difference between the two sample means, divided by the standard error of that difference.*

**Sofia:** And the standard error of a difference is... bigger than the standard error of one group, I'd guess?

**Alex Tutor:** Good guess. When both groups have similar spread, we pool their variances into one estimate, then combine the two sample sizes into the formula below.

$$
SE_{\bar{x}_2-\bar{x}_1} = \sqrt{s_p^2\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}, \quad s_p^2 = \frac{(n_1-1)s_1^2+(n_2-1)s_2^2}{n_1+n_2-2}
$$

*Pooled standard error: sp² blends both groups' variances, weighted by their degrees of freedom.*

**Alex Tutor:** For our data: control has mean $42.50 and s₁ ≈ $3.03 (n₁ = 10); variant has mean $52.50 and s₂ ≈ $3.37 (n₂ = 10). Your turn to pool them.

> **Exercise:** Control: n₁ = 10, mean $42.50, s₁ = $3.03. Variant: n₂ = 10, mean $52.50, s₂ = $3.37. Using the pooled-variance formula, what is the standard error of the difference between the two means?

**Alex Tutor:** With the standard error in hand, the t-statistic itself is one division away.

**Sofia:** Just to be precise: is 't' itself the thing we called a 'test statistic' back in the intro, or is that a different animal?

**Alex Tutor:** Same animal. Any single number computed from sample data and used to decide between H0 and H1 is a **statistic critical to the test** — a *test statistic*. t is just the one this particular test uses.

**Sofia:** Difference in means over standard error, like the formula said. Let me plug in your exact SE this time so we don't compound rounding.

> **Exercise:** The pooled standard error of the difference is \$1.4337. The variant mean is \$52.50 and the control mean is \$42.50. What is Student's t-statistic?

## Degrees of freedom and the verdict

**Alex Tutor:** One more ingredient before we can judge the t-value: **degrees of freedom (df)**. For the pooled two-sample test, df = n₁ + n₂ - 2 — here, 10 + 10 - 2 = 18.

**Sofia:** Why subtract 2 and not, say, 1 or 0?

**Alex Tutor:** Each group's own mean 'uses up' one degree of freedom to estimate that group's variance — two groups, two means used, two degrees lost. It's the same logic from the coin-counting trick you saw with a single sample, just doubled.

**Alex Tutor:** With df = 18, a statistics table (or software) gives a **critical value** of about 2.101 for a two-tailed test at the usual 5% significance level.

> **The decision rule**
>
> If the absolute value of the calculated t exceeds the critical value for the chosen significance level and df, we reject the null hypothesis (H0: the means are equal) in favour of the alternative (H1: they differ).

**Sofia:** So I compare our t against 2.101 and that tells me if the widget really did something?

> **Exercise:** With df = 18 and a critical value of 2.101 at the 5% level, a calculated t of 6.97 means we reject the null hypothesis that the two group means are equal.

**Alex Tutor:** Before the vocabulary, one reading check. A significant t is easy to over-sell in a meeting.

> **Exercise:** CatChow's upsell widget test (10 orders per group, mean order value \$42.50 without the widget and \$52.50 with it) gave t ≈ 6.97 against a critical value of 2.101 (df = 18). Which summary for the team is **correct**?

**Sofia:** What's the 5% actually called, formally? I keep saying '5% cutoff' in meetings and getting funny looks.

**Alex Tutor:** It's the **significance level (α)** — рівень значущості, if you want to sound properly official. Pick it *before* you look at the data, not after.

**Sofia:** And if the true effect is real but we still fail to reject H0 — say, because our sample was too small — what do we call that mistake?

**Alex Tutor:** A **type II error** — помилка другого роду: missing a real effect. Its evil twin, a type I error, is claiming a difference that isn't really there. Match these last few pieces of vocabulary before we move on.

> **Exercise:** Match each term to its definition.

**Alex Tutor:** Let's run the whole thing once more on fresh numbers, start to finish, so the steps stick.

> **Exercise:** A second test on the treats page: 8 sessions per group, control mean 30 items viewed, variant mean 35, and both groups have s = 4. Using the pooled formula, what is the t-statistic?

## When the classic t-test misbehaves: Welch's test and outliers

**Alex Tutor:** Two warnings. First: the classic pooled t-test assumes the two groups have roughly equal variances. Ours were close (9.17 vs 11.39) — fine. If they'd been wildly different, we'd use **Welch's t-test** instead, which adjusts the df downward.

**Alex Tutor:** Second, bigger warning: because the formula is built from means, one freakishly large outlier order can wreck it completely — the same problem outliers cause for any mean.

![A row of small kittens pulls a rope in a tug-of-war while one enormous cat on the other end drags them all across the line.](https://productovi.com/courses/statistics-for-pm-and-marketers/inline/t-test-illustration-4.webp)

*Means, and tests built on them, give one huge value a lot of pull.*

**Sofia:** What do we do if we suspect outliers, or the data just doesn't look bell-shaped?

**Alex Tutor:** Then the t-test is the wrong tool, full stop — next lesson covers the rank-based alternative that doesn't care about outliers or shape.

**Alex Tutor:** Quick gut-check on assumptions before you leave.

> **Exercise:** Which of these are assumptions behind the classic (pooled) Student's t-test for independent samples? Select all that apply.

**Alex Tutor:** And one case where the variances clearly don't match. Pick the right tool.

> **Exercise:** Control: 40 orders, s = \$8. Variant (a new premium bundle): 15 orders, s = \$25. You want to compare mean order value. Which test fits best?

## Wrap-up

**Sofia:** Honestly, our t was so far past 2.101 that this feels like overkill for such an obvious result.

**Alex Tutor:** Let's name-drop properly what we actually ran: the **t-критерій Стьюдента для незалежних вибірок** (Student's t-test for independent samples), with **t-критерій Велча** as the fallback for unequal variances. Full names, for your next report.

**Sofia:** Noted. So even something as ordinary as average order value — sorry, our beloved середній чек — deserved this much rigour, with a sample size (обсяг вибірки) of only 10 per side?

**Alex Tutor:** Sometimes the data really is that clean. But you only know that *because* you ran the test — 'looks obvious' has burned plenty of PMs before. Put that instinct into words for me.

> **Exercise:** In the upsell widget test (10 orders per group, \$42.50 vs \$52.50, t ≈ 6.97), Sofia, a product manager on the team, said the result 'feels obvious' and the test seems like overkill. In 2-3 sentences, explain why running the t-test still mattered here, even though the conclusion turned out to be clear-cut.

**Alex Tutor:** Nicely argued. Now, a messier version of this exact story, care of the checkout team's group chat.

**Sofia:** Why do I feel like I'm about to be blamed for something again.

**Alex Tutor:** Not this time — it's someone else's dataset to critique for once. Small mercy.

**Sofia:** I'll take it.

> **Exercise:** Review the checkout team's claim.

- Previous lesson: [Sample vs population and the normal distribution](https://productovi.com/courses/statistics-for-pm-and-marketers/samples-and-normal-distribution.md)
- Next lesson: [Mann–Whitney: when data isn't normal](https://productovi.com/courses/statistics-for-pm-and-marketers/mann-whitney.md)

---
Productovi Learning. Free, no sign-up needed to read.