Peeking in A/B testing turned a 5% false-positive rate into 27.6% in 100,000 simulated tests. When to stop an A/B test, and how sequential testing helps.
Correlation vs causation, answered up front: correlation means two metrics move together; causation means changing one actually changes the other. Product data is full of correlations that are not causal, because users choose which features to use, and the kind of user who chooses a feature is often different from the kind who doesn't. Before you invest in "the feature that drives retention", rule out the three usual suspects (a confounder, reverse causality and selection bias), and then test the idea with a randomised experiment or, if that's impossible, a natural experiment.
The dashboard that fooled everyone
CatChow, an online pet-food shop, launched auto-delivery: customers pick a food and a schedule, and it arrives every few weeks. Three months later, a slide appears in the quarterly review:
Customers with auto-delivery: 60% still ordering after six months. Customers without: 27.6%. Auto-delivery more than doubles retention!
The proposed next step: push auto-delivery hard, with pop-ups at checkout and a discount for switching. Revenue projections follow.
The correlation is real. But does auto-delivery cause the retention? Let's split the same customers by how loyal they were before auto-delivery existed:
Segment
With auto-delivery
Without auto-delivery
Loyal (3+ past orders)
70% of 800
66% of 400
New (0–2 past orders)
20% of 200
18% of 1,600
All customers
60% of 1,000
27.6% of 2,000
Within each segment, the gap nearly disappears: 4 points among loyal customers, 2 points among new ones. The big overall gap exists mostly because loyal customers were far more likely to sign up for auto-delivery (800 of the 1,200 loyal customers, versus 200 of the 1,800 new ones). Their loyalty came first; the feature just collected them.
That is a confounder: a third variable, here prior loyalty, that drives both the "cause" and the "effect".
Prior loyalty drove both who signed up for auto-delivery and who kept ordering, so most of the 60% vs 27.6% gap is loyalty; a randomised checkout prompt removes that arrow and measures the real effect
Three reasons correlation isn't causation
1. Confounders: a hidden third factor
A confounder influences both variables, creating a correlation even when neither affects the other. In product analytics, the most common confounders are:
User intent and engagement: engaged users do everything more, including using new features, opening emails and staying longer.
Seasonality: CatChow's ad spend and orders both rise in November. Part of that is the ads; part is that everyone shops before the holidays anyway.
Acquisition channel: users from a referral programme may behave differently from users from paid social in every metric at once.
2. Reverse causality: the arrow points the other way
Sometimes B causes A, not A causes B.
Customers who contact support churn more. Does support drive people away, or do customers contact support because something already went wrong?
Users who visit the help centre convert less. The help centre probably isn't hurting them; people who are confused go looking for help.
Heavy users have more saved addresses. More addresses don't make people order more; ordering more creates more addresses.
3. Selection bias: who ends up in the data
The groups you compare were formed by users' own choices or by a filter, not by chance. Customers who answer an NPS survey are more engaged than those who ignore it. Users who finish onboarding are, by definition, the ones who didn't give up. Any comparison between "did" and "didn't" inherits those differences.
A special case is survivorship: if you only analyse customers still active today, you'll find that everything they did in their first month "predicts" success, because the ones who did those things and left are no longer in the data.
How strong is the correlation, anyway?
Before arguing about causes, measure the relationship properly. The Pearson correlation coefficient r ranges from −1 to +1: close to ±1 means a tight linear relationship, close to 0 means no linear relationship. Two cautions:
r only captures linear relationships. A U-shaped relationship can have r≈0.
r2 tells you the share of variance the relationship "explains". An r of 0.3 sounds respectable, but r2=0.09: 91% of the variation comes from something else.
And of course, even r=0.95 says nothing about which way the arrow points, or whether there is an arrow at all.
How to test for causation
Run a randomised experiment
An A/B test is the cleanest tool because randomisation breaks every confounder at once. Chance, not the user, decides who gets the treatment, so loyal and new customers, heavy and light users, and every hidden factor end up spread evenly across groups. Any difference in outcomes is then caused by the treatment, plus noise that statistics can measure.
You often can't force users to use a feature, but you can randomise the invitation. For auto-delivery, CatChow can show the checkout prompt to a random half of customers and compare six-month retention between the halves, regardless of who actually signed up. This is sometimes called an encouragement design: it measures the effect of promoting the feature, which is exactly what the proposed pop-up campaign would do.
Use a natural experiment when you can't randomise
When a proper test is impossible, look for situations where something outside the users' control split them into groups:
Staggered rollouts (difference-in-differences). CatChow launched same-day delivery in Kyiv but not in Lviv. Weekly orders in Kyiv went from 1,000 to 1,150 (+15%), and in Lviv from 800 to 880 (+10%) over the same weeks. If Kyiv had followed Lviv's trend, it would have reached about 1,100 orders. The extra ~50 orders a week, roughly 4.5% above that baseline, are a reasonable estimate of the effect of same-day delivery. The key assumption is that both cities would have moved in parallel without the launch; check it by comparing their trends in the months before.
Thresholds (regression discontinuity). Free shipping starts at 30 dollars. Customers with carts of 29 and 31 dollars are nearly identical except for the shipping fee, so comparing behaviour just below and just above the threshold isolates its effect.
Accidents and outages. A day when recommendations were down by mistake is an unplanned test of what recommendations are worth.
Geo holdouts for marketing. Pause a campaign in a few randomly chosen regions and compare them with the rest. It's one of the most practical ways to measure whether ads cause sales or just claim credit for sales that would have happened anyway.
Control for what you can measure, and be honest about the rest
Regression and segmentation, like the loyalty split above, can adjust for confounders you know about and have measured. That is much better than a raw comparison, but it can't account for the ones you didn't think of. Treat these results as stronger evidence, not proof.
A checklist before you claim "X drives Y"
Who chose to be in each group, and could they differ in other ways?
Is there a plausible third factor (intent, season, channel, plan) behind both?
Could the arrow point the other way?
Are you only looking at users who "survived" to be in the data?
Does the effect hold within segments, not just overall?
Can you test it: an A/B test, an encouragement design or a natural experiment?
Key takeaways
Correlation is two metrics moving together; causation is one changing the other.
In product data, users choose their features, so confounders and selection bias are everywhere.
Watch for reverse causality: people with problems seek help, heavy users accumulate stuff.
Randomised experiments are the gold standard because randomisation balances every confounder, including the unknown ones.
When you can't randomise, use natural experiments (staggered rollouts, thresholds, geo holdouts) and state their assumptions.
FAQ
What is the difference between correlation and causation?
Correlation measures how two variables move together. Causation means that changing one variable produces a change in the other. Two metrics can be strongly correlated because a third factor drives both, because the effect runs the opposite way, or by coincidence.
Can correlation ever imply causation?
Not on its own. A correlation becomes evidence for causation when alternative explanations are ruled out: the cause comes before the effect, confounders are controlled or randomised away, and the result replicates. A randomised experiment is the most direct way to get there.
What is a confounding variable in product analytics?
A factor that affects both the thing you think is the cause and the outcome. User engagement is the classic example: engaged users adopt more features and also retain better, so almost every feature looks like it "drives retention" in raw data.
How do I prove a feature improves retention?
Randomise access to the feature, or randomise the prompt that invites users to it, and compare retention between the randomised groups. If that's impossible, use a staggered rollout and compare the change over time with a group that didn't get the feature yet.
Learn it hands-on
The free lesson Correlation is not causation uses CatChow's ad spend and order data to measure a relationship properly and then take it apart, with quizzes and an AI-checked task. When you want to adjust for several factors at once, continue with Multiple regression. Both are part of the free course Statistics for Product Managers and Marketers.
