Also called: holdout, long-term holdout, holdout test, control holdout, global holdout
A small group kept without a feature or campaign for a long time to measure its cumulative effect after launch.
Example
Common mistakes
A holdout that isn't random. "Users on old app versions" or "one country" differ in more than the feature; pick the holdout by random assignment.
Leaks into the holdout. Friends in the same group, shared devices or a marketing email forwarded to a colleague expose holdout users; randomize at the level where exposure spreads.
Too small to measure anything. A 1% holdout of a small product can't detect a realistic effect; size it for the effect you care about.
Letting it run forever. Holding people out of a working feature has a cost; set the end date and the metric before launch.
Reading many metrics until one moves. Name one primary metric up front; the rest are guardrails.
Learn it in the course
Channels and marketing attribution · Channel capacity and marginal CAC, a break-even cost that values the invitees a paid user brings, marketing attribution through the stores' privacy frameworks, and incrementality from a geo holdout.
A tracking plan for a new feature · Before payment reminders ship, plan the data: the questions, the events and properties that answer them, where each is logged, the metric definitions and the checks before launch.
Invites and viral growth · Viral growth from invites: the viral coefficient k, cycle time, why k below 1 still pays, and how to measure invites without flattering yourself.
A holdout group is a randomly chosen slice of users, usually 1–10%, who don't get a feature or a campaign while everyone else does, and who stay that way for weeks or months. Comparing the rest with the holdout shows the long-term, cumulative effect of what you shipped.
It answers questions a normal A/B test can't:
Does the effect last? A two-week test catches novelty; a holdout shows whether the lift survives after three months.
What did a whole set of changes add up to? A global holdout kept away from every launch of a quarter shows whether ten "winning" tests really added up to ten wins.
Is a campaign incremental? In marketing, a holdout that receives no emails, pushes or ads shows how many conversions would have happened anyway. This is the cleanest way to measure incrementality.
The effect is read like any experiment: the difference between exposed users and the holdout, with its confidence interval:
The price is real: holdout users miss something that works, so keep the group as small as the needed precision allows and set an end date up front. Randomize at the right level too. If a feature affects everyone in a shared space, like a group of friends, hold out whole groups, or the effect leaks into the holdout through their friends.
Halves launches automatic settle-up reminders to 40,000 active users but holds out 5% of groups (2,000 users), so no one in a holdout group gets reminders from a friend's side either. After 90 days, the metric is "recorded a settle-up in the last 30 days":
Standard error: SE=380000.36×0.64+20000.32×0.68=0.00000606+0.0001088≈0.0107.
95% interval: 4.0±1.96×1.07≈4.0±2.1, i.e. +1.9 to +6.1 pp.
The launch test showed +6 pp in its first two weeks; three months later the effect is smaller but clearly above zero. The cost of the holdout was about 2000×0.04=80 users who didn't settle up in a month, which Maya judged worth a trustworthy long-term number. (With users grouped, a precise interval should account for clustering, which widens it somewhat.)
