Reading a funnel: right censoring and date ranges

Data pulled on 31 August holds every sign-up from 6 July to 30 August. For the newest week, 24–30 August, 7 of 24 users have added a first expense so far: 7 ÷ 24 = 0.292 ≈ 29.2%. It looks like a disaster. Nothing broke.

Those users signed up between one and seven days before the data was pulled. Someone who joined on the 30th has had a single day. Their timeline is cut off on the right, at the moment of the export, so we don't yet know what they'll do. That's right censoring, and the data it produces is censored.

The key point: a censored user without a first expense is not a zero. It's "not yet". Their rate will climb over the coming days as the late converters arrive.

So a funnel number for a recent cohort is a minimum, not a result. Before reacting to a dip in the newest week, ask how many days its users have had compared with the window you care about. If it's fewer, the dip is the calendar, not the product.

Right censoring has a mirror image. Maya's record, week 3 at 83.3%, is a "so far" rate too, but week 3 has had about six weeks to grow it. A "so far" rate rewards old cohorts just for being old.

The fix has two rules:

1. A fixed conversion window for everyone. Count a first expense only on days 0–6 after sign-up. On that window week 3 is 13 ÷ 24 = 54.2%: the same as weeks 2 and 6, and below week 7 (58.3%).
2. Only cohorts that have lived through the whole window. With data pulled on 31 August, anyone who signed up by 23 August has had 7 full days, so weeks 1–7 are complete; week 8 isn't.

Across the seven complete weeks: 168 sign-ups, 83 first expenses within 7 days, 83 ÷ 168 = 0.494 ≈ 49.4%.

For the chart, either leave the incomplete week off until its window closes (6 September here) or show it greyed out and labelled "incomplete". Comparing a cohort's 7-day rate with another's six-week rate is comparing two different metrics.

Theo's instinct sounds statistical: "more users in the average means a more precise number". Average all eight weeks, including the unfinished one, and you get (83 + 7) ÷ 192 = 0.469 ≈ 46.9% instead of 49.4%.

The problem isn't precision; it's bias. The 24 users of week 8 haven't had their 7 days. Anyone among them who will add a first expense on day 3, 4, 5 or 6 is counted today as a non-convert. Adding them doesn't make the estimate sharper; it pulls it down by a predictable amount, because part of the sample is measured on a shorter window.

Precision and bias are different things:

• precision improves with more users measured the same way;
• bias comes from measuring some users differently, and more data doesn't cure it.

The rule is simple: an incomplete cohort never goes into an average with complete ones. Wait for its window to close, then include it. Until then, show it separately and marked as incomplete, or not at all.

Theo's monthly report takes first expenses dated in August (67) and divides by sign-ups dated in August (103): 65%. It looks like a funnel conversion, but the numerator and the denominator are different people.

• 10 of the 67 first expenses belong to users who signed up in late July. They're in the numerator, not in the denominator.
• Many August sign-ups, especially from the 24th on, haven't had their 7 days. They're in the denominator, with their conversions still to come.

The rule for funnels: the date range picks who enters. Then you follow those same people through their window, whatever date their later steps fall on. A first expense on 3 August by someone who signed up on 30 July belongs to the July cohort.

The "events in a period ÷ entries in the same period" shortcut is common in dashboards because it's easy to compute. It can even give plausible numbers in a steady month. But whenever sign-ups change from month to month, or a window crosses the month's edge, it mixes cohorts and drifts.

A monthly funnel number is fine, as long as it's built from a cohort. For Theo's August report, combine both rules from this lesson:

1. Entry by sign-up date: users who signed up in August.
2. Complete windows only: with data pulled on 31 August and a 7-day window, only sign-ups up to 23 August have had their full 7 days.

So the cohort is sign-ups from 1 to 23 August: 79 users. Of them, 39 have first_expense_day from 0 to 6. 39 ÷ 79 = 0.494 ≈ 49.4%.

Compare with Theo's 65.0%. The gap of almost 16 points is entirely construction: July users in the numerator, unfinished August users in the denominator. Nothing about August's users changed.

It's not a coincidence that the August cohort gives the same 49.4% as all seven complete weeks: the weekly rates have no trend, so any honest slice lands near the same level. That agreement is a useful sanity check. When an honest slice and a shortcut disagree by this much, trust the cohort.

The last week of August joins the August number once its window closes, on 6 September.

To judge the "split your first dinner" push in week 3, you need a baseline measured the same way: same window, complete cohorts, nearby weeks.

The four July sign-up weeks, 6 July to 2 August, give 96 users. Within 7 days: 9 + 13 + 13 + 11 = 46 first expenses. 46 ÷ 96 = 0.479 ≈ 47.9%.

Week 3, the push week, is 13 ÷ 24 = 54.2%, about 6 points above the July average. Before calling that an effect, look at the spread of the other weeks, measured the same way: from 37.5% (week 1) to 58.3% (week 7). Week 3 ties weeks 2 and 6, which had no push, and trails week 7.

With cohorts of 24, one person is about 4 percentage points, so swings of 10 or 20 points happen by chance. A single week's difference has to stand out from that natural range before it means anything. For a proper comparison you'd put an interval around each rate (the confidence intervals lesson in the statistics course shows how) or, better, run a test.

The baseline does its job: it shows there's nothing yet to credit.

A before-and-after comparison can credit a change only when the difference is larger than the natural spread, on the same window. Put the evidence about the week 3 push together:

• Maya's 83.3% is a "so far" rate that has had six weeks to grow. It can't be compared with younger weeks.
• On the same 7-day window, week 3 is 54.2%: level with weeks 2 and 6, which had no push, and below week 7.
• Its 6-point lead over the July average (47.9%) sits well inside the 37.5–58.3% swing of 24-user weeks.

So the honest statement is: the data can't credit the push. Not "it worked", and not "it worked a little".

Two overcorrections to avoid. The data doesn't show the push hurt either: no evidence of an effect isn't evidence of no effect (module 5 shows why, with test power). And "nothing can be said without an A/B test" is wrong too: you can say exactly what this comparison shows.

To learn whether the push helps, send it to a random half of several weeks' sign-ups and compare first expense within 7 days once every window has closed.

When before-and-after data can't answer whether a change works, design a comparison that can.

The tempting plan is to ship the push to everyone and watch the line. It can't work here: weekly 7-day rates already swing between 37.5% and 58.3% with no change at all, and anything else that shifts at the same time (a new channel, school holidays, an app release) lands in the same line.

The fair plan is to split each week's sign-ups at random:

• half get the push, half don't, so season, channels and app version are shared and the only systematic difference is the push;
• compare first expense within 7 days, read only after the window closes for everyone;
• run it for several weeks, long enough for a difference you care about to stand out from chance.

Decide in advance which number you'll compare and what difference would make you keep the push.

How it fits together: complete windows, entry cohorts and a baseline turn a chart into a comparison; when that still can't answer, randomize. Next, Activation: finding the aha moment asks which early action predicts staying.