Logistic regression and classification

Many product questions have yes/no answers: will this subscriber churn, will this visitor buy, will this user open the email. You can code the outcome as 1 (yes) and 0 (no) and fit an ordinary regression line, but it breaks in two ways.

• Impossible predictions. A straight line has no floor or ceiling. For a customer silent for a year it may predict 1.3; for a brand-new loyal one, −0.2. Neither is a probability.
• The wrong shape. Real probabilities flatten near the edges. Going from 1 to 2 months of silence might add a lot of churn risk; going from 11 to 12 adds little, because the risk is already near 1. A line forces the same step everywhere.

Logistic regression fixes both. It predicts a probability with an S-shaped curve that always stays between 0 and 1: flat near 0, steep in the middle, flat again near 1.

The output is a probability, not a verdict. Turning "62% churn risk" into "send the email" is a separate step, a classification threshold, which this lesson reaches at the end.

Odds compare the chance that something happens with the chance that it doesn't:

odds = p / (1 − p)

If 80% of similar customers churn, p = 0.8 and odds = 0.8 / 0.2 = 4: churning is four times as likely as staying. Some reference points:

• p = 0.5 → odds = 1 (even);
• p = 0.2 → odds = 0.2 / 0.8 = 0.25 (one churner for every four who stay);
• p = 0.9 → odds = 9.

To go back: p = odds / (1 + odds). Odds of 4 give 4 / 5 = 0.8.

Odds run from 0 to infinity, and they stretch out fast near the top: p = 0.99 already gives odds of 99. That stretching is exactly why logistic regression works with odds: they turn a number trapped between 0 and 1 into one with room to grow.

The typical slip is dividing by the wrong thing: p / 1 = 0.8 (that's just the probability again) or (1 − p) / p = 0.25 (those are the odds of staying, not of churning).

Odds still can't go below 0. Taking the natural logarithm fixes that: log-odds (the logit), ln(odds), runs from minus to plus infinity and is 0 at p = 0.5. That's the quantity logistic regression models with a straight line:

ln(odds) = b0 + b1x

To get a probability, go through three steps. With CatChow's churn model ln(odds) = −2 + 0.8 × months since last purchase, for a customer silent for 3 months:

1. Log-odds: −2 + 0.8 × 3 = 0.4.
2. Odds: e^0.4 ≈ 1.49.
3. Probability: 1.49 / (1 + 1.49) ≈ 0.60.

Steps 2 and 3 together are the logistic (sigmoid) function, p = 1 / (1 + e^−(b0 + b1x)), which gives the same 0.60.

The common slip is to stop at step 1 and report 0.4 as the probability. Log-odds can be any number, including negative ones: a brand-new customer has log-odds of −2, which is a probability of about 0.12, not "minus two".

In linear regression a coefficient adds a fixed amount to the outcome. In logistic regression, b1 adds a fixed amount to the log-odds, which is hard to picture. Undo the log and it becomes a multiplier:

odds ratio = e^b1

For CatChow's model, b1 = 0.8, so each extra month without a purchase multiplies the odds of churn by e^0.8 ≈ 2.23. An odds ratio of 1 means no effect; above 1 raises the odds, below 1 lowers them (e^−0.5 ≈ 0.61 would cut them by about 40%).

The trap: an odds ratio multiplies odds, not probability. Because probability flattens near 0 and 1, the same odds ratio moves the probability by different amounts depending on where you start:

• from p = 0.12 (odds 0.14), one month takes odds to 0.30 and p to about 0.23;
• from p = 0.60 (odds 1.49), it takes odds to 3.32 and p to about 0.77.

Neither is "2.23 times the probability". When you report a result to a team, translate it into probabilities at a few realistic starting points.

To see what an odds ratio does in practice, take one step by hand. With CatChow's churn model, ln(odds) = −2 + 0.8 × months since last purchase, a customer silent for 3 months has log-odds 0.4, odds e^0.4 ≈ 1.49 and churn probability ≈ 0.60. Add one month:

1. Multiply the odds by the odds ratio e^0.8 ≈ 2.23: 1.49 × 2.23 ≈ 3.32. (Same result as computing the new log-odds directly: −2 + 0.8 × 4 = 1.2, and e^1.2 ≈ 3.32.)
2. Convert back to a probability: p = odds / (1 + odds) = 3.32 / 4.32 ≈ 0.77.

The odds more than doubled; the probability rose by 17 percentage points, from 0.60 to 0.77. One more month would take the odds to about 7.4 and p to about 0.88: an even smaller step, as the curve flattens towards 1.

This is the S-curve at work. In the middle (p near 0.5) a coefficient moves the probability the most; near the floor or ceiling it barely moves it.

The slip to avoid is multiplying the probability itself: 0.60 × 2.23 ≈ 1.34, which is impossible for a probability and a clear sign that the odds ratio went to the wrong place.

Once a threshold turns probabilities into yes/no calls, every prediction can be checked against what really happened. The confusion matrix sorts them into four cells. For a churn model, where "positive" means "predicted to churn":

• true positive (TP): predicted churn, and the customer churned;
• false positive (FP): predicted churn, but the customer stayed (a false alarm);
• false negative (FN): predicted they'd stay, but they churned (a miss);
• true negative (TN): predicted they'd stay, and they stayed.

A decoding rule: the second word is what the model said (positive = churn); the first is whether it was right (true or false). A false negative is a "stay" call that turned out wrong.

The cells matter because the two kinds of error cost different things. At CatChow a false positive wastes a cheap retention email on someone who'd have stayed; a false negative loses a subscriber. Which error hurts more depends on the business, and that's what the threshold choice is about.

Always check which class the table calls "positive": some tools put "stayed" there, which swaps every label.

Accuracy is the share of all predictions that were right: (TP + TN) / everyone. It sounds like the natural score, but it breaks when one outcome is rare.

Suppose 5% of subscribers churn each month. A "model" that always predicts "stays" is right about the 95% who stay and wrong about every churner: accuracy 95%, yet it flags nobody and saves nobody. A real model that catches most churners at the cost of some false alarms can easily score lower accuracy, for example 90%, while being far more useful.

The fix is to measure performance on the class you care about:

• recall = TP / (TP + FN): of the customers who actually churned, what share did the model catch? The do-nothing model scores 0;
• precision = TP / (TP + FP): of the customers it flagged, what share actually churned?

Get into the habit of asking for the base rate first. If 95% of cases fall in one class, 95% accuracy is the floor you get for free, not an achievement. The same applies to fraud, refunds, failed deliveries and any other rare event worth predicting.

Precision and recall answer two different questions about the positive calls, and both come straight from the confusion matrix.

• Precision = TP / (TP + FP). Of everyone the model flagged as churning, what share really churned? It measures how many of your alarms are real.
• Recall = TP / (TP + FN). Of everyone who really churned, what share did the model flag? It measures how many real cases you catch.

For a model with 40 true positives, 10 false positives, 20 false negatives and 130 true negatives:

• precision = 40 / (40 + 10) = 40 / 50 = 0.80;
• recall = 40 / (40 + 20) = 40 / 60 ≈ 0.67;
• accuracy = (40 + 130) / 200 = 0.85.

So 80% of the retention emails go to people who were really about to leave, but a third of the leavers get no email at all.

The two formulas share a numerator and differ only in the denominator: precision adds false positives, recall adds false negatives. Mixing them up is the most common slip, so say the question in words before you pick the formula.

The classification threshold is the probability above which the model's answer becomes "yes". The default of 0.5 ("more likely than not") is a convention, not a law.

Moving the threshold trades one error for the other:

• lower threshold (say 0.2): more customers flagged. Recall rises, since fewer churners slip through; precision usually falls, since more false alarms get in;
• higher threshold (say 0.8): fewer flagged, fewer false alarms, more misses.

Accuracy has no fixed direction here. With a rare outcome, lowering the threshold often lowers accuracy, because it converts many correct "stays" into false alarms.

Pick the threshold from the costs of the two errors. At CatChow a retention email costs about $5; a lost subscriber costs about $150 of future revenue. One missed churner costs as much as 30 unnecessary emails, so a low threshold that favours recall makes sense. If the action were expensive, like a personal call from the sales team or a big discount, you'd raise it.

Re-check the confusion matrix after any change: the threshold is a business decision that statistics informs, not one it makes for you.