Multiple linear regression

Orders rarely depend on one thing. Multiple regression fits one equation with several predictors at once:

ŷ = b0 + b1x1 + b2x2 + … + bkxk

It is still fitted by least squares: the coefficients are chosen together so the sum of squared residuals is as small as possible.

The key to reading it is the phrase "holding the other predictors constant". In orders = b0 + b1 × ad_spend + b2 × is_holiday_week, b1 is the predicted change in orders per extra dollar of ad spend when holiday status doesn't change. It compares weeks that are alike on every other predictor in the equation.

That's the big gain over simple regression. If CatChow spends more in holiday weeks, a simple regression on ad spend gives ad spend the credit for some holiday orders too. Adding the holiday flag separates the two.

Two cautions. "Holding constant" is a property of the equation, not of the data collection: nobody froze anything. And it only adjusts for predictors that are in the model; a confounder you left out still leaks into the coefficients.

Some predictors are yes/no facts: is it a holiday week, did the user see the new checkout, is the customer on a subscription. Regression handles them as 0/1 variables: 1 when the fact is true, 0 when it isn't.

Its coefficient is then a fixed bump. In

orders = 12 + 0.05 × ad_spend + 4 × is_holiday_week

a holiday week adds 4 predicted orders on top of whatever ad spend predicts, at any spend level. On a chart, holiday and ordinary weeks get two parallel lines, 4 orders apart.

To predict, plug every predictor in and add up the terms. For a holiday week with ad_spend = 400:

12 + 0.05 × 400 + 4 × 1 = 12 + 20 + 4 = 36 orders.

Work term by term and write each one down; with several predictors it's easy to drop one. And keep units straight: ad_spend here is in dollars, so 400 means $400, not 400 hundred.

A flat bump is an assumption. If holidays made ads more effective, you would need an extra interaction term; this equation can't show that.

In orders = 12 + 0.05 × ad_spend + 4 × is_holiday_week, an ordinary week has the holiday flag at 0, so its term vanishes: 4 × 0 = 0. The prediction comes from the intercept and ad spend alone. For ad_spend = 600:

12 + 0.05 × 600 + 4 × 0 = 12 + 30 + 0 = 42 orders.

This is where the meaning of the intercept changes. In this equation, 12 is the prediction for an ordinary week with zero ad spend: the reference group is "not a holiday". The holiday coefficient says how holiday weeks differ from that reference.

The same logic works for categories with more than two values, like traffic channel. You pick one channel as the reference and add a 0/1 variable for every other one; each coefficient is then the gap from the reference channel. Never add a variable for every category plus an intercept: the variables would always add up to 1 and the model couldn't separate them from the intercept.

Two slips to avoid: adding the bump "just in case" when the flag is 0, and reading 12 as typical orders, when it's a corner case nobody may have observed.

A tempting shortcut is to order predictors by the size of their coefficients. It almost always misleads, because each coefficient carries its predictor's units.

In orders = 12 + 0.05 × ad_spend + 4 × is_holiday_week, 0.05 is orders per dollar and 4 is orders per holiday week. Saying the second is "80 times bigger" compares dollars to weeks. Measure spend in hundreds of dollars and its coefficient becomes 5, now bigger than 4, though nothing real has changed.

Two fair ways to compare:

• realistic changes. Multiply each coefficient by a change that actually happens. A typical budget step of $100 gives 0.05 × 100 = 5 orders; a holiday week gives 4. Similar effects;
• standardised coefficients. Rescale every predictor to standard-deviation units first, so each coefficient means "effect of a one-SD change".

Even then, a bigger effect isn't automatically the better lever. Ask also what each change costs, whether you can actually make it, and whether the coefficient is stable, which correlated predictors can undermine.

On the data a model was fitted on, R² can only go up or stay the same when you add a predictor, however useless that predictor is.

The reason is simple. Least squares picks the coefficients that make the squared residuals as small as possible. With a new column available, it could always set that column's coefficient to exactly zero and get the old fit back. So the new fit is at least as good, and in practice a random column almost always correlates with the leftover noise a little, so R² creeps up.

Add the weather in some other city, the day's lottery numbers and the office cat's naps, and R² on the past twenty weeks will rise. Forecasts for next month won't improve; they'll usually get worse.

That's why R² alone is a poor way to choose between models with different numbers of predictors. Two tools fix it:

• adjusted R² and information criteria (AIC, BIC), which charge a price for each extra predictor;
• a train/test split, which checks the model on data it never saw.

Adjusted R² corrects R² for the number of predictors, so a model only scores higher if a new predictor helps more than chance would:

R²adj = 1 − (1 − R²) × (n − 1) / (n − k − 1)

where n is the number of observations and k the number of predictors (not counting the intercept).

For a model with R² = 0.90, k = 5 predictors and n = 20 weeks:

• n − 1 = 19 and n − k − 1 = 14, so the penalty factor is 19 / 14 ≈ 1.357;
• 1 − R² = 0.10, and 0.10 × 1.357 ≈ 0.136;
• R²adj = 1 − 0.136 ≈ 0.86.

The penalty grows as k gets close to n. With 15 predictors on 20 weeks, the same R² = 0.90 would give 1 − 0.10 × 19 / 4 = 0.525. A small gap between R² and adjusted R² says the predictors are pulling their weight; a big gap says some are along for the ride.

Two slips: using n − k instead of n − k − 1, and forgetting to subtract from 1 at the end. Adjusted R² can even go negative for a hopeless model, which is a useful alarm.

Multiple regression has two classic traps, and they are easy to confuse.

• Multicollinearity is a problem between predictors. When two predictors move together tightly, like ad spend and discount depth raised in the same weeks, the model can't tell which deserves the credit. Their coefficients become unstable: a few new weeks of data can swing them a lot or even flip a sign.
• Overfitting is a problem of too much flexibility. With many predictors and few observations, the model starts fitting the noise of these particular weeks. It looks excellent on the data it was fitted on and disappoints on new data.

Two checks target overfitting:

• a train/test split (holdout): fit on one part of the data, say 80% of the weeks, and measure accuracy on the 20% the model never saw;
• an information criterion (AIC, BIC): a score that rewards good fit but subtracts points for every extra predictor. Lower is better; it's for comparing models fitted on the same data.

For multicollinearity the first check is simpler: look at the correlations between predictors before reading their coefficients.

A forecast is always about data the model hasn't seen. A train/test split imitates that before you rely on the model:

1. Set aside part of the data, often the last 20% of weeks, as the test set.
2. Fit every candidate model on the remaining training data only.
3. Compare the models on the test set and pick the best there.

Read the pair of scores together. A model with R² = 0.95 on training and 0.55 on test has learned the quirks of its training weeks: classic overfitting. A model with 0.85 and 0.82 is less impressive on paper but holds up, so it's the one to ship. A small drop from training to test is normal; a big one is the warning.

For time data, split by time (train on earlier weeks, test on later ones) rather than at random, so the test imitates the real future.

Keep the test set honest: if you keep tweaking the model until the test score looks good, the test set quietly becomes training data. Decide on the candidates first, then look.

Multicollinearity means two or more predictors are strongly correlated with each other. At CatChow, discount depth rises in the same weeks as ad spend, so the data has almost no weeks with big ads and small discounts, or the other way round. The model has nothing to separate their effects with.

What it breaks and what it doesn't:

• broken: individual coefficients. They become unstable, their uncertainty balloons, and their signs can flip. "Discount has the bigger coefficient" is not a safe conclusion;
• fine: overall predictions, as long as the predictors keep moving together the way they did. The model still knows that "big promo weeks" bring more orders.

Common fixes:

• drop one of the correlated predictors and describe the other as standing for both;
• combine them into one measure, like a promotion-intensity score;
• use stepwise regression, which adds or removes predictors one at a time by a fit criterion, though it can be fooled by chance on small data;
• best of all, create variation: run some weeks with ads but no deeper discount, and vice versa.