Overfitting & Regularisation
Why a model that scores 99% on training data can be worthless, and the levers that fix it.
Memorising Is Not Learning
The classic framing is the bias-variance trade-off. High bias is a model too simple to capture the pattern: it is wrong in the same way everywhere, and it underfits. High variance is a model flexible enough to fit the noise: it is different every time you retrain it on a new sample, and it overfits.
| Symptom | Diagnosis | Response |
|---|---|---|
| Training and validation both poor | Underfitting | More capacity, better features, train longer |
| Training excellent, validation poor | Overfitting | Regularise, or get more data |
| Both good, test poor | You leaked the test set | Rebuild the split |
| Validation better than training | Usually a bug or an easy split | Check the split and the augmentation |
L1 & L2 Penalties
Regularisation adds a penalty on the size of the weights, so the fit has to justify complexity. Two forms dominate, and they behave differently in a way worth remembering.
| L2 (ridge, weight decay) | L1 (lasso) | |
|---|---|---|
| Penalty | Sum of squared weights | Sum of absolute weights |
| Effect | Shrinks all weights smoothly | Drives some weights to exactly zero |
| Gives you | Stability with correlated features | Feature selection for free |
| Use when | Almost always, as the default | You want a sparse, interpretable model |
# the strength of the penalty is a hyperparameter you tune on validation
ridge = Ridge(alpha=1.0).fit(X_train, y_train) # L2
lasso = Lasso(alpha=0.01).fit(X_train, y_train) # L1 — many coefficients hit 0
# in deep learning the same idea is weight decay on the optimiser
optimiser = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)Dropout, Early Stopping & Augmentation
Neural networks have enough capacity to memorise almost any dataset, so regularisation is not optional. Three techniques carry most of the load.
| Technique | Mechanism | Watch for |
|---|---|---|
| Dropout | Randomly zeroes activations during training, so no unit can be relied on | Must be disabled at inference |
| Early stopping | Stops when validation loss stops improving | Needs patience, or it stops on noise |
| Data augmentation | Label-preserving transforms multiply the effective dataset | A transform that changes the label is poison |
| Batch normalisation | Stabilises training, mildly regularising | Interacts awkwardly with small batches |
| More data | The only one with no downside | Usually the expensive option |
Validate with cross-validation where the dataset is small, and with a fixed held-out split where it is large. For time series, split by time — a random split lets the model see the future, which produces a spectacular validation score and a useless model.
Interview Questions
What is the bias-variance trade-off?
Bias is error from a model too simple to capture the pattern; variance is error from fitting noise. Reducing one usually raises the other, and generalisation is the balance.
How do you detect overfitting?
Training performance far exceeds validation performance. If both are poor the model underfits; if both are good but test is poor, the test set has leaked.
L1 or L2?
L2 shrinks all weights smoothly and is the default. L1 drives some weights exactly to zero, giving feature selection and a sparser, more interpretable model.
Why must dropout be disabled at inference?
It exists to stop the network relying on any single unit during training. At inference you want the full network and a deterministic prediction.
What is data leakage?
Information about the target reaching the model when it should not — a feature encoding the answer, or preprocessing fitted before the split. It produces excellent validation scores and a model that fails in production.
How do you split time-series data?
By time, never randomly. A random split lets the model train on the future and score brilliantly on validation while being worthless in deployment.