Overfitting and the Bias-Variance Tradeoff
Why a model can fit its training data perfectly and fail on new data. A polynomial fit from degree 1 to 15 shows train error falling while test error explodes, and how regularisation helps.
Overfitting is when a model learns the noise in its training data, so training error is very low but error on new data is high. The bias-variance tradeoff says simple models have high bias and low variance, flexible models the reverse, and the best model sits between. Interviewers expect the diagnosis: a large gap between training and validation error.
Overfitting is when a model learns the noise in its training data and so performs worse on new data. Underfitting is when it is too simple to capture the pattern. The bias-variance tradeoff says that simple models have high bias and low variance, complex ones the reverse, and the best model sits in between.
- Underfitting shows as high training and test error, while overfitting shows as low training error with a large gap to much higher test error.
- Fitting polynomials to 20 noisy sine points, degree 3 gave test MSE 0.096, while degree 15 gave training MSE 0.002 and test MSE 60.388.
- Expected test error splits into bias squared, variance and irreducible noise; more model flexibility lowers bias and raises variance.
- L2 (ridge) regularisation shrinks all weights smoothly, while L1 (lasso) can push some weights to exactly zero and so also selects features.
- More training data helps high-variance overfit models, but it does not help high-bias models, which need more capacity or better features.
01The goal is generalisation
Training a model means fitting its parameters to examples. But we do not care how well it remembers those examples. We care how well it predicts new ones. The gap between the two is the heart of machine learning.
A student who memorises the answers to last year's exam scores perfectly on that paper and badly on this year's. An overfit model is that student.
02Underfitting and overfitting
Underfitting: the model cannot represent the real pattern, like fitting a straight line to a curve. Training error and test error are both high.
Overfitting: the model is flexible enough to bend around every training point, including the random noise. Training error is very low and test error is high.
A quick diagnosis: compare the two errors. High train error means underfitting (add capacity, better features). A big gap between low train error and high test error means overfitting (more data, simpler model, regularisation).
03Watching it happen
import numpy as np
rng = np.random.default_rng(0)
f = lambda x: np.sin(2 * np.pi * x)
xtr = np.sort(rng.uniform(0, 1, 20)); ytr = f(xtr) + rng.normal(0, 0.25, 20)
xte = np.sort(rng.uniform(0, 1, 200)); yte = f(xte) + rng.normal(0, 0.25, 200)
for degree in (1, 3, 5, 9, 15):
c = np.polyfit(xtr, ytr, degree)
train = ((np.polyval(c, xtr) - ytr) ** 2).mean()
test = ((np.polyval(c, xte) - yte) ** 2).mean()
print(degree, round(train, 3), round(test, 3))The output:
degree train MSE test MSE
1 0.218 0.285
3 0.018 0.096
5 0.017 0.101
9 0.013 0.543
15 0.002 60.388Train error keeps falling as the degree rises. Test error falls until degree 3, stays flat at 5, then climbs: 0.543 at degree 9 and 60.4 at degree 15, a model that nails the 20 training points and gives absurd answers between them. The noise itself has variance 0.0625 (the standard deviation 0.25 squared), so no model can reliably get below that on test data. Degree 3 at 0.096 is close to that floor.
04Bias and variance
Bias is the error from a model being too simple to match reality. A straight line has high bias against a sine wave, and it would be wrong in the same way on any sample of data.
Variance is how much the fitted model changes when the training sample changes. A degree-15 polynomial changes drastically if you swap a few points. It is sensitive to the particular noise it saw.
Expected test error splits into three parts: bias squared, variance, and irreducible noise. Making a model more flexible lowers bias and raises variance. The tradeoff is finding the flexibility where their sum is smallest, around degree 3 in the example above.
05What to do about overfitting
More training data is the most reliable cure, since a flexible model needs more points to pin it down. Simplify the model: fewer features, a lower polynomial degree, a shallower tree, fewer layers.
Regularisation adds a penalty on large weights so the model prefers smoother solutions. Ridge (L2) shrinks all weights. Lasso (L1) pushes some to exactly zero, which also selects features. A strength parameter controls the trade-off, and you pick it with cross-validation.
Other tools: early stopping (halt training when validation error starts to rise), dropout in neural networks, pruning in decision trees, and ensembles such as random forests, which average many high-variance models to cancel their noise.
Always measure on data the model never trained on. Hold out a validation set for choosing settings and a separate test set for the final score, as the next topic explains.
06Common interview questions
Q: How do you detect overfitting?
A: Compare training and validation error. A large gap, with training much lower, signals overfitting. Learning curves show it as the two curves failing to converge.
Q: Explain the bias-variance tradeoff.
A: Simple models have high bias and low variance, flexible models low bias and high variance. Total error is lowest at a balance between them, plus irreducible noise.
Q: L1 versus L2 regularisation?
A: L2 shrinks weights smoothly, L1 can set weights exactly to zero, giving sparse models.
Q: Does more data always fix overfitting?
A: It helps a lot for high-variance models. It does not help high-bias models, which need more capacity or better features.
Q: Why not just pick the model with the lowest training error?
A: Training error always improves with complexity, so it rewards memorisation rather than generalisation.