Can you explain the bias-variance tradeoff?
Answer
The bias-variance tradeoff decomposes a model’s expected prediction error into three parts: squared bias, variance, and irreducible noise. Bias is the error from overly simplified assumptions: a high-bias model misses the real pattern and underfits. Variance is the error from sensitivity to the particular training sample: a high-variance model wiggles to fit noise and overfits. The tradeoff arises because increasing model complexity typically decreases bias but increases variance, while simplifying does the reverse: total error as a function of complexity is U-shaped, and the goal is the sweet spot that minimizes the sum. Practically, high bias shows as large training error; high variance shows as a large gap between training and validation error, and each has its own remedies (more capacity/features for bias; more data, regularization, or simpler models for variance).
(1) Bias: Error from wrong assumptions: underfitting, poor fit on both train and test data.
(2) Variance: Error from sample sensitivity: overfitting, big train/test gap.
(3) Tradeoff: Complexity trades one for the other; total error 
is U-shaped: minimize the sum.
Figure 1: The classic tradeoff curve: bias² falls and variance rises as complexity grows; total error is their U-shaped sum, and the best model sits at the minimum, not at maximum complexity.
Mathematical Formulation:
![\mathbb{E}\big[(y - \hat{f}(x))^2\big] = \underbrace{\big(\mathbb{E}[\hat{f}(x)] - f(x)\big)^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}\big[(\hat{f}(x) - \mathbb{E}[\hat{f}(x)])^2\big]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{noise}}](https://s0.wp.com/latex.php?latex=%5Cmathbb%7BE%7D%5Cbig%5B%28y+-+%5Chat%7Bf%7D%28x%29%29%5E2%5Cbig%5D+%3D+%5Cunderbrace%7B%5Cbig%28%5Cmathbb%7BE%7D%5B%5Chat%7Bf%7D%28x%29%5D+-+f%28x%29%5Cbig%29%5E2%7D_%7B%5Ctext%7BBias%7D%5E2%7D+%2B+%5Cunderbrace%7B%5Cmathbb%7BE%7D%5Cbig%5B%28%5Chat%7Bf%7D%28x%29+-+%5Cmathbb%7BE%7D%5B%5Chat%7Bf%7D%28x%29%5D%29%5E2%5Cbig%5D%7D_%7B%5Ctext%7BVariance%7D%7D+%2B+%5Cunderbrace%7B%5Csigma%5E2%7D_%7B%5Ctext%7Bnoise%7D%7D&bg=ffffff&fg=000&s=3&c=20201002)
Where:
is the true relationship and 
the model’s prediction; expectations are over training sets.
measures how far the average model is from the truth; 
how much predictions scatter around that average.
is the irreducible error: noise in the data itself that no model can eliminate.
is the true relationship and 
the model’s prediction; expectations are over training sets.
measures how far the average model is from the truth; 
how much predictions scatter around that average.
is the irreducible error: noise in the data itself that no model can eliminate.
Figure 2: The tradeoff on real-shaped data: the high-bias model is too rigid to follow the curve (both errors high); the high-variance model chases every noisy point (train error low, test error high); the balanced model tracks the true function and minimizes test error.
Leave a Reply