What are the key differences between L1 loss and L2 loss?
Answer
L1 loss (mean absolute error) measures the average absolute difference between predictions and targets, while L2 loss (mean squared error) measures the average squared difference. That single change in the error function drives every practical difference: squaring amplifies large deviations, so L2 is more sensitive to outliers but enjoys a smooth gradient that shrinks to zero at the optimum; L1 treats all errors linearly, making it robust to outliers, but its gradient is a constant 
everywhere, so optimization can oscillate near the minimum instead of settling. When used as a regularization penalty rather than a regression loss, L1 additionally induces sparsity: it can drive the weights of uninformative features to exactly zero, performing implicit feature selection, whereas L2 only shrinks weights toward zero.
(1) Error Measure: L1 averages absolute errors 
; L2 averages squared errors 
, amplifying large deviations.
(2) Gradient Behavior: L1’s gradient is a constant (undefined at 0); L2’s gradient is proportional to the error and vanishes smoothly at the optimum.
(3) Practical Choice: L1 for outlier-robust regression and sparse models; L2 for smooth, stable optimization when Gaussian noise is a reasonable assumption.
| Feature | L1 Loss (MAE) | L2 Loss (MSE) |
|---|---|---|
| Error Calculation | Absolute difference | Squared difference |
| Outlier Sensitivity | Less sensitive | More sensitive |
| Gradient | Constant (+1 or -1) | Proportional to the error |
| Sparsity | Induces sparsity (feature selection) | Does not inherently induce sparsity |
| Optimization near minimum | Can be unstable | More stable |
Table 1: The five practical differences between L1 and L2 loss; every row follows from the choice of versus as the per-sample penalty.
Figure 1: Per-sample penalty as a function of the residual: L1 grows linearly (V-shape, robust to large errors), L2 grows quadratically (small errors are nearly free, large errors are heavily punished).
Mathematical Formulation:
Where:
is the true target and 
the model prediction for sample 
.
is the number of samples; both losses average over the dataset.
is the sign function: 
for positive residuals, 
for negative, undefined at 0 (subgradient ![[-1, 1]](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI0MyIgaGVpZ2h0PSIxNyIgdmlld0JveD0iMCAwIDQzIDE3Ij48cmVjdCB3aWR0aD0iMTAwJSIgaGVpZ2h0PSIxMDAlIiBzdHlsZT0iZmlsbDojY2ZkNGRiO2ZpbGwtb3BhY2l0eTogMC4xOyIvPjwvc3ZnPg==)
in practice).
is the true target and 
the model prediction for sample 
.
is the number of samples; both losses average over the dataset.
is the sign function: 
for positive residuals, 
for negative, undefined at 0 (subgradient ![[-1, 1]](https://s0.wp.com/latex.php?latex=%5B-1%2C+1%5D&bg=ffffff&fg=000&s=2&c=20201002)
in practice).
Figure 2: Gradient magnitude versus residual: L1’s constant gradient never decays (unstable near the optimum, undefined at zero), while L2’s gradient shrinks linearly and vanishes exactly at the optimum.
Leave a Reply