ML0050 Logistic Regression III

Written by

Why is Mean Squared Error (L2 Loss) an unsuitable loss function for logistic regression compared to cross-entropy?

Answer

Mean Squared Error (MSE) is unsuitable for logistic regression primarily because, combined with the sigmoid, it produces a non-convex loss landscape: optimization becomes harder and convergence less reliable. It also provides weaker gradients exactly when predictions are confidently wrong, slowing learning. Cross-entropy aligns with the Bernoulli distribution assumption behind binary classification, yields a convex loss for the single-neuron binary setting, and delivers strong gradients throughout.

(1) Wrong Assumption: MSE assumes Gaussian-distributed errors, while logistic regression models a Bernoulli (binary) outcome.
(2) Non-Convex Optimization: MSE on top of the sigmoid creates a non-convex surface: gradient descent can stall in flat regions or poor local behavior.
(3) Gradient Issues: With MSE, confident wrong predictions produce tiny gradients (the sigmoid saturates), slowing learning; cross-entropy keeps the gradient strong precisely there.
(4) Interpretation: Cross-entropy directly compares predicted probabilities to true labels, the natural measure for classification.

Three dimensional MSE loss surface over weight and bias for logistic regression showing flat plateaus and a narrow steep valley

Figure 1: The MSE loss surface over $(w, b)$ for a sigmoid classifier: wide, nearly flat plateaus (vanishing gradients where the sigmoid saturates) around a narrow curved valley: a landscape that is awkward and slow for gradient descent, unlike the convex bowl cross-entropy gives.

Mathematical Formulation:
$\mathcal{L}_{\text{MSE}} = \frac{1}{n} \sum_{i=1}^{n} \big(y_i - \sigma(z_i)\big)^2$
$\mathcal{L}_{\text{CE}} = -\frac{1}{n} \sum_{i=1}^{n} \Big[ y_i \log \sigma(z_i) + (1 - y_i) \log \big(1 - \sigma(z_i)\big) \Big]$
$\frac{\partial \mathcal{L}_{\text{CE}}}{\partial z_i} = \sigma(z_i) - y_i$

Where:

$y_i \in \{0, 1\}$ is the true label and $\sigma(z_i)$ the predicted probability for sample $i$ , with score $z_i = \mathbf{w}^{\top}\mathbf{x}_i + b$ .
MSE’s gradient carries an extra factor $\sigma'(z_i) = \sigma(z_i)(1 - \sigma(z_i))$ , which vanishes when the sigmoid saturates, exactly when the prediction is confidently wrong.
Cross-entropy’s gradient $\sigma(z_i) - y_i$ is simply the prediction error: large when the model is confidently wrong, with no saturating factor.