Tag: Loss

ML0050 Logistic Regression III
Why is Mean Squared Error (L2 Loss) an unsuitable loss function for logistic regression compared to cross-entropy?
Answer
Mean Squared Error (MSE) is unsuitable for logistic regression primarily because, combined with the sigmoid, it produces a non-convex loss landscape: optimization becomes harder and convergence less reliable. It also provides weaker gradients exactly when predictions are confidently wrong, slowing learning. Cross-entropy aligns with the Bernoulli distribution assumption behind binary classification, yields a convex loss for the single-neuron binary setting, and delivers strong gradients throughout.
(1) Wrong Assumption: MSE assumes Gaussian-distributed errors, while logistic regression models a Bernoulli (binary) outcome.
(2) Non-Convex Optimization: MSE on top of the sigmoid creates a non-convex surface: gradient descent can stall in flat regions or poor local behavior.
(3) Gradient Issues: With MSE, confident wrong predictions produce tiny gradients (the sigmoid saturates), slowing learning; cross-entropy keeps the gradient strong precisely there.
(4) Interpretation: Cross-entropy directly compares predicted probabilities to true labels, the natural measure for classification.

Figure 1: The MSE loss surface over $(w, b)$ for a sigmoid classifier: wide, nearly flat plateaus (vanishing gradients where the sigmoid saturates) around a narrow curved valley: a landscape that is awkward and slow for gradient descent, unlike the convex bowl cross-entropy gives.
Mathematical Formulation:
$\mathcal{L}_{\text{MSE}} = \frac{1}{n} \sum_{i=1}^{n} \big(y_i - \sigma(z_i)\big)^2$
$\mathcal{L}_{\text{CE}} = -\frac{1}{n} \sum_{i=1}^{n} \Big[ y_i \log \sigma(z_i) + (1 - y_i) \log \big(1 - \sigma(z_i)\big) \Big]$
$\frac{\partial \mathcal{L}_{\text{CE}}}{\partial z_i} = \sigma(z_i) - y_i$
Where:
$y_i \in \{0, 1\}$ is the true label and $\sigma(z_i)$ the predicted probability for sample $i$ , with score $z_i = \mathbf{w}^{\top}\mathbf{x}_i + b$ .
MSE’s gradient carries an extra factor $\sigma'(z_i) = \sigma(z_i)(1 - \sigma(z_i))$ , which vanishes when the sigmoid saturates, exactly when the prediction is confidently wrong.
Cross-entropy’s gradient $\sigma(z_i) - y_i$ is simply the prediction error: large when the model is confidently wrong, with no saturating factor.
Login to view more content
June 10, 2025
ML0022 Cross Entropy Loss
Explain how Cross Entropy Loss is used for a classification task.
Answer
Cross-entropy loss (log loss) measures the distance between two probability distributions: the model’s predicted class probabilities and the true distribution (a one-hot vector for labeled data). Training a classifier means minimizing this distance, which is exactly maximum-likelihood estimation of the model parameters. In the binary case the loss averages $-[y \log p + (1-y)\log(1-p)]$ ; in the multi-class case only the true class term survives the one-hot sum, reducing the per-sample loss to $-\log p_{\text{true}}$ . The logarithm is the key design choice: it penalizes confident wrong predictions severely: as the probability assigned to the true class approaches 0 the loss explodes toward infinity, giving a strong, well-behaved gradient exactly where the model errs most. It pairs naturally with sigmoid (binary) or softmax (multi-class) output layers, whose gradients under cross-entropy take the clean form $p - y$ .
(1) Definition: Negative log-likelihood of the true labels under the predicted distribution; minimizing it is maximum-likelihood training.
(2) Confidence Sensitivity: Loss grows unboundedly as the true-class probability approaches 0; confident mistakes are punished hardest.
(3) Standard Pairing: Sigmoid output + binary cross-entropy, softmax output + categorical cross-entropy; combined gradient simplifies to $p - y$ .

Figure 1: Per-sample loss $-\log(p)$ as a function of the probability assigned to the true class: near $p = 1$ the loss is almost zero, below $p = 0.1$ it grows steeply: confident errors dominate the gradient.
Mathematical Formulation:
$\mathcal{L}_{\text{binary}} = -\frac{1}{n} \sum_{i=1}^{n} \Big[ y_i \log p_i + (1 - y_i) \log(1 - p_i) \Big]$
$\mathcal{L}_{\text{multi}} = -\frac{1}{n} \sum_{i=1}^{n} \sum_{c=1}^{C} y_{ic} \log p_{ic} = -\frac{1}{n} \sum_{i=1}^{n} \log p_{i,\text{true}}$
Where:
$n$ is the number of samples and $C$ the number of classes.
$y_i \in \{0, 1\}$ is the true binary label; $p_i$ the predicted probability of the positive class.
$y_{ic}$ is the one-hot indicator (1 if sample $i$ belongs to class $c$ ); $p_{ic}$ the predicted probability of class $c$ , so the inner sum keeps only the true-class term.
Login to view more content
March 31, 2025

ML0021 L1 Loss L2 Loss

What are the key differences between L1 loss and L2 loss?

Answer

L1 loss (mean absolute error) measures the average absolute difference between predictions and targets, while L2 loss (mean squared error) measures the average squared difference. That single change in the error function drives every practical difference: squaring amplifies large deviations, so L2 is more sensitive to outliers but enjoys a smooth gradient that shrinks to zero at the optimum; L1 treats all errors linearly, making it robust to outliers, but its gradient is a constant $\pm 1$ everywhere, so optimization can oscillate near the minimum instead of settling. When used as a regularization penalty rather than a regression loss, L1 additionally induces sparsity: it can drive the weights of uninformative features to exactly zero, performing implicit feature selection, whereas L2 only shrinks weights toward zero.

(1) Error Measure: L1 averages absolute errors $|e|$ ; L2 averages squared errors $e^2$ , amplifying large deviations.
(2) Gradient Behavior: L1’s gradient is a constant $\pm 1$ (undefined at 0); L2’s gradient is proportional to the error and vanishes smoothly at the optimum.
(3) Practical Choice: L1 for outlier-robust regression and sparse models; L2 for smooth, stable optimization when Gaussian noise is a reasonable assumption.

Feature	L1 Loss (MAE)	L2 Loss (MSE)
Error Calculation	Absolute difference	Squared difference
Outlier Sensitivity	Less sensitive	More sensitive
Gradient	Constant (+1 or -1)	Proportional to the error
Sparsity	Induces sparsity (feature selection)	Does not inherently induce sparsity
Optimization near minimum	Can be unstable	More stable

Table 1: The five practical differences between L1 and L2 loss; every row follows from the choice of $|e|$ versus $e^2$ as the per-sample penalty.

L1 absolute loss V-shape versus L2 squared loss parabola as functions of the residual

Figure 1: Per-sample penalty as a function of the residual: L1 grows linearly (V-shape, robust to large errors), L2 grows quadratically (small errors are nearly free, large errors are heavily punished).

Mathematical Formulation:
$\mathcal{L}_{L1} = \frac{1}{n} \sum_{i=1}^{n} \left| \hat{y}_i - y_i \right|$
$\frac{\partial \mathcal{L}_{L1}}{\partial \hat{y}_i} = \frac{1}{n}\,\mathrm{sign}(\hat{y}_i - y_i)$
$\mathcal{L}_{L2} = \frac{1}{n} \sum_{i=1}^{n} \left( \hat{y}_i - y_i \right)^2$
$\frac{\partial \mathcal{L}_{L2}}{\partial \hat{y}_i} = \frac{2}{n}\left(\hat{y}_i - y_i\right)$

Where:

$y_i$ is the true target and $\hat{y}_i$ the model prediction for sample $i$ .
$n$ is the number of samples; both losses average over the dataset.
$\mathrm{sign}(\cdot)$ is the sign function: $+1$ for positive residuals, $-1$ for negative, undefined at 0 (subgradient $[-1, 1]$ in practice).

Gradient of L1 loss constant step function versus gradient of L2 loss linear in the residual

Figure 2: Gradient magnitude versus residual: L1’s constant gradient never decays (unstable near the optimum, undefined at zero), while L2’s gradient shrinks linearly and vanishes exactly at the optimum.

March 29, 2025

ML0001 Loss Curve Plot
The following training loss curves were produced under four different experiment settings. Which curve most likely corresponds to a correct training setup, and what does each of the remaining curves indicate?

Figure 1: Training loss curves recorded under four experiment settings (A–D). Only one reflects healthy training.
Answer
Curve A most likely corresponds to a correct training setup: the loss drops quickly in early epochs and then flattens as the model converges, which is the signature of steady learning with a well-tuned learning rate. Curve B, where the loss increases monotonically, indicates divergence: typically a learning rate far too large or a sign error in the loss or gradient. Curve C stays flat, meaning the weights are barely updating: a near-zero learning rate, broken gradient flow, or frozen parameters. Curve D oscillates sharply, the classic symptom of a learning rate too high for stable descent, so the optimizer keeps overshooting the minimum.
(1) Healthy Curve: A smooth, rapid decrease that levels off toward a floor, showing that gradients flow and the step size is well tuned.
(2) Failure Signatures: Rising loss means divergence, flat loss means no learning, and violent oscillation means unstable steps.
(3) First Knobs To Turn: Check the learning rate first, then gradient flow (vanishing or exploding), then the loss wiring itself.
Mathematical Formulation:
$\theta_{t+1} = \theta_t - \eta \, \nabla_\theta \mathcal{L}(\theta_t)$
$\mathcal{L}_t \to \mathcal{L}_{\min} \quad \text{as} \quad t \to T \text{ for a healthy run}$
Where:
$\theta_t$ denotes the model parameters at epoch $t$ , and $T$ is the final epoch.
$\eta$ is the learning rate, the single hyperparameter behind curves B and D in most real failures.
$\nabla_\theta \mathcal{L}(\theta_t)$ is the gradient of the training loss $\mathcal{L}$ ; if it vanishes or is disconnected, the curve goes flat as in C.
$\mathcal{L}_{\min}$ is the approximate floor the loss converges to, above zero in practice because of label noise and mini-batch variance.
Login to view more content
February 23, 2025

Tag: Loss

ML0050 Logistic Regression III

ML0022 Cross Entropy Loss

ML0021 L1 Loss L2 Loss

ML0001 Loss Curve Plot