ML0053 Hinge Loss for SVM

The hinge loss is the key element of Support Vector Machines: it penalizes both misclassified points and correctly classified points that lie inside the margin. Points classified correctly with margin to spare get zero loss; the loss grows linearly as a point moves toward or across the decision boundary. This structure is exactly what pushes the SVM to maximize the margin, promoting robust, generalizable boundaries.

(1) Zero Loss Zone: When , the point is correctly classified and outside (or exactly on) the margin, the loss is 0.
(2) Linear Penalty Zone: When falls below 1, i.e., inside the margin or misclassified, the loss grows linearly with the violation.
(3) Convex But Not Smooth: The kink at makes the function non-differentiable there, so optimization uses subgradients instead of plain gradients.

Figure 1: Hinge loss vs the margin : flat at zero once the point is beyond the margin (right of the dashed line at 1), ramping up linearly inside the margin and for misclassified points (left of it). The kink at 1 is where subgradients take over.

Mathematical Formulation:

Where:

• is the true label.
• is the raw model output (the signed score, before any threshold).
• The product is the (functional) margin: positive means correctly classified, ≥ 1 means correct with margin.