ML0051 Linear SVM

Written by

Can you explain the key concepts behind a Linear Support Vector Machine?

Answer

A Linear Support Vector Machine (Linear SVM) is a classifier that finds the optimal straight-line (or hyperplane) separating two classes by maximizing the margin between them. It relies on a few critical points (support vectors) and offers strong generalization, especially for linearly separable data.

Key Concepts of a Linear Support Vector Machine:
(1) Hyperplane: A decision boundary that separates data points of different classes.
(2) Margin: The distance between the hyperplane and the nearest data points from each class.
(3) Support Vectors: Data points that lie closest to the hyperplane and define the margin.
(4) Objective: Maximize the margin while minimizing classification errors.

Here is the Linear SVM Decision Function:
$f(\mathbf{x}) = \mathbf{w}^\top \mathbf{x} + b$
Where:
$\mathbf{x}$ is the input feature vector.
$\mathbf{w}$ is the weight vector.
$b$ is the bias term.

Here is the Linear SVM Classification Rule:
$\hat{y} = \mbox{sign}(\mathbf{w}^\top \mathbf{x} + b) = \mbox{sign}(f(\mathbf{x}))$
Where:
$\hat{y}$ is the predicted class label.
$\mbox{sign}(\cdot)$ returns +1 if the argument is ≥ 0, and −1 otherwise.

For Hard Margin SVM, here is the Optimization Objective:
$\min_{\mathbf{w}, b} \quad \frac{1}{2} \|\mathbf{w}\|^2$
Subject to:
$y_i(\mathbf{w}^\top \mathbf{x}_i + b) \geq 1 \quad \text{for all } i$
Where:
$y_i \in {-1, 1}$ is the class label for the i-th data point.
$\mathbf{x}_i$ is the i-th feature vector.