ML0052 Non-Linear SVM

A non-linear SVM classifies data that is not linearly separable by using a kernel function to implicitly project the data into a higher-dimensional space where a linear separator exists. This kernel trick provides flexibility for complex datasets while staying computationally efficient: the algorithm never computes the high-dimensional coordinates, only inner products through the kernel. The kernel choice (RBF, polynomial, sigmoid) strongly influences performance and adaptability.

(1) Kernel Trick: Replace inner products with a kernel , which measures similarity as if the data were mapped to a higher-dimensional space, where a linear separation becomes possible.
(2) Common Kernels: Polynomial (captures feature interactions of degree ), RBF/Gaussian (local similarity decaying with distance), sigmoid (imitates a neural activation).
(3) Objective: Find the margin-maximizing hyperplane in the transformed space; in the original space its image is a curved decision boundary.

Figure 1: Same data, two SVMs: the linear kernel can only slice the moons with a straight line (left), while the RBF kernel’s implicit high-dimensional map lets the boundary bend around both crescents (right).

Mathematical Formulation:

Where (polynomial, RBF, sigmoid kernels in order):

• are input vectors; scales the inner product or controls the RBF width (, with the Gaussian spread).
• is a constant (bias) term and the polynomial degree.
• is the squared Euclidean distance: nearby points get RBF similarity near 1, distant points near 0.

Figure 2: The kernel idea made concrete: in 1-D the blue class sits between the oranges and no single threshold separates them; after the explicit lift the classes part vertically and one straight line suffices. Kernels compute as if this lift happened, without ever forming the new coordinates.