ML0074 Linear vs Non-Linear Reduction

Linear methods rotate and project: the low-dimensional representation is a linear function of the original features, as in PCA, LDA, or truncated SVD. Non-linear methods learn a curved mapping, so they can unroll manifolds that no flat projection can represent, as in t-SNE, UMAP, Isomap, and autoencoders. The price of curvature is weaker guarantees: a linear projection preserves global geometry as well as any linear map can and gives a reusable transform for new points, while neighbor-graph methods such as t-SNE and UMAP preserve local neighborhoods but distort global distances and produce no reusable mapping (autoencoders and parametric t-SNE are the exceptions).

(1) Linear Methods: closed-form, fast, invertible on the kept subspace, and out-of-sample points reuse the same projection matrix. Examples: PCA, LDA, truncated SVD.
(2) Non-Linear Methods: capture curved manifolds and cluster structure that projections cannot. Examples: t-SNE and UMAP (neighbor graphs), Isomap (geodesic distances), autoencoders (a learned parametric map).
(3) Choose by Purpose: preprocessing for a downstream model calls for a linear method; 2D visualization of clusters calls for a non-linear one. Production pipelines often chain them: 10x Genomics’ Cell Ranger reduces single-cell gene expression with PCA first, then computes t-SNE or UMAP projections on the top components for its Loupe Browser maps.

Figure 1: Why curvature matters: the horseshoe’s two ends are close in ambient space but far apart along the manifold. PCA projects along a straight axis and collapses the ends together; a non-linear method unrolls the curve and keeps the true ordering.

Mathematical Formulation:

Where:

• is an orthonormal projection matrix and the -dimensional representation of .
• is a learned map such as an autoencoder’s encoder; its parameters are fitted by gradient descent.
• t-SNE and UMAP are transductive: they output the coordinates directly without an explicit , so new points need a parametric variant or a refit.

Feature	Linear (PCA, LDA, SVD)	Non-Linear (t-SNE, UMAP, Autoencoder)
Mapping	Fixed matrix	Learned or implicit
Global Geometry	Preserved as well as a linear map allows	Distorted: cluster sizes and distances not meaningful
New Points	Apply the same	Autoencoder: yes; t-SNE/UMAP: refit or parametric variant
Failure Mode	Curved manifolds collapse	Over-read clusters; stochastic restarts differ
Best Use	Preprocessing, compression, de-correlation	Visualization, cluster discovery

Figure 2: The same clustered data through both lenses. PCA (left) keeps global spread but overlaps clusters that are not linearly separable; the non-linear embedding (right) separates the clusters, at the cost of making distances between them uninterpretable.