ML0094 GMM vs K-Means

A Gaussian Mixture Model is a probabilistic clustering model that represents the data as a weighted sum of K Gaussian distributions, each with its own mean, covariance, and mixing weight. Where k-means assigns each point to exactly one cluster (hard assignment) by nearest centroid, a GMM assigns each point a responsibility (posterior probability) for each cluster (soft assignment), so a point on the boundary between two clusters can be 60% in one and 40% in another. K-means is the limiting case of a GMM with shared spherical covariances as that shared variance goes to zero, which collapses the E-step posterior to an argmax. The GMM is fit by Expectation-Maximization: the E-step computes responsibilities, the M-step updates means, covariances, and mixing weights, iterating until convergence. Production systems use GMMs where uncertainty matters: joint diarization and separation systems pair a complex Angular Central Gaussian Mixture Model for source separation with a von Mises-Fisher mixture for diarization, so overlapping speech can be attributed to multiple speakers at once, and Flash-GMM (2026) made GMM training viable at 100x larger scale via fused Triton kernels.

(1) Hard vs Soft Assignment: k-means assigns each point to one centroid (argmin distance); GMM assigns a probability vector (responsibilities) across all clusters, so border points express uncertainty instead of being forced into one group.
(2) Spherical vs Elliptical Clusters: k-means assumes spherical, equally sized clusters (isotropic variance); GMM allows each cluster to have its own full covariance matrix, capturing elliptical and differently sized clusters.
(3) Production Soft Clustering: speaker diarization uses cACGMM and vMFMM mixture models so frame-level embeddings can belong to multiple speakers during overlap, and Flash-GMM’s fused Triton kernel achieved 20x speedup and 100x larger dataset scale, making soft GMM clustering a viable drop-in for k-means in approximate nearest-neighbor search with 2-12 point recall@10 gains.

Figure 1: k-means forces each point into one cluster via nearest-centroid (hard assignment with spherical Voronoi cells), while a GMM assigns posterior probabilities (soft assignment) and can model elliptical clusters with per-component covariance matrices.

The EM algorithm for GMMs iterates two steps. In the E-step, each point’s responsibility for cluster k is computed as the posterior . In the M-step, the parameters are updated: the mean is the responsibility-weighted average, the covariance is the responsibility-weighted scatter, and the mixing weight is the average responsibility. Each iteration is guaranteed never to decrease the log-likelihood, but it converges only to a local optimum, so multiple random restarts are standard. The key trade-off versus k-means is cost: k-means is O(nKd) per iteration with a simple distance computation, while GMM is O(nKd^2) per iteration because of the multivariate Gaussian PDF with full covariance, plus the matrix inversion in the PDF. Flash-GMM (2026) addressed this by eliminating the full responsibility matrix from GPU memory, reducing memory from O(nK) to O(KD) and enabling 100x larger datasets on a single GPU.

Figure 2: The EM loop for GMMs: the E-step computes soft responsibilities from current parameters, the M-step updates means, covariances, and mixing weights from responsibility-weighted statistics, and the cycle repeats until log-likelihood converges.

Mathematical Formulation:

Where:

• is the mixing weight of cluster (prior probability, sums to 1), is its mean, and is its covariance matrix.
• is the responsibility of cluster for point : the posterior probability that point was generated by component . This is the soft assignment that k-means replaces with a hard argmax.
• The M-step updates are weighted by responsibilities: is the responsibility-weighted mean, is the responsibility-weighted covariance, and is the average responsibility. k-means is the limit when all and , collapsing responsibilities to 0 or 1.