ML0055 KNN Regression

K-Nearest Neighbors (KNN) Regression is a simple, instance-based algorithm that predicts a continuous output by finding the nearest training points and averaging their target values. Its simplicity makes it easy to understand and implement, but performance is sensitive to and the distance metric, and prediction is computationally expensive on large datasets since every query compares against all training samples. Like its classification sibling, KNN regression is a “lazy learner”: it builds no explicit model and simply memorizes the training data.

(1) Instance-Based: No explicit model is learned; predictions come from stored training data and similarity.
(2) Distance-Based: Find the nearest neighbors of the query point (commonly Euclidean distance).
(3) Averaging Neighbors: The prediction is the mean of those neighbors’ target values.
(4) Sensitive To K And Metric: Small tracks noise; large oversmooths.
(5) No Training Phase: All computation happens during prediction.

Figure 1: KNN regression (K=5) on a noisy sine wave: each prediction averages the 5 nearest targets, producing a locally-flat, step-like curve that follows the trend but clips the peaks and troughs, the signature of local averaging.

Mathematical Formulation:

Where:

• is the prediction for the query point; the target of the -th nearest neighbor; the number of neighbors (first formula: uniform average).
• In the weighted variant, is inversely proportional to the distance between query and neighbor: closer points influence the prediction more.
• is a small constant avoiding division by zero when a neighbor sits exactly on the query.