ML0031 Linear Regression

Written by

What are the advantages and disadvantages of linear regression?

Answer

Linear regression models the target as a weighted sum of the input features plus an intercept, fitting the weights by minimizing squared error. Its advantages make it the default baseline: it is simple and interpretable: each coefficient directly states the strength and direction of a feature’s relationship with the target; it is computationally cheap, with a closed-form solution (normal equations) and fast training even on large data; and it works well when the true relationship is approximately linear. Its disadvantages stem from the same simplicity: it assumes linearity, so it underfits genuinely non-linear relationships; it is sensitive to outliers because squared errors let extreme points dominate the fit; multicollinearity between correlated features makes coefficient estimates unstable and hard to interpret; and the model cannot capture interactions or complexity unless you explicitly engineer such features. Simple linear regression uses one feature; multiple linear regression extends the same idea to many.

(1) Interpretable & Fast: Coefficients read directly as feature effects; closed-form or cheap iterative fitting.
(2) Linearity Assumption: Underfits non-linear patterns: the model class is a hyperplane.
(3) Fragilities: Outlier-sensitive (squared loss), unstable under multicollinearity, no built-in interactions or nonlinearity.

Scatter of data points with a fitted regression line and vertical residual segments

Figure 1: Least-squares fit: the line minimizes the sum of squared vertical residuals (orange segments). Note how the one distant outlier pulls the fitted line toward itself, the sensitivity that comes with squaring errors.

Mathematical Formulation:
$h_\theta(x) = \theta_0 + \sum_{j=1}^{p} \theta_j x_j = \theta^{\top} x$
$\hat{\theta} = \arg\min_{\theta} \sum_{i=1}^{n} \big(y_i - h_\theta(x_i)\big)^2 = (X^{\top}X)^{-1} X^{\top} y$

Where:

$h_\theta(x)$ is the predicted value for feature vector $x$ .
$\theta_0$ is the intercept (bias); $\theta_j$ the weight of feature $x_j$ ; $p$ the number of features.
$X$ is the $n \times (p+1)$ design matrix and $y$ the target vector; the closed form exists when $X^{\top}X$ is invertible (no perfect multicollinearity).