Stats Archives - Page 2 of 2 - Interview for Machine Learning

What is the difference between probability and likelihood?

Answer

Probability and likelihood use the same joint density $p(x \mid \theta)$ but read it in opposite directions. Probability fixes the parameters $\theta$ and asks how plausible different data $x$ are; likelihood fixes the observed data and asks which parameter values would have made that data plausible. A probability distribution must sum or integrate to 1 over all possible data; a likelihood carries no such constraint over the parameters. Maximum likelihood estimation is exactly this direction flip: hold the sample fixed and choose $\theta$ to make the observed data as probable as possible.

(1) Same Formula, Two Directions: with $\theta$ fixed, $p(x \mid \theta)$ is a probability over data; with $x$ fixed at the observed sample, the same expression becomes the likelihood $L(\theta \mid x)$ , a function of $\theta$ .
(2) Normalization: probabilities integrate to 1 over all possible data, while likelihoods do not integrate to 1 over $\theta$ , so a likelihood is never “the probability of the parameter”.
(3) Where Each Is Used: probability drives prediction and simulation of unseen data; likelihood drives parameter estimation. Amazon’s DeepAR forecaster is trained by maximizing the likelihood of observed demand series, then at inference emits probability distributions over future demand.

Two panels: left, a probability density over data values with parameters fixed; right, a likelihood curve over the parameter theta with data fixed, peaking at the maximum likelihood estimate

Figure 1: The same density viewed two ways. With $\theta$ fixed, the curve over data values is a probability distribution and integrates to 1 (left). With the data fixed at the observed sample, the curve over $\theta$ is the likelihood (right), whose peak is the maximum likelihood estimate.

Mathematical Formulation:
$L(\theta \mid x) = p(x \mid \theta)$
$\hat{\theta}_{\mathrm{MLE}} = \arg\max_{\theta}\ \sum_{i=1}^{N} \log p(x_i \mid \theta)$

Where:

$x$ is the observed data and $\theta$ the model parameters.
$p(x \mid \theta)$ is the density of the data under parameters $\theta$ ; $L(\theta \mid x)$ is the same expression read as a function of $\theta$ .
$x_i$ with $i\in\{1,\ldots,N\}$ indexes independent samples, so the joint likelihood factorizes into a product and the logarithm turns it into a sum.

Feature	Probability	Likelihood
What Varies	The data $x$	The parameters $\theta$
What Is Fixed	The parameters $\theta$	The observed data $x$
Normalization	Binomial example: P(X = 3) = C(10,3)0.3^30.7^7 = 0.267	No constraint over $\theta$
Answers	“What data should I expect?”	“Which parameters fit the data I saw?”
Typical Use	Prediction, simulation, p-values	Estimation (MLE), model comparison

Tag: Stats

ML0067 Probability vs Likelihood