ML0096 Maximum Likelihood Estimation

Maximum likelihood estimation is a method for estimating the parameters of a statistical model by finding the parameter values that make the observed data most probable. Given a dataset and a parametric model, MLE chooses the parameters that maximize the likelihood function: the joint probability of the observed data under the model. Equivalently, we maximize the log-likelihood, which converts the product of probabilities into a sum of log-probabilities, making optimization tractable and numerically stable. MLE is the foundation of most modern ML training: an ICLR 2025 blog post explicitly derives the standard classification cross-entropy loss from the MLE principle, showing that minimizing cross-entropy is equivalent to maximizing the conditional log-likelihood of the training data under the model. For a simple coin flip example, if you observe 3 heads in 4 flips, the MLE of the heads probability is simply 3/4, the sample proportion.

(1) Likelihood vs Probability: probability is the chance of data given fixed parameters; likelihood is the same function viewed as a function of parameters with the data fixed, and MLE finds the parameters that maximize it.
(2) Log-Likelihood Trick: taking the log converts the product into the sum , which is easier to differentiate, numerically stable, and decomposes additively over data points for stochastic gradient descent.
(3) Connection to Production ML: an ICLR 2025 blog derives cross-entropy loss as the negative log-likelihood under a categorical model, so training a classifier with cross-entropy is MLE; logistic regression is MLE under a Bernoulli model, and linear regression with squared error is MLE under a Gaussian noise model.

Figure 1: Coin flip MLE: with 3 heads in 4 flips, the likelihood and log-likelihood both peak at , the sample proportion of heads.

The coin flip example makes the principle concrete. Suppose you flip a coin 4 times and observe 3 heads and 1 tail. The coin has an unknown probability of heads. The likelihood of observing this data is , and the log-likelihood is . Taking the derivative and setting it to zero gives , which solves to . This is the sample proportion, and it is the MLE because the likelihood is maximized there. The same principle scales to neural networks: the cross-entropy loss is the negative average log-likelihood, and gradient descent on this loss is MLE for the network’s parameters. A 2024 arXiv paper established finite-sample guarantees for MLE in logistic regression, showing the sample complexity depends on both dimension and signal strength, with distinct regimes at different signal-to-noise ratios.

Mathematical Formulation:

Where:

• is the parameter value that maximizes the likelihood; is the probability of observation under the model with parameters .
• is the log-likelihood, which converts the product into a sum for tractability; maximizing is equivalent to maximizing the likelihood because log is monotonically increasing.
• For the coin flip: , and setting gives , the sample proportion. For classification: , and minimizing is the cross-entropy loss.