DL0186 Energy-Based Models and Contrastive Divergence

An energy-based model refuses to parameterize a normalized density directly. It learns only a scalar energy function , low for plausible configurations and high for implausible ones, and recovers a density by exponentiating and normalizing. The price of that freedom is the partition function , an integral over the whole input space that is intractable for anything beyond a toy problem. Differentiating the log-likelihood splits into a positive phase that lowers energy at observed data and a negative phase that raises energy wherever the model currently places mass, and the negative phase is an expectation under that normally needs MCMC samples at equilibrium. Contrastive divergence keeps the same two-phase update but initializes the chain at the training data rather than at noise, then runs only transition steps, in practice . The estimate is biased because the -step distribution is not the equilibrium one, yet the bias shrinks near the optimum where the data distribution is already close to a fixed point of the sampler, and the variance is far lower than that of a long chain.

(1) Energy, Not Probability: the network outputs one unconstrained scalar per input, so any architecture works and no normalization layer or invertibility constraint is required.
(2) The Partition Function Is The Whole Problem: is a sum or integral over every possible configuration, so exact likelihood and exact sampling are both out of reach.
(3) Positive And Negative Phase: the exact gradient is a difference of two expectations of , one under the data and one under the model, which is why training is often described as pushing energy down at real points and up at fantasy points.
(4) Chain Started At The Data: CD replaces the equilibrium expectation with samples from a chain seeded at , which already sits in a high-probability region, so a very short run produces a usable contrast.
(5) Truncation Trades Bias For Cost: per-update cost drops from a full mixing time to transitions, at the cost of a systematically biased gradient that ignores model mass far from the data.
(6) CD Is An Update Rule, Not A True Gradient: the CD-1 update is not the exact gradient of any objective function, so convergence guarantees are weak and divergence is possible in principle.

Figure 1: The two-phase update. Exact maximum likelihood would draw the negative sample from the model at equilibrium, costing a full chain run per parameter update. CD keeps the identical update form but seeds the chain at the data and stops after steps, which turns an expensive unbiased gradient into a cheap biased one.

Mathematical Formulation:

Where:

• is the model density and the learned energy, with lower energy meaning higher probability.
• is the partition function, an integral over the full input space (a sum for discrete ) that depends on and therefore cannot be ignored during training.
• is a configuration and the parameters of the energy network.
• is the average negative log-likelihood, so the first expectation is the positive phase and the second is the negative phase.
• The identity is what converts an intractable normalizer into an expectation, which is exactly why sampling becomes the bottleneck.

The Contrastive Divergence Approximation:

Where:

• is the empirical distribution used as the chain initialization, and is the distribution after applications of the transition operator.
• is any transition operator leaving invariant, typically block Gibbs for an RBM or Langevin dynamics for a continuous EBM.
• is the truncation depth, with the standard choice and larger reducing bias at linear cost.
• As we recover and the update becomes the exact likelihood gradient, so CD is a controllable interpolation between a free estimate and an unbiased one.
• The two-KL form is the original motivation rather than an exact objective; a small ignored term involving is what makes CD not the gradient of any function.

The restricted Boltzmann machine is the case where every piece is closed-form, which is why CD was invented there. Because there are no visible-visible or hidden-hidden couplings, the conditionals factorize completely and one Gibbs sweep is a single pass of matrix multiplies plus sigmoids, following the chain v → h → v’ → h’. CD-1 then reduces to the difference between a data correlation and a reconstruction correlation, computable with two forward passes and no backpropagation through a sampler.

Binary RBM Case:

Here is the data vector, , and come from one further Gibbs sweep. The positive term raises the weight between co-active data units and the negative term subtracts the same statistic measured on the model’s one-step reconstruction, so learning stops precisely when the reconstruction statistics match the data statistics.

Figure 2: The structural blind spot. A -step chain started at stays inside the data basin, so the negative phase only ever penalizes points the model already ranks near the data. A spurious mode behind an energy barrier keeps its low energy indefinitely because CD never samples there, and this is why CD-trained models can assign high likelihood to garbage.

The practical consequences follow directly from that picture. Slow mixing is not a nuisance but the defining constraint, since the time to cross a barrier grows exponentially in its height, so raising from 1 to 20 buys much less than the cost suggests. Persistent CD attacks the same problem differently by keeping a set of chains alive across parameter updates instead of restarting them at data, which lets the fantasy particles drift into spurious regions given enough updates, at the cost of instability when the learning rate is large relative to the mixing rate. Modern continuous EBMs use short-run Langevin dynamics with a replay buffer plus gradient clipping and energy regularization, and increasingly avoid MCMC entirely through score matching or its denoising variant, which never touches because the score is independent of the normalizer.