Tag: NN

DL0202 RL Production Recommendation

What are the challenges of using Reinforcement Learning in a production recommendation system, such as the YouTube home feed or Spotify’s carousels?

Answer

The hard part is almost never the learning algorithm. A production recommender is an off-policy problem with a combinatorial action space, a corrupted reward signal, and a closed feedback loop, and each of those violates an assumption that textbook RL takes for granted. You cannot explore freely on paying users, so the only training data is what yesterday’s serving policy logged, which makes importance weighting and its variance the central engineering problem rather than a footnote. The action is not one item but a slate of K items drawn from a catalogue of order $10^{8}$ , so enumerating a Q value per action is impossible and the policy must be factorized or decomposed. Finally, the reward you actually want, long-term satisfaction, arrives days after the action, and is confounded by position bias, seasonality, and the recommendations themselves.

(1) Off-Policy By Construction: training data comes from a logging policy $\beta$ , not from the policy you are optimizing, so every gradient needs an importance correction whose variance grows with the mismatch.
(2) Combinatorial Slate Actions: a page of $K$ items from $N$ candidates gives $\binom{N}{K}$ actions, forcing decompositions such as top-K correction or SlateQ‘s per-item value factorization.
(3) Reward Specification And Delay: clicks are immediate and gameable, retention is the real objective and is observed days later, so credit assignment spans sessions rather than steps.
(4) Feedback Loops And Non-Stationarity: the deployed policy shapes tomorrow’s training distribution, so a small early bias compounds instead of averaging out, and the catalogue and user base drift underneath the model.
(5) Offline Evaluation Is Unreliable: off-policy estimators are high variance on large action spaces, so offline value gains routinely fail to reproduce in an A/B test.
(6) Infrastructure Cost: you must log propensities at serving time, keep the randomization budget small, and still return a ranked page inside a tens-of-milliseconds latency budget.

Closed-loop diagram with four boxes in a row: serving policy pi-theta with retrieval and ranker over a 10^8 item catalogue, slate of K items shown to the user where position bias enters, logged feedback carrying the action, the propensity beta and a delayed reward, and off-policy training with IPS weights, clipping and top-K correction, with a return arrow deploying new policy weights back to serving

Figure 1: The loop is the problem. Supervised ranking treats the log as a fixed dataset, but here the policy under training generates its own future training distribution, so bias compounds across deployments and the reward attached to each logged action is both delayed and confounded by presentation.

The standard production recipe, popularized by YouTube’s top-K REINFORCE recommender, is to keep policy-gradient learning but reweight each logged trajectory by the ratio between the new policy and the logging policy. Two corrections then appear. The first is weight clipping, which caps the influence of any single log line and trades unbounded variance for a controlled downward bias. The second is the top-K correction factor, which accounts for the fact that the system shows $K$ slots rather than sampling one item, and which flattens the gradient on items the policy already places with near certainty. Both are variance-control devices, and understanding why they are needed is the difference between a candidate who has read the paper and one who has shipped the system.

Mathematical Formulation:
$J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}}\left[\sum_{t=0}^{T} \gamma^{t} r_{t}\right]$
$w_t = \dfrac{\pi_{\theta}(a_t \mid s_t)}{\beta(a_t \mid s_t)}$
$\tilde{w}_t = \min(w_t, c)$
$\lambda_K(s_t, a_t) = K\left(1 - \pi_{\theta}(a_t \mid s_t)\right)^{K-1}$
$\hat{g} = \sum_{t} \tilde{w}_t\, \lambda_K\, R_t\, \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t)$
$\mathrm{ESS} = \dfrac{\left(\sum_i w_i\right)^{2}}{\sum_i w_i^{2}}$

Where:

$J(\theta)$ is the expected discounted return of the target policy over a user trajectory $\tau$ , with discount $\gamma \in (0,1]$ and per-step reward $r_t$ .
$s_t$ is the user state (interaction history and context), $a_t$ the item actually shown and logged, and $R_t$ the return following that action.
$\pi_{\theta}$ is the target policy and $\beta$ the logging (behavior) policy; $w_t$ is their ratio, and it is undefined wherever $\beta(a \mid s) = 0$ .
$c$ is the clipping threshold; $\tilde{w}_t$ is the truncated weight, which is biased but bounded.
$\lambda_K$ is the top-K correction for showing $K$ slots instead of one; it tends to $K$ for rare items and to $0$ as $\pi_{\theta}(a \mid s)$ approaches $1$ .
$\mathrm{ESS}$ is the effective sample size over $i \in \{1,\ldots,N\}$ logged records, and it is the number that decides whether a 10M-row log is really 10M rows.

Log-scale line chart of effective sample size fraction versus policy mismatch sigma, showing an unclipped importance-sampling curve collapsing from near one to about one thousandth as sigma grows to three, a curve clipped at one hundred decaying more slowly, and a curve clipped at ten still retaining several percent, with an annotation marking that at sigma equal to two the unclipped estimator retains about two percent of the logged rows

Figure 2: Why off-policy learning is a variance problem before it is an optimization problem. As the target policy drifts from the logging policy, the effective sample size of the log collapses exponentially, so a huge log can carry the statistical weight of a small one. Clipping restores usable sample size at the price of a systematic downward bias on exactly the actions the new policy likes most.

This is also why the choice between supervised ranking, a contextual bandit, and full sequential RL is a real design decision rather than a question of ambition. A bandit captures most of the exploration benefit with a single-step correction and no long-horizon credit assignment, and it is what Spotify’s explainable-recommendation work uses for carousel and explanation selection. Full RL earns its cost only when an action genuinely changes the future state, for example when promoting a new creator today changes what the user is willing to watch next month.

Property	Supervised ranker	Contextual bandit	Full sequential RL
Objective	Predict the immediate label, click or watch probability	Maximize immediate reward under uncertainty	Maximize discounted return across future sessions
Data required	Logged impressions and labels only	Logged action plus its propensity	Full trajectories with propensities and delayed rewards
Correction needed	None in principle, though position bias still needs debiasing	Single-step IPS or doubly robust	Per-step IPS, clipping, top-K factor, bootstrapped value targets
Exploration cost	None, and therefore no counterfactual coverage	Small randomized slot, epsilon-greedy or Thompson sampling	Randomization plus pessimism penalties for out-of-support actions
Dominant failure	Myopic optimization that rewards clickbait	No credit for downstream session value	Variance explosion offline and feedback-loop drift online

August 21, 2026

DL0197 Continuous Incremental Online Learning

What is continuous learning after model deployment, and how do incremental learning and online learning differ in their handling of new data streams and catastrophic forgetting?

Answer

Continuous learning is the practice of keeping a deployed model current as the serving distribution drifts, by updating its parameters from the live stream instead of freezing the artifact that passed offline evaluation. The umbrella covers three regimes that differ in how much data one update sees and how often it ships: periodic batch retraining on the accumulated corpus, incremental learning that resumes from the current weights on a new chunk or task, and online learning that takes one low-learning-rate step per example or micro-batch and never revisits it. Incremental learning still controls the update: it can mix a replay sample of old data into every batch, run a few epochs over the chunk, and gate the result behind a shadow evaluation, so forgetting is a tunable quantity. Online learning gives up that control by construction, because each example is seen once, in arrival order, with no shuffling and no held-out replay, which makes the gradient sequence strongly non-i.i.d. and makes catastrophic forgetting the default rather than the exception. The core tension in both is the stability-plasticity dilemma: enough plasticity to absorb today’s traffic, enough stability to keep yesterday’s competence.

(1) Why Forgetting Happens: gradients computed only on new data are free to move weights that encoded the old distribution, since nothing in the loss references it. Breaking the i.i.d. sampling assumption of SGD is the mechanism, not model capacity.
(2) Incremental Learning Batches The Stream: updates operate on a chunk, task, or day of data, so replay ratios, epoch counts, and evaluation gates are all design knobs.
(3) Online Learning Streams One Pass: a single step per example, bounded memory, and no revisits, which buys seconds-level freshness and pays with instability and high variance.
(4) Three Families Of Mitigation: rehearsal (replay buffers, generative replay), regularization (EWC, distillation from the previous checkpoint), and parameter isolation (adapters, LoRA branches, masks).
(5) Forgetting Is Measurable: report per-task retention and average accuracy from the accuracy matrix, never a single aggregate on fresh traffic, which hides collapse on older slices.
(6) The Dual Failure Is Loss Of Plasticity: a model updated too conservatively for months stops learning at all, so stability alone is not the objective.

Diagram with a live traffic box on the left feeding three horizontal lanes: batch retraining on weeks of shuffled logged data with full offline evaluation and weekly deploys, incremental learning on a new chunk plus a replay sample with a few epochs from current weights and hourly to daily shadow-gated deploys, and online learning taking a single low-learning-rate step per example with learning-rate clipping, canary and rollback guardrails shipping in seconds

Figure 1: The gradient step is identical in all three lanes. What changes is how much data one update sees, how often it ships, and what protects the old distribution: shuffling over the full corpus, an explicit replay sample plus a shadow gate, or nothing but a small learning rate and a rollback button.

In production the choice is usually driven by label latency rather than by an appetite for novelty. Recommendation and ad ranking get feedback in seconds and genuinely benefit from near-online updates, so they run streaming updates on the embedding tables and the last layers while the backbone is refreshed on a slower incremental schedule. Fraud, credit, and medical models often wait days or weeks for a trustworthy label, so a per-example update would be trained on noise; incremental daily or weekly chunks with a replay mix are the safer default. The practical recipe that survives contact with traffic is rarely exotic: keep a stratified replay buffer covering old slices, mix roughly 5 to 20 percent of every batch from it, use a small re-warmed learning rate, and gate every candidate on a fixed regression suite of historical slices before promotion. Parameter isolation is attractive when tasks are known and separable, because a per-task adapter cannot be overwritten, but it grows parameters linearly in the number of tasks and needs task identity at inference unless the routing is learned.

Mathematical Formulation:
$\theta_t = \theta_{t-1} - \eta_t \nabla \ell(\theta_{t-1}; z_t)$
$B_t = \alpha B^{new}_t + (1-\alpha) B^{replay}_t$
$\mathcal{L}(\theta) = \mathcal{L}_{new}(\theta) + \lambda \Omega(\theta)$
$\Omega(\theta) = \tfrac{1}{2}\sum_i F_i (\theta_i - \theta^{*}_i)^2$
$A_K = \frac{1}{K}\sum_{j=1}^{K} a_{K,j}$
$f_j = \max_{k} a_{k,j} - a_{K,j}$

Where:

$\theta_t$ are the parameters after the $t$ -th update, $\eta_t$ the step size, and $z_t = (x_t, y_t)$ the arriving example; the pure online case uses each $z_t$ exactly once.
$B_t$ is the batch actually used at step $t$ , built from fresh data $B^{new}_t$ and a buffer sample $B^{replay}_t$ , with replay ratio $1-\alpha$ typically in the 0.05 to 0.20 range.
$\mathcal{L}_{new}$ is the loss on the incoming chunk and $\Omega$ a stability penalty anchored at the previous checkpoint $\theta^{*}$ , with $\lambda$ trading plasticity for retention.
$F_i$ is the diagonal Fisher information for parameter $i$ under the old task, so weights the old task depended on move less; setting $F_i = 1$ reduces the penalty to plain L2-to-previous.
$a_{k,j}$ is accuracy on task $j$ measured after training through task $k$ , with $j$ and $k$ indexing the $K$ chunks seen so far.
$A_K$ is average accuracy over everything seen and $f_j$ the forgetting on task $j$ , the drop from its best-ever value to its current one.

Figure 2: Sequential updates without rehearsal do not degrade gracefully, they collapse: task 1 loses 69 accuracy points by task 10. A 5 percent replay buffer recovers most of the gap to joint retraining at a fraction of the compute, while a stability penalty alone lands in between because it also suppresses learning on the new chunk.

Property	Batch retraining	Incremental learning	Online learning
Data per update	The full accumulated corpus, shuffled	One chunk, day, or task plus a replay sample	One example or micro-batch, in arrival order
Passes over data	Many epochs, i.i.d. sampling holds	A few epochs inside the chunk, old data only via buffer	Exactly one, no revisits
Freshness	Days to weeks behind traffic	Hours to a day behind	Seconds to minutes behind
Forgetting exposure	None by construction, old data is in every epoch	Bounded and tunable through replay ratio and penalty weight	High, the update sees only the current regime
Main safeguard	Full offline evaluation before promotion	Replay buffer, EWC or distillation, shadow eval on old slices	Small clipped learning rate, canary traffic, instant rollback
Dominant failure	Staleness under drift, and rising retraining cost	Buffer becomes unrepresentative, or the penalty freezes learning	Label noise and feedback loops steer the model within hours
Good fit	Slow drift, regulated or audited models	New domains, languages, or product surfaces arriving over time	Fast implicit feedback such as clicks, prices, or trending content

August 21, 2026

DL0189 KAN vs MLP: Kolmogorov-Arnold Networks

How does a KAN (Kolmogorov-Arnold Network) differ from a traditional MLP in its use of learnable splines on edges instead of fixed activations on nodes, and what are the trade-offs in expressivity, interpretability, and scaling?

Answer

An MLP puts learnable scalars on the edges and a fixed nonlinearity on the nodes: every edge is one number in a weight matrix, and every hidden unit applies the same hand-chosen $\sigma$ . A KAN swaps those two roles. Every edge carries its own learnable univariate function $\phi_{l,j,i}$ , implemented as a B-spline of grid size $G$ and order $k$ plus a SiLU residual branch, and every node does nothing but sum its incoming edge outputs. Both families are universal approximators, so the difference is not what can be represented but where the capacity sits and what that placement costs. Moving capacity onto the edges buys a per-edge object you can plot, sparsify, prune, and even snap to a symbolic formula, and it buys a fast spline approximation rate on smooth low-dimensional targets. It costs a factor of about $(G+k)$ in parameters at the same layer shape, roughly an order of magnitude in wall-clock training time because per-edge splines do not collapse into one dense GEMM, and a new hyperparameter (the spline grid) that must cover the actual range of the activations.

(1) Role Swap, Not A New Theorem: the Kolmogorov-Arnold representation theorem motivates the design, but the depth-2 form it guarantees can require pathological inner functions, so KANs generalize it to arbitrary depth and width and rely on smoothness of the target, not on the theorem, for their advantage.
(2) Edge Function Is Spline Plus Residual: each $\phi$ is a learned combination of $G+k$ basis functions added to a scaled SiLU, so a KAN is not activation-free; the fixed nonlinearity survives as a residual path that keeps gradients alive outside the grid.
(3) Nodes Are Pure Summation: no elementwise nonlinearity between layers, which is exactly why the univariate curves are individually meaningful.
(4) Parameter Cost Multiplies: a $64 \times 64$ layer holds 4,160 parameters as an MLP and about 32,768 spline coefficients as a KAN with $G=5, k=3$ .
(5) Interpretability Is A Workflow: L1 plus entropy regularization on the edge functions, pruning of dead edges, then symbolic snapping of each surviving curve to a candidate like $\sin$ , $\exp$ , or $x^2$ .
(6) Scaling Is The Weak Point: per-edge spline evaluation is memory-bound and GEMM-unfriendly, and at matched parameters and FLOPs an MLP still wins on vision, language, and audio benchmarks.

Side-by-side diagram of the same two-input toy layer. On the left an MLP sends two inputs through scalar weights w1 and w2 into a node containing a fixed sigma, then to the output y. On the right a KAN sends each input through a box containing a learnable spline curve drawn over knot ticks, and both spline outputs feed a node that only sums, producing y.

Figure 1: The role swap in its smallest form. In the MLP the edge is a single learnable number and the nonlinearity is a fixed $\sigma$ baked into the node; in the KAN the edge is a learnable spline over a knot grid and the node only adds. Capacity moves from a matrix of scalars to a grid of spline coefficients, which is why the same layer shape costs about $(G+k)$ times more parameters.

Expressivity behaves differently in the two regimes that matter in practice. On smooth, low-dimensional, compositional targets (symbolic regression, ODE and PDE solution operators, small physical laws) the spline basis is close to the right basis, so error falls quickly with parameters and grid extension lets you refine an already trained model by re-fitting a finer grid instead of restarting. On high-dimensional perception data the picture inverts: a controlled comparison at matched parameters and FLOPs found MLPs ahead on machine-vision, language, and audio tasks, with KANs winning only on symbolic formula representation, and KANs forgetting more than MLPs in a standard class-incremental setting. Interpretability is the more robust claim. Because a node only sums, each edge curve is a genuine one-dimensional function of one variable, so you can plot all of them, drive most toward zero with sparsity penalties, prune the graph down to a handful of edges, and read off a formula. That workflow is what made KANs useful as a scientific assistant rather than as a general drop-in replacement for a dense layer.

Mathematical Formulation:
$\mathrm{MLP}(x) = W_L \sigma(W_{L-1} \cdots \sigma(W_1 x))$
$f(x) = \sum_{q=1}^{2n+1} \Phi_q \left( \sum_{p=1}^{n} \phi_{q,p}(x_p) \right)$
$x_j^{(l+1)} = \sum_{i=1}^{n_l} \phi_{l,j,i}\left(x_i^{(l)}\right)$
$\phi(x) = w_b \, \mathrm{silu}(x) + w_s \sum_{m=1}^{G+k} c_m B_m(x)$
$P_{\mathrm{MLP}} = O(L n^2)$
$P_{\mathrm{KAN}} = O(L n^2 (G+k))$
$\ell \propto N^{-4}$

Where:

$x$ is the input vector and $x_j^{(l+1)}$ the $j$ -th activation of layer $l+1$ , obtained by summation only in a KAN.
$W_l$ and $\sigma$ are the MLP’s learnable weight matrices and its fixed elementwise nonlinearity.
$\phi_{q,p}$ and $\Phi_q$ are the inner and outer univariate functions of the Kolmogorov-Arnold representation, with $n$ the input dimension; $\phi_{l,j,i}$ is the learnable edge function from unit $i$ of layer $l$ to unit $j$ of layer $l+1$ .
$B_m$ are B-spline basis functions with local support over the knot grid, $c_m$ their learned coefficients, and $w_b, w_s$ the scales of the SiLU residual and the spline branch.
$G$ is the number of grid intervals and $k$ the spline order, so each edge holds $G+k$ coefficients; typical values are $G=5, k=3$ .
$L$ is depth, $n$ the layer width, and $P$ the parameter count, so the KAN pays the extra factor $(G+k)$ at identical layer shape.
$\ell$ is test error against parameter count $N$ ; the $N^{-4}$ rate is the cubic-spline approximation rate and holds only when the target is smooth and effectively low-dimensional.

Two-panel chart. Left panel: grouped bars comparing an MLP baseline at one times against a KAN, showing eight times the parameters at the same layer shape and about ten times the training time per epoch at the same parameter count. Right panel: log-log plot of test error against parameter count, with a steep KAN curve following an N to the minus four rate marked by grid-extension points at G equals 3, 5, 10 and 20, and a much shallower dashed MLP curve following an N to the minus one rate.

Figure 2: Cost against payoff. At matched layer shape a KAN with $G=5, k=3$ carries about 8x the parameters, and at matched parameter count it trains roughly 10x slower because each edge evaluates its own spline instead of joining one dense matmul. The right panel shows the regime where that price buys something: on a smooth low-dimensional target, grid extension (3 → 5 → 10 → 20) walks the same trained model down a steep spline-approximation curve that an equally sized MLP does not follow.

Property	MLP	KAN
Nonlinearity location	Fixed sigma on every node	Learnable phi on every edge; nodes only sum
Learnable object per edge	One scalar weight	G+k spline coefficients plus base and spline scales
Params for a 64 to 64 layer	4,160	About 32,768 at G=5, k=3 (about 41k with the two scales)
Hardware behaviour	One dense GEMM, cuBLAS and tensor-core friendly	Per-edge basis evaluation, memory-bound, about 10x slower at matched params
Input domain requirement	None; sigma is defined on all of R	Knot grid must cover the activation range, so grid updates or normalization are mandatory
Interpretability route	Inspect weights or use post-hoc attribution; features stay entangled	Plot each curve, sparsify, prune, snap to a symbolic form
Where it wins	Vision, language, audio at matched params and FLOPs; anything throughput-bound	Symbolic regression, small smooth scientific targets, operator learning
Main failure mode	Activation choice is a fixed prior; little internal structure to read	Grid hyperparameters, slow training, and worse forgetting on class-incremental benchmarks

August 20, 2026

DL0188 Liquid Neural Networks

What are Liquid Neural Networks (LNNs), and how do their continuous-time ODE-based dynamics with time-varying parameters differ from fixed-weight RNNs and Transformers?

Answer

A Liquid Neural Network is a recurrent model whose hidden state is defined by an ordinary differential equation in continuous time rather than by a fixed-stride update rule, and whose effective time constant is computed from the current input and state instead of being frozen at training time. The canonical instance is the Liquid Time-Constant (LTC) network, where a neuron leaks toward a resting value at rate $1/\tau$ and is simultaneously driven by a nonlinearity $f(x, I; \theta)$ that also appears in the denominator of the effective time constant, so each neuron can behave like a fast detector on one input and a slow integrator on another. The trained parameters $\theta$ are static after training; what is “liquid” is the input-dependent time constant and synaptic drive, which makes the system a family of dynamics selected on the fly rather than one fixed linear-time-invariant filter. Because the state is a function of physical time, the elapsed interval $\Delta t$ is an explicit argument to a numeric solver, which is what lets LNNs consume irregularly sampled sensor streams without imputation. Against this, a gated RNN applies the same learned transition once per token at an implicit constant stride, and a Transformer has no recurrent state at all: it mixes tokens by attention over the whole prefix, paying $O(L^2)$ prefill and carrying a linearly growing KV cache.

(1) Continuous-Time State: the hidden vector is the solution of an ODE evaluated by a solver, so the model interpolates between observations instead of skipping to the next index.
(2) Liquid Time Constant: $\tau_{sys} = \tau / (1 + \tau f)$ varies per neuron per step, giving input-dependent timescales from one fixed parameter set.
(3) Elapsed Time Is An Input: unequal gaps between samples are handled exactly by integrating over $\Delta t$ , which matters for event streams, medical records, and asynchronous sensors.
(4) Sparse Auditable Wiring: Neural Circuit Policies wire LTC neurons in a sensory, inter, command, motor hierarchy, and a 19-neuron command layer has flown real drones and driven real cars.
(5) Cost Scales With Solver Steps: compute is $O(L k d^2)$ for $k$ solver steps per observation, with no KV cache, but the recurrence is inherently sequential.
(6) Provable Boundedness: the LTC construction bounds both the state and $\tau_{sys}$ , so the dynamics cannot blow up the way an unconstrained neural ODE can.

Flow diagram of a liquid time-constant cell: an irregularly sampled input stream feeds a nonlinear drive function f of state and input, which both drives the ODE state x of t and, through a lower box, sets the liquid time constant tau over one plus tau times f; the state is integrated by an ODE solver taking k fused Euler steps across each delta t, then read out linearly, with a dashed feedback arrow returning the state at the next observation and a timeline below showing unequally spaced observation ticks with grey internal solver steps between them

Figure 1: One nonlinearity does two jobs. The same $f(x, I; \theta)$ supplies the drive toward the resting potential $A$ and sets the effective time constant, and the solver spans whatever $\Delta t$ the sensor happened to deliver, so the grey internal steps, not the sequence index, determine the compute bill.

The contrast with a fixed-weight RNN is easiest to see through the timescale. An LSTM or GRU also has a forget gate that modulates memory retention, but the gate acts on a fixed discrete stride: the model learns one retention profile for “one step” and has no representation of how much real time a step covered. If your samples arrive at 3 ms and then at 400 ms, the gated RNN sees two identical steps unless you hand it the gap as an extra feature and hope it learns the exponential relationship. The LTC instead solves for the exponential, because $\Delta t$ enters the integrator directly. The contrast with a Transformer is structural rather than temporal. Attention is a stateless global mixer: it can reach any earlier token in one hop, which is why it dominates language, but it has no notion of physical duration beyond whatever positional scheme you inject, and its cost grows quadratically with context while its memory grows linearly. An LNN keeps a fixed-size state regardless of horizon, which is exactly what an embedded controller running at 50 Hz for hours needs, and exactly the wrong trade for retrieving a fact from 200,000 tokens of text.

Mathematical Formulation:
$\frac{dx}{dt} = -\frac{x}{\tau} + f(x, I; \theta)\,(A - x)$
$\tau_{sys}(x, I) = \frac{\tau}{1 + \tau f(x, I; \theta)}$
$\frac{\tau}{1 + \tau f_{max}} \leq \tau_{sys} \leq \tau$
$h_t = \sigma(W h_{t-1} + U I_t + b)$

Where:

$x(t) \in \mathbb{R}^{d}$ is the continuous-time neuron state and $I(t)$ the sensory input at that instant.
$f(x, I; \theta)$ is a bounded nonlinearity (typically a sigmoid of an affine map) with static trained parameters $\theta$ ; it appears twice, as the drive and inside the time constant.
$\tau$ is the per-neuron leak time constant, $A$ the resting potential the drive pulls toward, and $f_{max}$ the supremum of $f$ that bounds the fastest achievable dynamics.
$\tau_{sys}$ is the liquid (effective) time constant; the bracketing inequality is what guarantees the state stays bounded and the solver stays stable.
The last line is the fixed-weight RNN baseline: $h_t$ depends on step index $t$ only, with $W, U, b$ applied identically at every step and no $\Delta t$ anywhere in the update.

Two panel chart. Left panel plots the effective time constant tau_sys against input drive f on a logarithmic vertical axis for three base time constants 0.1, 0.5 and 2.0 seconds, all curves collapsing toward one over f as the drive grows. Right panel plots state x of t against time for a square input pulse between one and three seconds, comparing a fixed leaky integrator with a constant one second time constant against a liquid time constant cell whose effective time constant is 0.21 seconds while the input is on and 0.84 seconds when it is off, showing a much faster rise and a partially sustained level

Figure 2: Left, the whole point of the $1 + \tau f$ denominator: a strong input collapses $\tau_{sys}$ toward $1/f$ , so one neuron spans a wide band of timescales. Right, the same input pulse through a fixed leaky integrator and an LTC cell with identical $\tau$ ; the liquid cell charges roughly five times faster while driven and relaxes more slowly once the drive is removed, behaviour a single fixed-weight filter cannot produce.

The practical objection to LNNs is the solver. Every observation costs $k$ evaluations of $f$ , stiff dynamics push $k$ up, and the whole recurrence is sequential, so training cannot be parallelized across time the way attention or a convolutional scan can. Closed-form Continuous-time (CfC) networks answer this by replacing the numeric integration with an analytic approximation of the LTC solution, keeping the input-dependent timescale while deleting the solver from the graph, at reported speedups of one to five orders of magnitude in training and inference over solver-based LTCs.

Closed Form And Cost:
$x(t) = \sigma(-f\,t) \odot g + [1 - \sigma(-f\,t)] \odot h$
$C_{LNN} = O(L\,k\,d^2)$
$C_{TF} = O(L^2 d + L\,d^2)$

Here $f, g, h$ are separate learned heads of $(x, I)$ and $\sigma(-f t)$ acts as a time-dependent gate that interpolates between two candidate states as elapsed time grows, which is the closed-form stand-in for exponential relaxation. Note that $C_{LNN}$ has no term in $L^2$ and no cache: an LNN controller at $d = 64$ can run indefinitely in constant memory, while the Transformer term grows with every token retained.

Property	Liquid (LTC / CfC)	Fixed-weight RNN (LSTM / GRU)	Transformer
State definition	Continuous state from an ODE, evaluated by a solver or a closed-form gate	Discrete hidden and cell state, one update per step	No recurrent state; every token attends over the full prefix
Handling of elapsed time	Delta t is an explicit solver argument, so unequal gaps are exact	Implicit constant stride; gaps must be imputed or appended as a feature	Position index or RoPE; real timestamps must be encoded as extra inputs
Effective timescale	Input-dependent, tau over one plus tau f, bounded above by tau	Set by trained gate weights, modulated only within one fixed stride	None; the mixing radius is the context window
Cost per observation	k solver evaluations, sequential, no cache	One matmul pass, sequential, no cache	Parallel in training, but KV cache grows linearly with L
Typical deployed size	Tens to a few thousand neurons, including 19-neuron flight and driving policies	Thousands to millions of parameters	Hundreds of millions to hundreds of billions
Dominant failure mode	Stiff dynamics inflate solver steps; no parallelism over time	Vanishing gradients and timescale mismatch on long horizons	Quadratic prefill, and no built-in notion of physical duration

August 20, 2026

DL0123 3D-Aware Visual Pretraining

How does 3D-aware visual pre-training bridge spatial intelligence between 2D generative video models and 3D physical world simulators?

Answer

3D-aware visual pre-training trains a visual backbone or a video generator with objectives that can only be solved by recovering scene geometry: per-pixel pointmaps, metric depth, camera rays, and cross-view correspondence, instead of appearance statistics alone. The two ends it connects have complementary deficits. A 2D video diffusion model absorbs enormous visual and dynamic diversity from internet footage, but its geometry is implicit, so it has no queryable state: you cannot ask it for the distance to a mug in centimeters, and a long rollout quietly changes room layout because nothing anchors the scene. A physics simulator has the opposite profile, exact meshes, contacts, and metric scale, but its asset library and rendering diversity are tiny compared with the real world. Pre-training with geometric targets creates the shared interface: the same representation gives the generator a camera and depth channel it can be conditioned on, and gives the simulator a path to ingest real scenes as reconstructions, so the pipeline runs video → pointmaps → 3D Gaussians or mesh → simulator asset in one direction and simulator depth, pose, and contact labels back into the pre-training loss in the other.

(1) Geometry-Grounded Pretext Tasks: pointmap, ray, and depth regression plus multi-view matching force the backbone to encode where surfaces are, not just what they look like; ordinary video supplies the multiple views for free.
(2) Camera As A First-Class Input: Plücker ray embeddings or pose tokens turn a text-to-video model into a controllable renderer, which is exactly the interface a simulator viewport exposes.
(3) Persistent State Beats Frame Memory: caching an explicit 3D representation and re-projecting it into each new frame removes the layout drift and object-identity loss that pure 2D autoregressive rollouts accumulate.
(4) Two-Way Data Flow: real-to-sim reconstruction fills the simulator’s asset gap, while simulator renders provide perfect metric depth, pose, and contact supervision that no internet video can give.
(5) 3D Awareness Is Measurable: linear probes on frozen features show that self-supervised 2D backbones encode single-view depth surprisingly well yet fail at multi-view consistency, and 3D-aware fine-tuning lifts both plus downstream segmentation.
(6) Metric Scale Is The Robot Requirement: scale-invariant depth is enough for a nice-looking novel view and useless for a grasp, so metric anchoring is the part embodied policies actually consume.

Diagram: internet video corpora and simulator renders both feed a shared backbone whose pointmap, camera-pose, and differentiable-render heads serve two consumers, a camera-controlled video generator and a 3D physics simulator

Figure 1: One backbone, two supervision sources, two consumers. Unlabeled video supplies visual diversity and self-consistency signals, simulator renders supply exact depth and pose, and the geometry heads are what the video generator and the simulator both read from.

The concrete form most systems settle on is a feed-forward multi-view transformer that predicts a pointmap per frame in a common coordinate frame, so camera intrinsics, extrinsics, depth, and correspondence all fall out of one prediction rather than a fragile SfM plus MVS chain. DUSt3R established the pointmap formulation for two views, MASt3R added a matching head, and VGGT scaled it to many frames with attention alternating between frame-local and global blocks. That output is directly liftable: initialize 3D Gaussians from the points for a renderable scene, or run meshing plus material estimation for a body a rigid-body engine can collide against. Two caveats decide whether the bridge is load-bearing. First, monocular training recovers geometry only up to an unknown similarity transform, so metric scale has to be injected from calibrated stereo, a known camera baseline, simulator ground truth, or object-size priors. Second, the reprojection and correspondence losses assume a static scene, so dynamic content needs motion masks or a 4D formulation or the model folds object motion into depth as flying pixels.

Mathematical Formulation:
$u = \pi(K, T_c, X)$
$\bar{X}_i = X_i / s$
$s = \frac{1}{N}\sum_i \lVert X_i \rVert_2$
$\mathcal{L}_{geo} = \sum_i \lVert \bar{X}_i - \bar{X}^{*}_i \rVert_1$
$\mathcal{L}_{corr} = \sum_i \lVert u'_i - \pi(K', T_{c'}, X_i) \rVert_2$
$\mathcal{L}_{rgb} = \lVert I_c - \mathcal{R}(G, K, T_c) \rVert_1$
$\mathcal{L} = \mathcal{L}_{2D} + \lambda_g \mathcal{L}_{geo}$
$+\ \lambda_c \mathcal{L}_{corr} + \lambda_r \mathcal{L}_{rgb}$

Where:

$u$ is a pixel coordinate and $X \in \mathbb{R}^3$ the predicted scene point for that pixel, expressed in the reference camera’s frame.
$\pi$ is the pinhole projection, $K$ the intrinsics, and $T_c$ the pose of camera $c$ ; primed symbols refer to a second view.
$i \in \{1, \ldots, N\}$ indexes valid pixels, and $X^{*}$ is the ground-truth pointmap from a simulator, LiDAR, or a reconstruction pipeline.
$s$ is the mean point norm used for scale normalization, so $\mathcal{L}_{geo}$ on $\bar{X}$ is scale-invariant; dropping the division makes the loss metric and requires metric labels.
$\mathcal{L}_{corr}$ is the reprojection error against a matched pixel $u'$ in the other view, and it is valid only where the scene is static between the two frames.
$\mathcal{R}$ is a differentiable renderer (splatting or volume rendering) over a lifted representation $G$ , letting 2D images supervise 3D structure through $\mathcal{L}_{rgb}$ .
$\mathcal{L}_{2D}$ is the original appearance objective (diffusion denoising or masked prediction) and $\lambda_g, \lambda_c, \lambda_r \geq 0$ weight the geometric terms.

Log-scale bar chart of global attention cost relative to a single view for 1, 2, 4, 8, 16, and 32 views at 1369 tokens per view, growing as the square of the view count

Figure 2: Joint multi-view geometry is what makes a representation 3D-aware, but attention over all views at once costs $O((VN)^2)$ , so a 32-view window is roughly 1,000x a single view. This is why practical models alternate frame-local and global attention and slide a bounded view window.

Property	2D generative video model	3D-aware pre-trained representation	3D physics simulator
Scene state	Implicit in activations and past frames; not queryable	Explicit pointmaps, depth, poses, optionally 3D Gaussians	Exact meshes, transforms, joints, contact manifolds
Camera control	Text prompts only, unless pose conditioning is added	Ray or pose embeddings make trajectory a direct input	Arbitrary and exact by construction
Metric scale	Absent	Available only if metric supervision or calibration is injected	Exact, in SI units
Visual and asset diversity	Internet scale, effectively unbounded	Inherits video-scale diversity, adds geometry heads	Limited to authored assets and shaders
Typical failure mode	Layout drift, object teleporting, impossible occlusions	Scale ambiguity, flying pixels on moving objects, cost of joint views	Appearance gap and unmodeled deformable or fluid behavior

August 17, 2026

DL0104 Score-Based Diffusion

What is the score-based / score-matching interpretation of diffusion models, as used in systems like Stable Diffusion?

Answer

The score-based view says a diffusion model never learns a density; it learns the score, the gradient of the log density $s(x) = \nabla_x \log p(x)$ . This is attractive because the gradient annihilates the intractable normalizing constant: if $p(x) = e^{-E(x)}/Z$ then $\nabla_x \log p(x) = -\nabla_x E(x)$ with no $Z$ anywhere, so an unnormalized energy model becomes trainable. Direct score matching still needs the intractable $\nabla_x \log p_{\text{data}}(x)$ , and the fix is denoising score matching: perturb the data with a known Gaussian kernel, and the score of the perturbed conditional is available in closed form, which reduces the loss to predicting the noise that was added. Because a single small noise level leaves the score undefined off the data manifold and unreliable in low-density regions, the model is trained over a continuum of noise levels, which turns into the forward SDE of Song et al. and makes sampling the simulation of a reverse-time SDE or its deterministic probability-flow ODE. The punchline is that DDPM’s noise-prediction network is the same object in different clothing: $s_\theta(x_t,t) = -\epsilon_\theta(x_t,t)/\sigma_t$ , so the discrete-time denoising story and the continuous-time score story are two parameterizations of one model.

(1) Model The Gradient, Not The Density: learning $\nabla_x \log p(x)$ removes the partition function, so the network output is an unconstrained vector field of the same shape as the data.
(2) Denoising Score Matching Makes It Tractable: Vincent’s identity shows that regressing onto $\nabla_{x_t}\log p(x_t \mid x_0)$ has the same minimizer as matching the true marginal score, and that target is just the scaled added noise.
(3) Multiple Noise Scales Are Essential: real data lies near a low-dimensional manifold where the score is ill-defined, and large noise levels smooth the density so the vector field is informative everywhere and connects separated modes.
(4) Sampling Is Numerical Integration: annealed Langevin dynamics, the reverse SDE, and the probability-flow ODE all consume only the learned score; the ODE additionally gives exact likelihoods and deterministic, invertible latents.
(5) Unifies VE And VP Diffusions: NCSN’s variance-exploding noise ladder and DDPM’s variance-preserving chain are two discretizations of the same SDE family, differing in $f$ , $g$ , and the loss weighting $\lambda_t$ .

The geometric picture is worth holding onto. The score at a point is the direction of steepest increase in log density, so the learned field points from wherever you are toward nearby high-probability regions, and its magnitude grows as you move into the tails. Sampling then means dropping a Gaussian sample somewhere in space and following that field while injecting just enough noise to keep the chain from collapsing onto a single mode. The reason this needs annealing is that a score trained only at tiny noise is accurate on the data manifold and near-arbitrary far from it, where every initial sample actually starts.

Contour plot of a two-mode density with an overlaid arrow field pointing toward the modes, and a jagged Langevin sampling path starting in the empty lower-left corner and ending inside the left mode

Figure 1: The learned score is a vector field pointing toward high-density regions; a Langevin chain follows it from a random start and ends up distributed according to $p(x)$ , never evaluating the density itself.

Mathematical Formulation:
$s_\theta(x,t) \approx \nabla_x \log p_t(x)$
$x_t = \alpha_t x_0 + \sigma_t \epsilon$
$\nabla_{x_t} \log p(x_t \mid x_0) = -\frac{\epsilon}{\sigma_t}$
$\mathcal{L}(\theta) = \mathbb{E}\left[ \lambda_t \left\| s_\theta(x_t,t) + \frac{\epsilon}{\sigma_t} \right\|^2 \right]$
$dx = f(x,t)\,dt + g(t)\,dw$
$dx = \left[ f - g^2 s_\theta \right] dt + g\,d\bar{w}$
$dx = \left[ f - \tfrac{1}{2} g^2 s_\theta \right] dt$
$s_\theta(x_t,t) = -\frac{\epsilon_\theta(x_t,t)}{\sigma_t}$

Where:

$s_\theta(x,t)$ is the network output, a vector with the same shape as $x$ , approximating the score of the perturbed marginal $p_t$ .
$x_0$ is a clean data point, $x_t$ its noisy version, and $\epsilon \sim \mathcal{N}(0,I)$ the Gaussian draw.
$\alpha_t$ and $\sigma_t$ define the noise schedule; variance-preserving keeps $\alpha_t^2 + \sigma_t^2 = 1$ , variance-exploding fixes $\alpha_t = 1$ and grows $\sigma_t$ .
$t \in [0,T]$ indexes the noise level, with $p_T$ essentially pure Gaussian noise; $\lambda_t$ is the loss weighting, and the choice $\lambda_t = \sigma_t^2$ turns the objective into the plain noise-prediction MSE.
$f(x,t)$ is the drift and $g(t)$ the diffusion coefficient of the forward SDE; $w$ is a Wiener process and $\bar{w}$ its reverse-time counterpart, so the second SDE is integrated from $t = T$ down to $t = 0$ .
The final line is the reparameterization identity linking the score view to DDPM’s $\epsilon_\theta$ ; required initial condition for sampling: $x_T \sim \mathcal{N}(0, \sigma_T^2 I)$ for the VE case.

Three panels showing the same two-mode density smoothed by increasing Gaussian noise, with score arrows that are confined near the modes at small noise and fill the whole plane at large noise

Figure 2: Why a ladder of noise levels is required: at $\sigma = 0.05$ the score carries no usable signal away from the data, while at $\sigma = 1.5$ the smoothed field points inward from everywhere and bridges the two modes.

Property	Score SDE (VE, NCSN)	DDPM (VP, noise-pred)	Flow matching
Network output	Score, magnitude scales like 1/sigma_t	Unit-variance noise estimate	Velocity field of a probability path
Corruption	Add noise, variance grows to data scale	Shrink signal, keep total variance at 1	Straight interpolation between data and noise
Loss	Weighted denoising score matching	Same loss with weighting sigma_t squared	Regression onto the conditional velocity
Sampler	Annealed Langevin or predictor-corrector	Ancestral chain, or DDIM as an ODE	ODE solver on a near-straight path
Exact likelihood	Yes, via the probability-flow ODE	Only an ELBO in discrete time	Yes, continuous normalizing flow change of variables

August 16, 2026

DL0102 Diffusion vs GAN vs VAE

Explain the fundamental principles of Diffusion Models, and compare their pros and cons with GANs, VAEs, and Flow-based models.

Answer

A diffusion model defines a fixed forward process that gradually corrupts data into Gaussian noise over $T$ steps, then learns the reverse process that removes a little noise at a time. The forward chain has no parameters and admits a closed form, so training reduces to a plain regression: sample a clean image $x_0$ , sample a timestep $t$ and noise $\epsilon$ , build the noisy state $x_t$ , and ask the network to predict $\epsilon$ . This denoising objective is a weighted variational bound on the log-likelihood and is also equivalent to score matching, which is why the same network can be plugged into an SDE or ODE sampler. The practical consequence is that diffusion buys stable training and excellent mode coverage at the price of many network evaluations per sample, whereas a GAN generates in one forward pass but fights an unstable min-max game, a VAE trains stably but produces blurry samples under a Gaussian decoder, and a normalizing flow gives exact likelihoods but pays for invertibility with architectural constraints and parameter count.

(1) Fixed Corruption, Learned Reversal: only the reverse direction has parameters, so there is no adversary and no discriminator to balance, which removes the main source of GAN training instability.
(2) Closed-Form Training Target: because $q(x_t \mid x_0)$ is Gaussian, any timestep can be sampled directly without simulating the chain, making each training step $O(1)$ in $T$ .
(3) Score Interpretation: the noise predictor is a rescaled estimate of $\nabla_{x_t}\log q(x_t)$ , which links diffusion to Langevin dynamics and to probability-flow ODE samplers such as DDIM.
(4) Sampling Is The Bottleneck: quality scales with the number of function evaluations, so a 1000-step DDPM sampler costs three orders of magnitude more compute per image than a GAN generator.
(5) Coverage Versus Speed: the likelihood-flavored objective penalizes dropped modes, which is exactly the failure GANs are known for, so the two families sit on opposite corners of the quality/diversity/speed trade-off.

Diagram of a diffusion chain: five states from a clean image to pure noise, with upper arrows showing the fixed forward noising process q and lower arrows showing the learned reverse denoiser p theta, plus a dashed arc marking the closed-form jump from x0 to any xt

Figure 1: The forward process is fixed and parameter-free, and the dashed arc is the closed form that lets training jump straight to any $x_t$ ; every arrow in the lower reverse path costs one network evaluation at sampling time.

The four families differ mostly in how they pay for tractability. A VAE keeps an explicit encoder and decoder and optimizes a lower bound, so the mismatch between the true posterior and the Gaussian approximation shows up as blur and posterior collapse. A normalizing flow refuses any approximation and computes exact likelihoods, but every layer must be invertible with a cheap Jacobian determinant, which caps expressiveness per parameter. A GAN drops likelihood entirely and optimizes a discriminator-defined divergence, which produces sharp samples in one step but offers no signal that the whole data distribution is covered. Diffusion sidesteps all three constraints by turning generation into a long sequence of easy denoising problems, and then spends compute at inference to pay for it.

Mathematical Formulation:
$q(x_t \mid x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}\, x_{t-1}, \beta_t I)$
$\bar{\alpha}_t = \prod_{s=1}^{t} (1 - \beta_s)$
$x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \epsilon$
$\mathcal{L}_{\text{simple}} = \mathbb{E}_{t, x_0, \epsilon} [ \| \epsilon - \epsilon_\theta(x_t, t) \|^2 ]$
$s_\theta(x_t, t) = -\frac{\epsilon_\theta(x_t, t)}{\sqrt{1 - \bar{\alpha}_t}}$

Where:

$x_0$ is a clean data sample and $x_t$ the noised state at step $t$ , with $x_T$ approximately standard Gaussian.
$\epsilon \sim \mathcal{N}(0, I)$ is the injected noise and $\epsilon_\theta$ the learned noise predictor with parameters $\theta$ .
$t \in \{1, \ldots, T\}$ is sampled uniformly during training, and $s$ indexes the product defining the cumulative signal retention.
$\beta_t$ is the noise schedule, typically small and increasing, and $\bar{\alpha}_t$ is the cumulative signal factor giving signal-to-noise ratio $\bar{\alpha}_t / (1 - \bar{\alpha}_t)$ .
$s_\theta$ is the implied score function, the quantity an SDE or probability-flow ODE sampler integrates.
Required initial condition: sampling starts from $x_T \sim \mathcal{N}(0, I)$ , which is only valid if the schedule drives $\bar{\alpha}_T$ to (near) zero.

Property	Diffusion	GAN	VAE	Normalizing Flow
Training objective	Denoising regression on a fixed forward process, no adversary	Min-max game against a discriminator	ELBO: reconstruction plus KL to the prior	Exact log-likelihood via change of variables
Network calls per sample	10 to 1000, or 1 to 4 after distillation	1	1	1
Sample fidelity	State of the art for images, video, and audio	Very sharp, still strong under a one-step budget	Blurry with a Gaussian decoder	Below GAN and diffusion at equal compute
Mode coverage	Broad, the objective penalizes missing mass	Mode collapse is the classic failure	Broad but over-smoothed	Broad, likelihood forbids ignoring regions
Density estimate	Bound or ODE-based estimate, not exact in closed form	None	Lower bound only	Exact and tractable
Architecture constraint	Free, any denoiser such as a U-Net or a DiT	Free	Free	Invertible layers with cheap Jacobian determinant
Typical failure	Slow sampling, schedule and guidance artifacts	Unstable training, collapsed diversity	Blur and posterior collapse	Very large models for modest sample quality

Triangle diagram of the generative learning trilemma with vertices high sample quality, fast sampling, and mode coverage; GAN sits on the quality-speed edge, VAE and normalizing flow on the speed-coverage edge, and diffusion on the quality-coverage edge

Figure 2: The generative learning trilemma: each family wins two corners and pays on the third, and most modern work on diffusion (distillation, consistency training, rectified flow) is an attempt to drag it toward the fast-sampling corner without losing coverage.

August 16, 2026

DL0100 GAN Framework

Explain the GAN framework: what are the two networks optimizing, and why does the generator learn anything at all?

Answer

A Generative Adversarial Network trains two networks against each other. The generator $G$ maps a latent noise vector $z \sim p_z$ (typically $\mathcal{N}(0, I)$ ) to a sample in data space, and the discriminator $D$ outputs the probability that its input came from the real dataset rather than from $G$ . They share one value function in a two-player minimax game: $D$ maximizes the log-likelihood of correctly classifying real and fake batches, while $G$ minimizes the same quantity, so the generator’s only learning signal is the gradient that flows backward through the discriminator into the fake samples. The theory is clean: for a fixed $G$ the optimal discriminator is the density ratio $p_{data}/(p_{data} + p_g)$ , and substituting it turns the generator’s objective into the Jensen-Shannon divergence between the data distribution and the model distribution, uniquely minimized when $p_g = p_{data}$ . The practical payoff is that $G$ is an implicit model: it never evaluates a likelihood, it just produces a sample in one forward pass, which is why GANs remain attractive whenever sampling latency matters.

(1) Two Networks One Objective: there is a single value function $V(D,G)$ ; the discriminator ascends it and the generator descends it, so no explicit reconstruction or likelihood term is ever written down.
(2) Discriminator As A Density Ratio: the optimal $D^*$ encodes $p_{data}(x)/p_g(x)$ , which is exactly the information the generator needs about where it is over- or under-producing mass.
(3) Implicit Sampling Model: $p_g$ is defined only through the pushforward of $p_z$ by $G$ , so sampling is a single forward pass but density evaluation is impossible.
(4) Non-Saturating Generator Loss: the theoretical $\log(1 - D(G(z)))$ term has almost no gradient while the generator is bad, so implementations maximize $\log D(G(z))$ instead.
(5) Alternating Updates, No Loss Curve To Read: training alternates discriminator and generator steps and seeks a saddle point, not a minimum, so a falling loss means nothing and quality is judged with FID or human inspection.

Diagram of the GAN framework: latent noise feeds the generator which produces fake samples, real samples and fake samples both feed the discriminator which outputs a probability of being real, and a dashed path carries the generator gradient back from the discriminator output into the generator

Figure 1: The generator never sees a real sample directly; it only receives gradient that has been routed backward through the discriminator, which is why the quality of $D$ bounds what $G$ can learn.

A training step draws a minibatch of real samples and a minibatch of latents, updates $D$ on both, then updates $G$ with $D$ held fixed. The original paper allowed $k$ discriminator steps per generator step; almost everyone now uses $k = 1$ and controls the balance with regularization instead. The three interventions that matter most in practice are spectral normalization or an R1 gradient penalty to keep the discriminator smooth, two-timescale learning rates so the discriminator can stay slightly ahead, and an exponential moving average of the generator weights for evaluation. The classic failure modes are mode collapse, where $G$ concentrates on a few outputs that currently fool $D$ , and vanishing generator gradient, where a discriminator with near-perfect separation returns almost nothing useful.

Mathematical Formulation:
$\min_G \max_D V(D, G)$
$V(D,G) = \mathbb{E}_{x \sim p_{data}}[\log D(x)]$
$\quad + \mathbb{E}_{z \sim p_z}[\log (1 - D(G(z)))]$
$D^*(x) = \frac{p_{data}(x)}{p_{data}(x) + p_g(x)}$
$C(G) = 2 \, \mathrm{JSD}(p_{data} \, \| \, p_g) - \log 4$
$\mathcal{L}_G = -\mathbb{E}_{z \sim p_z}[\log D(G(z))]$

Where:

$V(D,G)$ is the shared value function, maximized by $D$ and minimized by $G$ .
$x \sim p_{data}$ is a real sample and $z \sim p_z$ a latent vector, so $G(z)$ is a generated sample and $D(\cdot) \in (0,1)$ is the estimated probability of being real.
$p_g$ is the implicit distribution induced by pushing $p_z$ through $G$ ; it is never evaluated, only sampled.
$D^*$ is the optimal discriminator for a fixed $G$ , and $C(G) = \max_D V(D,G)$ is the resulting generator criterion; $\mathrm{JSD}$ is the Jensen-Shannon divergence, so $C(G) \geq -\log 4$ with equality only at $p_g = p_{data}$ .
$\mathcal{L}_G$ is the non-saturating generator loss actually used in code; it shares the same fixed point as the minimax form but has large gradient when $D(G(z))$ is near 0, which is the required starting condition of training.

Log-scale plot of generator gradient magnitude versus the discriminator score on fake samples, showing the saturating loss curve one over one minus D staying near one at low scores while the non-saturating curve one over D rises to one hundred

Figure 2: Gradient magnitude of the two generator losses with respect to $D(G(z))$ . At the start of training the discriminator wins easily, the saturating form contributes almost nothing, and the non-saturating form is roughly 100x larger at a score of 0.01.

Property	GAN	VAE	Diffusion
Training objective	Adversarial minimax, no likelihood term	ELBO, a lower bound on log-likelihood	Weighted denoising regression on noised inputs
Sampling cost	One forward pass through the generator	One forward pass through the decoder	20 to 1000 network evaluations unless distilled
Sample sharpness	Very sharp, no pixel-averaging term to blur output	Typically blurry from the reconstruction term	Very sharp, current state of the art on text-to-image
Mode coverage	Weakest; mode collapse is the signature failure	Good coverage, over-smoothed samples	Good coverage from a likelihood-style objective
Training stability	Saddle-point game, needs penalties and careful balance	Single stable objective	Single stable objective, scales predictably

August 16, 2026

DL0097 Variational Autoencoder VAE

What is a variational autoencoder (VAE), and how does it differ from a plain autoencoder?

Answer

A variational autoencoder is a latent-variable generative model trained by amortized variational inference, not a compression network with a smaller bottleneck. A plain autoencoder learns a deterministic map $z = f_\phi(x)$ and minimizes reconstruction error alone, so nothing constrains where codes land: the latent space is an arbitrary point cloud with holes, and decoding a random vector usually produces garbage. A VAE instead makes the encoder output the parameters of a distribution $q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \sigma_\phi^2(x))$ , samples $z$ from it, and optimizes the evidence lower bound (ELBO), which adds a KL term pulling every posterior toward a fixed prior $p(z) = \mathcal{N}(0, I)$ . That single extra term is what turns an encoder-decoder pair into a generator: after training you can draw $z \sim \mathcal{N}(0, I)$ and decode it, because the aggregate posterior now covers the prior. The sampling step is made differentiable by the reparameterization trick, $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0, I)$ , which keeps the stochastic node out of the gradient path.

(1) Probabilistic Encoder: the encoder emits a mean and a log-variance per latent dimension rather than a point, so each input maps to a small blob of latent space instead of a single coordinate.
(2) Two-Term Objective: the loss is reconstruction plus a KL regularizer against the prior; a plain autoencoder has only the first term, which is exactly why it is not generative.
(3) Reparameterization Trick: sampling is rewritten as a deterministic function of the parameters and an external noise draw, giving a low-variance pathwise gradient instead of a high-variance score-function estimator.
(4) Sampling From The Prior: because the KL term forces the posteriors to overlap and fill the prior, ancestral sampling ( $z \sim p(z)$ , then decode) yields plausible data, and interpolation between two codes stays on the data manifold.
(5) Explicit Likelihood Bound: the ELBO is a lower bound on $\log p_\theta(x)$ , so a VAE gives a comparable density estimate, whereas an autoencoder’s reconstruction error has no probabilistic meaning.

Two-row diagram: the top row shows a plain autoencoder mapping input x through an encoder to a single deterministic code and back through a decoder; the bottom row shows a VAE whose encoder emits mu and log sigma squared, a sampling step z equals mu plus sigma times epsilon, and a decoder, with a KL term pulling the posterior toward the standard normal prior

Figure 1: The structural difference is one node: the plain autoencoder passes a single deterministic code to the decoder, while the VAE passes a sample from a learned Gaussian that the KL term keeps anchored to $\mathcal{N}(0, I)$ .

Two practical details dominate real training runs. The first is the balance between the two loss terms: with a strong decoder or a large KL weight, the cheapest solution is to set $q_\phi(z|x) = p(z)$ and ignore the latent entirely, a failure called posterior collapse that shows up as a KL term decaying to near zero while reconstruction stalls. Standard mitigations are KL annealing (ramp the weight from 0 to 1 over the first epochs) and free bits (do not penalize a dimension until its per-dimension KL exceeds a floor of roughly 0.05 to 0.5 nats). The second is the choice of likelihood: a diagonal Gaussian decoder is equivalent to an MSE reconstruction loss, which averages over plausible outputs and is the direct cause of the blurry samples VAEs are known for; discretized logistic or categorical likelihoods sharpen results noticeably. Modern image systems exploit this honestly, using a KL-regularized autoencoder as the perceptual compressor and putting the generative burden on a diffusion model in that latent space, as in Stable Diffusion.

Mathematical Formulation:
$\log p_\theta(x) \geq \mathcal{L}(\theta, \phi; x)$
$\mathcal{L} = \mathcal{L}_{rec} - \beta \, D_{KL}(q_\phi \| p)$
$\mathcal{L}_{rec} = \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)]$
$q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \sigma_\phi^2(x))$
$z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon$
$\epsilon \sim \mathcal{N}(0, I)$
$D_{KL} = -\frac{1}{2} \sum_{j=1}^{d} \delta_j$
$\delta_j = 1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2$

Where:

$\mathcal{L}$ is the ELBO, a lower bound on the marginal log-likelihood $\log p_\theta(x)$ ; the gap between them equals $D_{KL}(q_\phi(z|x) \| p_\theta(z|x))$ , so maximizing the bound both fits the data and sharpens the approximate posterior.
$x$ is the observation and $z \in \mathbb{R}^{d}$ the latent code, with $d \ll \dim(x)$ in the usual bottleneck setting.
$q_\phi(z|x)$ is the encoder (recognition model) with parameters $\phi$ , and $p_\theta(x|z)$ is the decoder (likelihood) with parameters $\theta$ .
$p(z) = \mathcal{N}(0, I)$ is the fixed prior, and $j \in \{1, \ldots, d\}$ indexes latent dimensions in the closed-form KL, where $\mu_j$ and $\sigma_j$ are the encoder outputs for example $x$ .
$\odot$ is elementwise multiplication and $\epsilon$ is the external noise draw that makes the sample differentiable in $\mu$ and $\sigma$ .
$\beta$ is the KL weight; $\beta = 1$ recovers the exact ELBO, $\beta > 1$ gives the beta-VAE disentanglement regime, and $\beta = 0$ degenerates to a plain autoencoder. Networks are typically initialized so $\log \sigma_j^2 \approx 0$ , which starts training near the prior.

Two scatter panels of a two-dimensional latent space: on the left the plain autoencoder codes sit in six distant clusters far from the origin with large empty gaps and prior samples falling in a hole, on the right the VAE codes form overlapping blobs filling the unit and two-sigma circles of the standard normal prior so prior samples land on data

Figure 2: Encoded training data in a 2D latent space. The plain autoencoder is free to scatter codes anywhere, so a draw from $\mathcal{N}(0, I)$ lands in a hole; the KL term compresses the VAE’s aggregate posterior onto the prior, so the same draw hits populated territory.

Property	Plain Autoencoder	VAE	VQ-VAE
Encoder output	One deterministic code vector per input	Distribution parameters, mean and log-variance	Continuous vector snapped to the nearest codebook entry
Training objective	Reconstruction error only	ELBO: reconstruction minus beta times KL to N(0, I)	Reconstruction plus codebook and commitment losses, no KL
Latent geometry	Arbitrary scale, holes and gaps between clusters	Smooth and prior-matched, interpolation stays on-manifold	Discrete grid of K entries, no notion of interpolation
Generating new data	Not supported, a random code decodes to noise	Draw z from the prior and decode, one forward pass	Needs a learned prior over codes, such as a transformer
Typical failure	Memorizes an identity map when the bottleneck is wide	Posterior collapse, blurry Gaussian-likelihood samples	Codebook collapse with most entries unused
Common use	Denoising, compression, anomaly detection	Generative modeling and the latent space of latent diffusion	Discrete tokens for autoregressive image and audio models

August 16, 2026

DL0096 Autoencoders Practical Uses

What is an autoencoder? What are the practical uses of autoencoders?

Answer

An autoencoder is a network trained to reconstruct its own input through a narrow bottleneck: an encoder maps $x \in \mathbb{R}^{D}$ to a code $z \in \mathbb{R}^{d}$ with $d \ll D$ , and a decoder maps that code back to $\hat{x}$ , with the reconstruction error as the only supervision. Because the target is the input itself, training needs no labels, which is why autoencoders are the canonical self-supervised compression model. The bottleneck is what makes the code useful: the network cannot pass everything through, so it must keep the structure that is shared across typical inputs and drop the rest. With linear encoder and decoder and squared loss, the optimum spans exactly the same subspace as the top $d$ principal components; nonlinear layers buy a curved manifold instead of a flat one. The practical uses split into three families depending on which part of the trained model you keep: the error (anomaly detection), the code (compression, retrieval, visualization, latent space for a diffusion model), or the decoder (denoising, inpainting, generation once the latent is regularized).

(1) Bottleneck Plus Reconstruction Loss: the objective is $\|x - \hat{x}\|_2^2$ (or cross-entropy for binary pixels), and the capacity constraint, not the loss, is what forces a compressed representation.
(2) Undercomplete Versus Regularized: an undercomplete model constrains $d$ directly, while sparse, denoising, and contractive variants keep a wide code and constrain it with an $L_1$ penalty, input corruption, or a Jacobian penalty.
(3) Linear Case Recovers PCA: a single linear layer each side with MSE learns the principal subspace, so any gain over PCA must come from nonlinearity, not from the architecture being a neural network.
(4) Anomaly Detection By Error Score: train only on normal data, then score new samples with $s(x) = \|x - \hat{x}\|_2^2$ and flag $s(x) > \tau$ , with $\tau$ set from a percentile of validation errors.
(5) Compression Into A Working Latent Space: modern latent diffusion models run denoising inside an autoencoder latent that shrinks a $512 \times 512 \times 3$ image to $64 \times 64 \times 4$ , a 48x reduction in elements, which is what makes high-resolution sampling affordable.
(6) Not Generative By Default: a vanilla autoencoder learns no density over $z$ , so decoding an arbitrary code usually yields garbage; sampling requires a VAE, VQ-VAE, or a prior fitted over the codes afterwards.

Diagram of an autoencoder: a 784-dimensional input passes through a 256-unit hidden layer into a 32-dimensional code, then back out through a 256-unit hidden layer to a 784-dimensional reconstruction, with both ends feeding a reconstruction loss box

Figure 1: The flow is $x$ → $z$ → $\hat{x}$ ; the $784 \rightarrow 32$ bottleneck is a 24.5x squeeze, and nothing but the reconstruction error supervises what the code contains.

Where do they actually earn their place in a system? Anomaly detection on sensor, telemetry, and transaction data is the most common production use, because normal data is abundant and labeled failures are not. Denoising and restoration use the decoder: train with corrupted inputs and clean targets, and the model learns to project back onto the data manifold. Dimensionality reduction gives a cheap nonlinear preprocessor, typically compressing to 32 or 64 dimensions before a nearest-neighbor index or a t-SNE plot, which is far cheaper than running the visualization on raw features. Two uses dominate current deep learning: the perceptual autoencoder that provides the latent space for latent diffusion, and masked autoencoders, where a ViT reconstructs 75% masked patches as a pretraining task and the encoder is then fine-tuned for classification or detection. Recommender systems also use the pattern directly, reconstructing a user’s sparse interaction vector to predict the entries that are missing.

Histogram of reconstruction errors: normal samples cluster at low error, anomalous samples spread into a long right tail, with a vertical threshold line drawn at the 99th percentile of the normal validation errors

Figure 2: Illustrative anomaly scores from an autoencoder trained on normal data only. A threshold at the 99th percentile of normal validation errors fixes the false-positive rate at 1% by construction, and the achievable recall depends entirely on how far the anomalous tail separates.

Mathematical Formulation:
$z = f_{\theta}(x)$
$\hat{x} = g_{\phi}(z)$
$\mathcal{L}(\theta, \phi) = \frac{1}{N} \sum_{i=1}^{N} \| x_i - \hat{x}_i \|_2^2$
$s(x) = \| x - g_{\phi}(f_{\theta}(x)) \|_2^2$
$\mathcal{L}_{\mathrm{DAE}} = \mathbb{E} \| x - g_{\phi}(f_{\theta}(\tilde{x})) \|_2^2$

Where:

$z \in \mathbb{R}^{d}$ is the code and $\hat{x} \in \mathbb{R}^{D}$ the reconstruction of the input $x \in \mathbb{R}^{D}$ .
$D$ is the input dimension and $d$ the code dimension, with $d \ll D$ in the undercomplete case.
$f_{\theta}$ is the encoder and $g_{\phi}$ the decoder, with trainable parameters $\theta$ and $\phi$ optimized jointly.
$i \in \{1, \ldots, N\}$ indexes the $N$ training examples, and $\mathcal{L}$ is the mean squared reconstruction loss.
$s(x)$ is the anomaly score, and a sample is flagged when $s(x) > \tau$ for a threshold $\tau$ chosen from a high percentile of normal validation scores.
$\tilde{x}$ is a corrupted copy of $x$ (additive noise, masking, or dropout on inputs); the denoising objective still targets the clean $x$ , which is what prevents the identity solution even when $d \geq D$ .

Property	Undercomplete autoencoder	PCA	VAE
Mapping	Nonlinear encoder and decoder, learned by SGD	Single linear projection, closed-form via SVD	Nonlinear, but the encoder outputs a distribution over z
Objective	Reconstruction error only	Maximum retained variance, equivalent to linear MSE	Reconstruction plus a KL term pulling z toward a prior
Latent geometry	Arbitrary, often with holes and unused directions	Orthogonal ordered axes, fully interpretable scale	Smooth and roughly isotropic, so interpolation is meaningful
Sampling new data	Not supported without fitting a prior over codes afterwards	Only under an explicit Gaussian model such as probabilistic PCA	Yes, sample the prior and decode
Typical use	Anomaly scoring, denoising, compact features for indexing	Fast baseline, whitening, exploratory analysis	Latent space for diffusion, generative sampling, recsys
Main caveat	Too much capacity collapses to near-identity and the code stops being informative	Cannot represent curved manifolds; cost grows with feature count	Blurry reconstructions and posterior collapse if the KL weight is too high

August 16, 2026