Author: admin

DL0190 GNN Message Passing Paradigm

What is the Message Passing Paradigm in Graph Neural Networks, and how does it unify node, edge, and graph-level predictions through neighborhood aggregation and update functions?

Answer

Message passing is the observation that almost every graph neural network layer can be written as three functions applied in the same order: a message function that runs once per edge, a permutation-invariant aggregation over each node’s neighborhood, and an update function that mixes the aggregate with the node’s previous state. Stacking $K$ such layers gives every node a representation that summarizes its $K$ -hop neighborhood, so depth and receptive field are the same knob. GCN, GraphSAGE, GAT, and GIN differ only in how they instantiate the message and the aggregator, which is why one implementation of the scatter-gather loop covers all of them. The unification across prediction granularities is equally mechanical, because the encoder is identical in all three cases and only the readout changes: a node label reads $h_v$ directly, an edge or link score reads the pair $(h_u, h_v)$ , and a graph label reads a pooled summary of all node states. Training therefore differs in the loss and label granularity, not in the architecture.

(1) Message Function: $\phi$ runs once per directed edge and may use the source state, the target state, and the edge features, which is how bond types or relation types enter the computation.
(2) Permutation-Invariant Aggregation: the neighborhood is an unordered multiset, so $\bigoplus$ must be sum, mean, max, or attention-weighted sum; anything order-dependent makes the layer ill-defined.
(3) Update Function: $\psi$ combines the old state with the aggregate, usually a linear map plus nonlinearity, and in deep stacks a residual connection to keep the node’s own signal alive.
(4) Depth Equals Receptive Field: after $K$ rounds a node has seen exactly its $K$ -hop subgraph, so long-range tasks need either depth or a shortcut mechanism.
(5) One Encoder, Three Heads: node, edge, and graph predictions are three readouts over the same embedding matrix, which lets you pretrain on one granularity and fine-tune on another.
(6) Cost Scales With Edges: a layer is $O(|E|d + |V|d^2)$ , sparse in the graph and independent of the diameter, which is why message passing scales where dense pairwise attention does not.

Diagram of one message passing layer: a small graph on the left with a target node v receiving solid message arrows from three neighbors and dashed edges to two-hop nodes, feeding a three-box pipeline on the right labeled message function phi per edge, permutation-invariant aggregation with sum mean or max, and update function psi producing the new node state

Figure 1: One layer, three functions. Every incident edge produces a message, the messages collapse into a single vector through a permutation-invariant operator, and the update mixes that vector with the node’s previous state. The dashed two-hop nodes contribute nothing at this layer; they only reach $v$ after a second round, which is what makes depth and receptive field the same quantity.

The reason this paradigm generalizes so well is that it commits to locality and permutation equivariance and nothing else. There is no assumption of a fixed node ordering, a fixed degree, or a fixed graph size, so the same trained weights apply to a 12-atom molecule and a 40-atom molecule. The interesting design freedom sits in the aggregator. Mean pooling normalizes away degree, which helps on citation graphs where degree is a popularity artifact and hurts on tasks where the count itself is the signal. Sum pooling keeps the count and is what makes GIN as discriminative as the 1-Weisfeiler-Lehman test, the theoretical ceiling for standard message passing. Max pooling behaves like a feature detector and is robust to noisy neighbors but blind to multiplicity.

Mathematical Formulation:
$m_{uv}^{(k)} = \phi^{(k)}(h_v^{(k-1)}, h_u^{(k-1)}, e_{uv})$
$a_v^{(k)} = \bigoplus_{u \in \mathcal{N}(v)} m_{uv}^{(k)}$
$h_v^{(k)} = \psi^{(k)}(h_v^{(k-1)}, a_v^{(k)})$
$h_G = R(\{ h_v^{(K)} : v \in V \})$

Where:

$h_v^{(k)} \in \mathbb{R}^{d}$ is the state of node $v$ after $k$ rounds, with $h_v^{(0)}$ the input node features.
$m_{uv}^{(k)}$ is the message sent from neighbor $u$ to $v$ , and $a_v^{(k)}$ is the aggregated neighborhood vector.
$\phi^{(k)}$ and $\psi^{(k)}$ are the learned message and update functions, typically small MLPs or a single linear layer with a nonlinearity.
$\bigoplus$ is a permutation-invariant operator over a multiset, most often sum, mean, or max.
$\mathcal{N}(v)$ is the neighbor set of $v$ and $e_{uv}$ the optional edge feature vector.
$k \in \{1, \ldots, K\}$ indexes rounds, so $K$ is both the layer count and the hop radius of the receptive field.
$R$ is the graph readout, itself permutation invariant, producing the whole-graph vector $h_G$ .

Diagram showing an input graph feeding a stack of K message passing layers that produces a node embedding matrix, which then branches into three heads: a node head applied per node, an edge head applied to the concatenation of two endpoint embeddings, and a graph head applied after a permutation-invariant readout pooling

Figure 2: The encoder is shared and only the readout changes. Node classification consumes $h_v$ , link prediction consumes a symmetric function of the endpoint pair, and graph regression consumes a pooled summary. This is why a single library implements all three task families with one message passing loop and three thin heads.

The Three Readouts:
$\hat y_v = f_n(h_v^{(K)})$
$\hat y_{uv} = f_e([h_u^{(K)} ; h_v^{(K)}])$
$\hat y_G = f_g(h_G)$

For link prediction the pair function should be symmetric on undirected graphs, so practitioners use the Hadamard product $h_u \odot h_v$ or a dot product rather than a raw concatenation, which is order-dependent unless both orders are trained. A concrete instantiation makes the abstraction less slippery. GCN takes the message to be a degree-normalized copy of the neighbor state and folds the update into one linear map:

GCN As A Message Passing Layer:
$c_{uv} = 1 / \sqrt{d_u d_v}$
$h_v^{(k)} = \sigma \Big( W^{(k)} \sum_{u \in \tilde{\mathcal{N}}(v)} c_{uv} h_u^{(k-1)} \Big)$
$\text{cost} = O(K|E|d + K|V|d^2)$

Here $d_u$ is the degree of $u$ and $\tilde{\mathcal{N}}(v)$ includes $v$ itself through the self-loop, so GCN is a fixed-coefficient weighted mean with no learned message. GAT replaces $c_{uv}$ with a learned attention weight, GraphSAGE concatenates the self state instead of summing it in, and GIN uses a sum with an MLP update. Only the choice of $\phi$ , $\bigoplus$ , and $\psi$ changes.

Property	Sum	Mean	Max
Keeps degree information	Yes, the magnitude grows with degree	No, degree is normalized away	No, only the strongest signal survives
Distinguishes multisets	Injective with an MLP update, so 1-WL expressive (GIN)	Confuses {a, a} with {a}	Confuses {a, b} with {a, b, b}
Scale stability	Poor on heavy-tailed degrees, needs normalization	Good, activations stay bounded	Good, but gradients reach one neighbor only
Typical use	Molecular property prediction where atom counts matter	Citation and social graphs with hub nodes	Point clouds and noisy neighborhoods

Two structural limits follow directly from the paradigm rather than from any particular implementation. Over-smoothing means that repeated neighborhood averaging drives node states toward a low-dimensional, degree-dependent subspace, so accuracy on node tasks often peaks at 2 to 4 layers. Over-squashing means that a node’s $K$ -hop neighborhood can grow exponentially while its state stays a fixed $d$ -vector, so information from distant nodes is compressed through bottleneck edges and effectively lost. Both are reasons that deeper is not automatically better, and both motivate residual connections, jumping-knowledge readouts, graph rewiring, and virtual global nodes.

August 20, 2026

DL0189 KAN vs MLP: Kolmogorov-Arnold Networks

How does a KAN (Kolmogorov-Arnold Network) differ from a traditional MLP in its use of learnable splines on edges instead of fixed activations on nodes, and what are the trade-offs in expressivity, interpretability, and scaling?

Answer

An MLP puts learnable scalars on the edges and a fixed nonlinearity on the nodes: every edge is one number in a weight matrix, and every hidden unit applies the same hand-chosen $\sigma$ . A KAN swaps those two roles. Every edge carries its own learnable univariate function $\phi_{l,j,i}$ , implemented as a B-spline of grid size $G$ and order $k$ plus a SiLU residual branch, and every node does nothing but sum its incoming edge outputs. Both families are universal approximators, so the difference is not what can be represented but where the capacity sits and what that placement costs. Moving capacity onto the edges buys a per-edge object you can plot, sparsify, prune, and even snap to a symbolic formula, and it buys a fast spline approximation rate on smooth low-dimensional targets. It costs a factor of about $(G+k)$ in parameters at the same layer shape, roughly an order of magnitude in wall-clock training time because per-edge splines do not collapse into one dense GEMM, and a new hyperparameter (the spline grid) that must cover the actual range of the activations.

(1) Role Swap, Not A New Theorem: the Kolmogorov-Arnold representation theorem motivates the design, but the depth-2 form it guarantees can require pathological inner functions, so KANs generalize it to arbitrary depth and width and rely on smoothness of the target, not on the theorem, for their advantage.
(2) Edge Function Is Spline Plus Residual: each $\phi$ is a learned combination of $G+k$ basis functions added to a scaled SiLU, so a KAN is not activation-free; the fixed nonlinearity survives as a residual path that keeps gradients alive outside the grid.
(3) Nodes Are Pure Summation: no elementwise nonlinearity between layers, which is exactly why the univariate curves are individually meaningful.
(4) Parameter Cost Multiplies: a $64 \times 64$ layer holds 4,160 parameters as an MLP and about 32,768 spline coefficients as a KAN with $G=5, k=3$ .
(5) Interpretability Is A Workflow: L1 plus entropy regularization on the edge functions, pruning of dead edges, then symbolic snapping of each surviving curve to a candidate like $\sin$ , $\exp$ , or $x^2$ .
(6) Scaling Is The Weak Point: per-edge spline evaluation is memory-bound and GEMM-unfriendly, and at matched parameters and FLOPs an MLP still wins on vision, language, and audio benchmarks.

Side-by-side diagram of the same two-input toy layer. On the left an MLP sends two inputs through scalar weights w1 and w2 into a node containing a fixed sigma, then to the output y. On the right a KAN sends each input through a box containing a learnable spline curve drawn over knot ticks, and both spline outputs feed a node that only sums, producing y.

Figure 1: The role swap in its smallest form. In the MLP the edge is a single learnable number and the nonlinearity is a fixed $\sigma$ baked into the node; in the KAN the edge is a learnable spline over a knot grid and the node only adds. Capacity moves from a matrix of scalars to a grid of spline coefficients, which is why the same layer shape costs about $(G+k)$ times more parameters.

Expressivity behaves differently in the two regimes that matter in practice. On smooth, low-dimensional, compositional targets (symbolic regression, ODE and PDE solution operators, small physical laws) the spline basis is close to the right basis, so error falls quickly with parameters and grid extension lets you refine an already trained model by re-fitting a finer grid instead of restarting. On high-dimensional perception data the picture inverts: a controlled comparison at matched parameters and FLOPs found MLPs ahead on machine-vision, language, and audio tasks, with KANs winning only on symbolic formula representation, and KANs forgetting more than MLPs in a standard class-incremental setting. Interpretability is the more robust claim. Because a node only sums, each edge curve is a genuine one-dimensional function of one variable, so you can plot all of them, drive most toward zero with sparsity penalties, prune the graph down to a handful of edges, and read off a formula. That workflow is what made KANs useful as a scientific assistant rather than as a general drop-in replacement for a dense layer.

Mathematical Formulation:
$\mathrm{MLP}(x) = W_L \sigma(W_{L-1} \cdots \sigma(W_1 x))$
$f(x) = \sum_{q=1}^{2n+1} \Phi_q \left( \sum_{p=1}^{n} \phi_{q,p}(x_p) \right)$
$x_j^{(l+1)} = \sum_{i=1}^{n_l} \phi_{l,j,i}\left(x_i^{(l)}\right)$
$\phi(x) = w_b \, \mathrm{silu}(x) + w_s \sum_{m=1}^{G+k} c_m B_m(x)$
$P_{\mathrm{MLP}} = O(L n^2)$
$P_{\mathrm{KAN}} = O(L n^2 (G+k))$
$\ell \propto N^{-4}$

Where:

$x$ is the input vector and $x_j^{(l+1)}$ the $j$ -th activation of layer $l+1$ , obtained by summation only in a KAN.
$W_l$ and $\sigma$ are the MLP’s learnable weight matrices and its fixed elementwise nonlinearity.
$\phi_{q,p}$ and $\Phi_q$ are the inner and outer univariate functions of the Kolmogorov-Arnold representation, with $n$ the input dimension; $\phi_{l,j,i}$ is the learnable edge function from unit $i$ of layer $l$ to unit $j$ of layer $l+1$ .
$B_m$ are B-spline basis functions with local support over the knot grid, $c_m$ their learned coefficients, and $w_b, w_s$ the scales of the SiLU residual and the spline branch.
$G$ is the number of grid intervals and $k$ the spline order, so each edge holds $G+k$ coefficients; typical values are $G=5, k=3$ .
$L$ is depth, $n$ the layer width, and $P$ the parameter count, so the KAN pays the extra factor $(G+k)$ at identical layer shape.
$\ell$ is test error against parameter count $N$ ; the $N^{-4}$ rate is the cubic-spline approximation rate and holds only when the target is smooth and effectively low-dimensional.

Two-panel chart. Left panel: grouped bars comparing an MLP baseline at one times against a KAN, showing eight times the parameters at the same layer shape and about ten times the training time per epoch at the same parameter count. Right panel: log-log plot of test error against parameter count, with a steep KAN curve following an N to the minus four rate marked by grid-extension points at G equals 3, 5, 10 and 20, and a much shallower dashed MLP curve following an N to the minus one rate.

Figure 2: Cost against payoff. At matched layer shape a KAN with $G=5, k=3$ carries about 8x the parameters, and at matched parameter count it trains roughly 10x slower because each edge evaluates its own spline instead of joining one dense matmul. The right panel shows the regime where that price buys something: on a smooth low-dimensional target, grid extension (3 → 5 → 10 → 20) walks the same trained model down a steep spline-approximation curve that an equally sized MLP does not follow.

Property	MLP	KAN
Nonlinearity location	Fixed sigma on every node	Learnable phi on every edge; nodes only sum
Learnable object per edge	One scalar weight	G+k spline coefficients plus base and spline scales
Params for a 64 to 64 layer	4,160	About 32,768 at G=5, k=3 (about 41k with the two scales)
Hardware behaviour	One dense GEMM, cuBLAS and tensor-core friendly	Per-edge basis evaluation, memory-bound, about 10x slower at matched params
Input domain requirement	None; sigma is defined on all of R	Knot grid must cover the activation range, so grid updates or normalization are mandatory
Interpretability route	Inspect weights or use post-hoc attribution; features stay entangled	Plot each curve, sparsify, prune, snap to a symbolic form
Where it wins	Vision, language, audio at matched params and FLOPs; anything throughput-bound	Symbolic regression, small smooth scientific targets, operator learning
Main failure mode	Activation choice is a fixed prior; little internal structure to read	Grid hyperparameters, slow training, and worse forgetting on class-incremental benchmarks

August 20, 2026

DL0188 Liquid Neural Networks

What are Liquid Neural Networks (LNNs), and how do their continuous-time ODE-based dynamics with time-varying parameters differ from fixed-weight RNNs and Transformers?

Answer

A Liquid Neural Network is a recurrent model whose hidden state is defined by an ordinary differential equation in continuous time rather than by a fixed-stride update rule, and whose effective time constant is computed from the current input and state instead of being frozen at training time. The canonical instance is the Liquid Time-Constant (LTC) network, where a neuron leaks toward a resting value at rate $1/\tau$ and is simultaneously driven by a nonlinearity $f(x, I; \theta)$ that also appears in the denominator of the effective time constant, so each neuron can behave like a fast detector on one input and a slow integrator on another. The trained parameters $\theta$ are static after training; what is “liquid” is the input-dependent time constant and synaptic drive, which makes the system a family of dynamics selected on the fly rather than one fixed linear-time-invariant filter. Because the state is a function of physical time, the elapsed interval $\Delta t$ is an explicit argument to a numeric solver, which is what lets LNNs consume irregularly sampled sensor streams without imputation. Against this, a gated RNN applies the same learned transition once per token at an implicit constant stride, and a Transformer has no recurrent state at all: it mixes tokens by attention over the whole prefix, paying $O(L^2)$ prefill and carrying a linearly growing KV cache.

(1) Continuous-Time State: the hidden vector is the solution of an ODE evaluated by a solver, so the model interpolates between observations instead of skipping to the next index.
(2) Liquid Time Constant: $\tau_{sys} = \tau / (1 + \tau f)$ varies per neuron per step, giving input-dependent timescales from one fixed parameter set.
(3) Elapsed Time Is An Input: unequal gaps between samples are handled exactly by integrating over $\Delta t$ , which matters for event streams, medical records, and asynchronous sensors.
(4) Sparse Auditable Wiring: Neural Circuit Policies wire LTC neurons in a sensory, inter, command, motor hierarchy, and a 19-neuron command layer has flown real drones and driven real cars.
(5) Cost Scales With Solver Steps: compute is $O(L k d^2)$ for $k$ solver steps per observation, with no KV cache, but the recurrence is inherently sequential.
(6) Provable Boundedness: the LTC construction bounds both the state and $\tau_{sys}$ , so the dynamics cannot blow up the way an unconstrained neural ODE can.

Flow diagram of a liquid time-constant cell: an irregularly sampled input stream feeds a nonlinear drive function f of state and input, which both drives the ODE state x of t and, through a lower box, sets the liquid time constant tau over one plus tau times f; the state is integrated by an ODE solver taking k fused Euler steps across each delta t, then read out linearly, with a dashed feedback arrow returning the state at the next observation and a timeline below showing unequally spaced observation ticks with grey internal solver steps between them

Figure 1: One nonlinearity does two jobs. The same $f(x, I; \theta)$ supplies the drive toward the resting potential $A$ and sets the effective time constant, and the solver spans whatever $\Delta t$ the sensor happened to deliver, so the grey internal steps, not the sequence index, determine the compute bill.

The contrast with a fixed-weight RNN is easiest to see through the timescale. An LSTM or GRU also has a forget gate that modulates memory retention, but the gate acts on a fixed discrete stride: the model learns one retention profile for “one step” and has no representation of how much real time a step covered. If your samples arrive at 3 ms and then at 400 ms, the gated RNN sees two identical steps unless you hand it the gap as an extra feature and hope it learns the exponential relationship. The LTC instead solves for the exponential, because $\Delta t$ enters the integrator directly. The contrast with a Transformer is structural rather than temporal. Attention is a stateless global mixer: it can reach any earlier token in one hop, which is why it dominates language, but it has no notion of physical duration beyond whatever positional scheme you inject, and its cost grows quadratically with context while its memory grows linearly. An LNN keeps a fixed-size state regardless of horizon, which is exactly what an embedded controller running at 50 Hz for hours needs, and exactly the wrong trade for retrieving a fact from 200,000 tokens of text.

Mathematical Formulation:
$\frac{dx}{dt} = -\frac{x}{\tau} + f(x, I; \theta)\,(A - x)$
$\tau_{sys}(x, I) = \frac{\tau}{1 + \tau f(x, I; \theta)}$
$\frac{\tau}{1 + \tau f_{max}} \leq \tau_{sys} \leq \tau$
$h_t = \sigma(W h_{t-1} + U I_t + b)$

Where:

$x(t) \in \mathbb{R}^{d}$ is the continuous-time neuron state and $I(t)$ the sensory input at that instant.
$f(x, I; \theta)$ is a bounded nonlinearity (typically a sigmoid of an affine map) with static trained parameters $\theta$ ; it appears twice, as the drive and inside the time constant.
$\tau$ is the per-neuron leak time constant, $A$ the resting potential the drive pulls toward, and $f_{max}$ the supremum of $f$ that bounds the fastest achievable dynamics.
$\tau_{sys}$ is the liquid (effective) time constant; the bracketing inequality is what guarantees the state stays bounded and the solver stays stable.
The last line is the fixed-weight RNN baseline: $h_t$ depends on step index $t$ only, with $W, U, b$ applied identically at every step and no $\Delta t$ anywhere in the update.

Two panel chart. Left panel plots the effective time constant tau_sys against input drive f on a logarithmic vertical axis for three base time constants 0.1, 0.5 and 2.0 seconds, all curves collapsing toward one over f as the drive grows. Right panel plots state x of t against time for a square input pulse between one and three seconds, comparing a fixed leaky integrator with a constant one second time constant against a liquid time constant cell whose effective time constant is 0.21 seconds while the input is on and 0.84 seconds when it is off, showing a much faster rise and a partially sustained level

Figure 2: Left, the whole point of the $1 + \tau f$ denominator: a strong input collapses $\tau_{sys}$ toward $1/f$ , so one neuron spans a wide band of timescales. Right, the same input pulse through a fixed leaky integrator and an LTC cell with identical $\tau$ ; the liquid cell charges roughly five times faster while driven and relaxes more slowly once the drive is removed, behaviour a single fixed-weight filter cannot produce.

The practical objection to LNNs is the solver. Every observation costs $k$ evaluations of $f$ , stiff dynamics push $k$ up, and the whole recurrence is sequential, so training cannot be parallelized across time the way attention or a convolutional scan can. Closed-form Continuous-time (CfC) networks answer this by replacing the numeric integration with an analytic approximation of the LTC solution, keeping the input-dependent timescale while deleting the solver from the graph, at reported speedups of one to five orders of magnitude in training and inference over solver-based LTCs.

Closed Form And Cost:
$x(t) = \sigma(-f\,t) \odot g + [1 - \sigma(-f\,t)] \odot h$
$C_{LNN} = O(L\,k\,d^2)$
$C_{TF} = O(L^2 d + L\,d^2)$

Here $f, g, h$ are separate learned heads of $(x, I)$ and $\sigma(-f t)$ acts as a time-dependent gate that interpolates between two candidate states as elapsed time grows, which is the closed-form stand-in for exponential relaxation. Note that $C_{LNN}$ has no term in $L^2$ and no cache: an LNN controller at $d = 64$ can run indefinitely in constant memory, while the Transformer term grows with every token retained.

Property	Liquid (LTC / CfC)	Fixed-weight RNN (LSTM / GRU)	Transformer
State definition	Continuous state from an ODE, evaluated by a solver or a closed-form gate	Discrete hidden and cell state, one update per step	No recurrent state; every token attends over the full prefix
Handling of elapsed time	Delta t is an explicit solver argument, so unequal gaps are exact	Implicit constant stride; gaps must be imputed or appended as a feature	Position index or RoPE; real timestamps must be encoded as extra inputs
Effective timescale	Input-dependent, tau over one plus tau f, bounded above by tau	Set by trained gate weights, modulated only within one fixed stride	None; the mixing radius is the context window
Cost per observation	k solver evaluations, sequential, no cache	One matmul pass, sequential, no cache	Parallel in training, but KV cache grows linearly with L
Typical deployed size	Tens to a few thousand neurons, including 19-neuron flight and driving policies	Thousands to millions of parameters	Hundreds of millions to hundreds of billions
Dominant failure mode	Stiff dynamics inflate solver steps; no parallelism over time	Vanishing gradients and timescale mismatch on long horizons	Quadratic prefill, and no built-in notion of physical duration

August 20, 2026

DL0187 Hardware-Aware Neural Architecture Search

What is Neural Architecture Search (NAS), and how does Hardware-Aware NAS optimize architectures for edge and mobile deployment constraints such as latency, memory, and energy?

Answer

NAS automates architecture design by specifying three things: a search space of candidate operations and connectivity, a search strategy that proposes architectures, and an evaluation method that scores them. Classical NAS maximizes validation accuracy alone, which reliably produces models that are unusable on a phone because accuracy is monotone in capacity. Hardware-aware NAS changes the objective rather than the search algorithm: the target device becomes part of the reward, either as a hard constraint on measured latency or as a multi-objective term that trades accuracy against cost. The critical detail is that FLOPs and parameter counts are poor latency proxies, because depthwise convolutions, squeeze-excite blocks, and grouped convolutions are memory-bandwidth bound rather than compute bound, so the cost signal must come from on-device measurement or a layer-wise latency lookup table (LUT) calibrated on the actual CPU, GPU, DSP, or NPU. Memory enters as a separate ceiling on peak activation footprint (the binding constraint on microcontrollers with a few hundred KB of SRAM), and energy enters through measured joules per inference, which tracks DRAM traffic more than arithmetic. Because a full search costs thousands of GPU-hours per device, production systems amortize it with a weight-sharing supernet that is trained once and then queried per deployment target.

(1) Three Components: search space, search strategy (reinforcement learning, evolution, or gradient-based), and evaluation. The space dominates the outcome, since a badly chosen space caps the achievable Pareto front no matter how good the optimizer is.
(2) FLOPs Are Not Latency: two blocks within 5% of each other in FLOPs can differ by 1.8x in measured milliseconds, so proxy metrics silently select the wrong architecture.
(3) Cost In The Objective: either a hard constraint $\mathrm{Lat}(\alpha) \leq T$ , a soft reward such as MnasNet’s $\mathrm{Acc} \cdot (\mathrm{Lat}/T)^{w}$ , or a differentiable expected latency term added to the loss.
(4) Latency Lookup Tables: profile each candidate operator once per device and sum per-layer times, which makes the cost model cheap and differentiable but blind to operator fusion and thermal throttling.
(5) Memory And Energy Are Separate Constraints: peak activation RAM bounds what fits in SRAM, and energy per inference is dominated by off-chip memory traffic, so neither is implied by a latency target.
(6) Amortization Is The Production Trick: a once-for-all supernet trained with progressive shrinking yields deployable subnets for a new phone in minutes instead of a fresh search per device.

Flow diagram of a hardware-aware NAS loop: a search space of MBConv blocks with kernel and expansion choices feeds a sampler or controller, which emits a subnet that is scored for accuracy by a weight-sharing supernet and for cost by a hardware cost model built from a layer-wise latency lookup table plus on-device runs, both feeding a multi-objective reward that is fed back to the sampler, with the selected subnet fine-tuned, quantized, and deployed

Figure 1: The loop is sample subnet → score accuracy on a weight-sharing supernet → query a hardware cost model → combine into one multi-objective reward → update the sampler. Only the cost model is device-specific, which is why the same search pipeline produces a different winner for a mid-range CPU than for an NPU.

The cost model is where most engineering effort goes. A layer-wise LUT stores the measured runtime of every candidate operator at every feature-map shape on the target, so predicting a candidate’s latency is a table sum instead of a deployment. This is fast enough to put inside a training loop, and because the sum is linear in the per-operator times it can be made differentiable with respect to architecture probabilities. The failure mode is LUT drift: a compiler that fuses convolution with batch normalization and activation, a scheduler that changes clock frequency under load, or a runtime that picks a different kernel for a specific channel count will all break additivity, and errors compound across dozens of small layers. Practical pipelines therefore validate the LUT against end-to-end on-device measurements for a sample of candidates and refit when the mean absolute percentage error drifts past a few percent.

Scatter plot of measured mobile CPU latency in milliseconds against multiply-accumulate cost in MFLOPs for nine candidate blocks, with compute-bound blocks and memory-bound blocks distinguished, and two double-headed arrows marking pairs of blocks that have nearly equal FLOPs but 1.8 to 1.9 times different measured latency

Figure 2: Measured latency versus FLOPs for candidate blocks on one mobile CPU. The ordering is not preserved: a grouped convolution and a squeeze-excite MBConv cost roughly 1.9x and 1.8x the latency of blocks with nearly identical FLOPs, because both are memory-bandwidth and kernel-launch bound. Searching against FLOPs would pick the slow block.

Mathematical Formulation:
$\max_{\alpha \in \mathcal{A}} \mathrm{Acc}(\alpha, w^{*}(\alpha))$
$\text{s.t.}\ \mathrm{Lat}(\alpha, H) \leq T$
$R(\alpha) = \mathrm{Acc}(\alpha) \cdot (\mathrm{Lat}(\alpha)/T)^{w}$
$\mathcal{L}(\theta, \alpha) = \mathrm{CE}(\theta, \alpha) + \lambda \log \mathrm{Lat}(\alpha)$
$\mathrm{Lat}(\alpha) = \sum_{l=1}^{L} \sum_{o \in \mathcal{O}} p_{l,o} \, t_{l,o}$
$\max_{l} (a_{l} + a_{l+1}) \leq M_{\mathrm{SRAM}}$

Where:

$\alpha$ is a candidate architecture drawn from the search space $\mathcal{A}$ , and $w^{*}(\alpha)$ are its weights, either trained from scratch or inherited from a supernet.
$\mathrm{Lat}(\alpha, H)$ is the latency on target hardware $H$ and $T$ the deployment budget, so the same $\alpha$ has a different cost on a CPU, a DSP, and an NPU.
$R(\alpha)$ is the soft multi-objective reward, with exponent $w$ negative (MnasNet uses $w = -0.07$ ) so that exceeding the budget is penalized smoothly rather than rejected outright.
$\mathcal{L}$ is the differentiable alternative: cross-entropy on weights $\theta$ plus a log-latency regularizer weighted by $\lambda$ , which controls where on the Pareto front the search lands.
$p_{l,o}$ is the softmax probability of choosing operator $o \in \mathcal{O}$ at layer $l$ , and $t_{l,o}$ the measured LUT entry for that operator at that layer’s shape, so expected latency is linear in $p$ and its gradient is exactly $t_{l,o}$ .
$a_{l}$ is the activation tensor size at layer $l$ ; since an inplace-scheduled runtime holds an input and an output buffer simultaneously, the peak pair sum must fit the on-chip budget $M_{\mathrm{SRAM}}$ .

Accuracy versus latency scatter and Pareto frontiers for two devices: a mobile CPU frontier and an NPU frontier over the same set of architectures, with a cloud of dominated candidates below, a vertical dashed line at a 20 millisecond budget, and two starred architectures showing that the best subnet under the budget differs between the two devices

Figure 3: The same architecture set has a different Pareto front per device. Architecture Y misses a 20 ms budget on the CPU at 26 ms but clears it on the NPU at 16 ms, so the optimal pick flips. This is why hardware-aware NAS is rerun (or a supernet requeried) for each target rather than solved once.

Budget Decomposition For A 30 ms Camera Frame:
$t_{pre} + t_{infer} + t_{post} \leq 30\ \text{ms}$
$3\ \text{ms} + 22\ \text{ms} + 5\ \text{ms} = 30\ \text{ms}$

The search target $T$ is never the product-level frame budget. Resize, color conversion, and normalization consume a few milliseconds before the network runs, and non-maximum suppression or decoding consumes more afterwards, so the network budget that goes into the reward is what remains. Getting this wrong by 5 ms produces an architecture that is Pareto-optimal against the wrong constraint, which is a more common production failure than a weak search algorithm.

Property	RL / evolutionary controller	Differentiable supernet	Once-for-all supernet
Typical search cost	Thousands of accelerator hours per device and per budget	Roughly 200 to 400 GPU hours, one run per device and budget	About 1,200 GPU hours once, then minutes for each new target
How hardware cost enters	Measured on-device latency of the sampled model inside the reward	Expected latency from a layer-wise LUT, differentiable in the architecture weights	Accuracy and latency predictors queried by cheap evolutionary search
Retraining after search	Full training of the winning architecture	Full retraining of the derived subnet	None required; subnets are directly deployable, fine-tuning optional
Search-time memory	Low, one candidate resident at a time	High, all candidate operators resident unless paths are binarized	High while training the supernet, negligible during per-device search
Main weakness	Cost scales linearly with the number of devices and budgets	LUT drift and weight co-adaptation bias the candidate ranking	Complex progressive-shrinking and distillation recipe, residual supernet gap

August 20, 2026

DL0186 Energy-Based Models and Contrastive Divergence

What is the Energy-Based Model (EBM) formulation, and how does Contrastive Divergence approximate the intractable partition function gradient without sampling full Markov chains?

Answer

An energy-based model refuses to parameterize a normalized density directly. It learns only a scalar energy function $E_{\theta}(x)$ , low for plausible configurations and high for implausible ones, and recovers a density by exponentiating and normalizing. The price of that freedom is the partition function $Z(\theta)$ , an integral over the whole input space that is intractable for anything beyond a toy problem. Differentiating the log-likelihood splits into a positive phase that lowers energy at observed data and a negative phase that raises energy wherever the model currently places mass, and the negative phase is an expectation under $p_{\theta}$ that normally needs MCMC samples at equilibrium. Contrastive divergence keeps the same two-phase update but initializes the chain at the training data rather than at noise, then runs only $k$ transition steps, in practice $k = 1$ . The estimate is biased because the $k$ -step distribution is not the equilibrium one, yet the bias shrinks near the optimum where the data distribution is already close to a fixed point of the sampler, and the variance is far lower than that of a long chain.

(1) Energy, Not Probability: the network outputs one unconstrained scalar per input, so any architecture works and no normalization layer or invertibility constraint is required.
(2) The Partition Function Is The Whole Problem: $Z(\theta)$ is a sum or integral over every possible configuration, so exact likelihood and exact sampling are both out of reach.
(3) Positive And Negative Phase: the exact gradient is a difference of two expectations of $\nabla_{\theta}E$ , one under the data and one under the model, which is why training is often described as pushing energy down at real points and up at fantasy points.
(4) Chain Started At The Data: CD replaces the equilibrium expectation with samples from a chain seeded at $x^{+}$ , which already sits in a high-probability region, so a very short run produces a usable contrast.
(5) Truncation Trades Bias For Cost: per-update cost drops from a full mixing time to $k$ transitions, at the cost of a systematically biased gradient that ignores model mass far from the data.
(6) CD Is An Update Rule, Not A True Gradient: the CD-1 update is not the exact gradient of any objective function, so convergence guarantees are weak and divergence is possible in principle.

Flow diagram with a top lane showing a data batch feeding the positive phase that lowers energy at the data point, and a bottom lane where the same data point initializes a Markov chain of k short steps producing the negative sample, with both phases feeding a single parameter update box

Figure 1: The two-phase update. Exact maximum likelihood would draw the negative sample from the model at equilibrium, costing a full chain run per parameter update. CD keeps the identical update form but seeds the chain at the data and stops after $k$ steps, which turns an expensive unbiased gradient into a cheap biased one.

Mathematical Formulation:
$p_{\theta}(x) = \frac{e^{-E_{\theta}(x)}}{Z(\theta)}$
$Z(\theta) = \int e^{-E_{\theta}(x)}\,dx$
$\log p_{\theta}(x) = -E_{\theta}(x) - \log Z(\theta)$
$\nabla_{\theta}\log Z(\theta) = -\mathbb{E}_{p_{\theta}}[\nabla_{\theta}E_{\theta}]$
$\nabla_{\theta}\mathcal{L} = \mathbb{E}_{p_{\mathrm{data}}}[\nabla_{\theta}E] - \mathbb{E}_{p_{\theta}}[\nabla_{\theta}E]$

Where:

$p_{\theta}(x)$ is the model density and $E_{\theta}(x) \in \mathbb{R}$ the learned energy, with lower energy meaning higher probability.
$Z(\theta)$ is the partition function, an integral over the full input space (a sum for discrete $x$ ) that depends on $\theta$ and therefore cannot be ignored during training.
$x$ is a configuration and $\theta$ the parameters of the energy network.
$\mathcal{L} = -\mathbb{E}_{p_{\mathrm{data}}}[\log p_{\theta}]$ is the average negative log-likelihood, so the first expectation is the positive phase and the second is the negative phase.
The identity $\nabla_{\theta}\log Z = -\mathbb{E}_{p_{\theta}}[\nabla_{\theta}E]$ is what converts an intractable normalizer into an expectation, which is exactly why sampling becomes the bottleneck.

The Contrastive Divergence Approximation:
$p_k = p_0 T_{\theta}^{k}$
$\nabla_{\theta}\mathcal{L}_{\mathrm{CD}} = \mathbb{E}_{p_0}[\nabla_{\theta}E] - \mathbb{E}_{p_k}[\nabla_{\theta}E]$
$\mathcal{L}_{\mathrm{CD}} \approx D_{KL}(p_0 \| p_{\theta}) - D_{KL}(p_k \| p_{\theta})$

Where:

$p_0 = p_{\mathrm{data}}$ is the empirical distribution used as the chain initialization, and $p_k$ is the distribution after $k$ applications of the transition operator.
$T_{\theta}$ is any transition operator leaving $p_{\theta}$ invariant, typically block Gibbs for an RBM or Langevin dynamics for a continuous EBM.
$k$ is the truncation depth, with $k = 1$ the standard choice and larger $k$ reducing bias at linear cost.
As $k \to \infty$ we recover $p_k = p_{\theta}$ and the update becomes the exact likelihood gradient, so CD is a controllable interpolation between a free estimate and an unbiased one.
The two-KL form is the original motivation rather than an exact objective; a small ignored term involving $\partial p_k / \partial \theta$ is what makes CD not the gradient of any function.

The restricted Boltzmann machine is the case where every piece is closed-form, which is why CD was invented there. Because there are no visible-visible or hidden-hidden couplings, the conditionals factorize completely and one Gibbs sweep is a single pass of matrix multiplies plus sigmoids, following the chain v → h → v’ → h’. CD-1 then reduces to the difference between a data correlation and a reconstruction correlation, computable with two forward passes and no backpropagation through a sampler.

Binary RBM Case:
$E_{\theta}(v,h) = -v^{\top}Wh - b^{\top}v - c^{\top}h$
$p(h_j = 1 \mid v) = \sigma(c_j + W_{:j}^{\top}v)$
$\Delta W = \eta\,(v_0 h_0^{\top} - v_1 h_1^{\top})$

Here $v_0$ is the data vector, $h_0 \sim p(h \mid v_0)$ , and $(v_1, h_1)$ come from one further Gibbs sweep. The positive term raises the weight between co-active data units and the negative term subtracts the same statistic measured on the model’s one-step reconstruction, so learning stops precisely when the reconstruction statistics match the data statistics.

One-dimensional energy landscape curve with a deep data basin on the left and a spurious model basin on the right separated by a barrier, showing the data point where the positive phase lowers energy, the k-step chain sample nearby where the negative phase raises energy, and a distant full-chain sample in the spurious basin that a short chain never reaches

Figure 2: The structural blind spot. A $k$ -step chain started at $x^{+}$ stays inside the data basin, so the negative phase only ever penalizes points the model already ranks near the data. A spurious mode behind an energy barrier keeps its low energy indefinitely because CD never samples there, and this is why CD-trained models can assign high likelihood to garbage.

The practical consequences follow directly from that picture. Slow mixing is not a nuisance but the defining constraint, since the time to cross a barrier grows exponentially in its height, so raising $k$ from 1 to 20 buys much less than the cost suggests. Persistent CD attacks the same problem differently by keeping a set of chains alive across parameter updates instead of restarting them at data, which lets the fantasy particles drift into spurious regions given enough updates, at the cost of instability when the learning rate is large relative to the mixing rate. Modern continuous EBMs use short-run Langevin dynamics with a replay buffer plus gradient clipping and energy regularization, and increasingly avoid MCMC entirely through score matching or its denoising variant, which never touches $Z(\theta)$ because the score $\nabla_x \log p_{\theta}(x) = -\nabla_x E_{\theta}(x)$ is independent of the normalizer.

Property	CD-k	Persistent CD	Score matching
Negative sample source	k transitions from the current data batch	Persistent fantasy particles carried across updates	None; no sampling during training
Cost per update	k sweeps, usually one	One sweep, plus buffer memory	One or two extra backward passes for derivatives
Main bias or error	Truncation bias; distant spurious modes never penalized	Chains lag a moving target, so samples are stale	Consistent but blind to relative mass between separated modes
Stability	Very stable, low variance, forgiving of large learning rates	Needs a small learning rate so chains can keep up	Stable, but plain score matching struggles in high dimension without noise
Best fit	RBMs and discrete models with exact block Gibbs	Longer training runs where density quality matters more than speed	Continuous high-dimensional data, the route taken by diffusion models

August 20, 2026

DL0185 CycleGAN Cycle Consistency Loss

How does CycleGAN use Cycle Consistency Loss to train unpaired image-to-image translation, and what are its limitations regarding mode collapse and semantic preservation?

Answer

CycleGAN trains two generators at once, $G: X \to Y$ and $F: Y \to X$ , each with its own PatchGAN discriminator, using only samples from the two marginals $p_X$ and $p_Y$ and never a matched pair. The adversarial terms alone are hopelessly under-constrained, because any mapping whose pushforward equals $p_Y$ is optimal, including one that ignores the input entirely. The cycle consistency loss adds an $\ell_1$ reconstruction penalty in both directions, $F(G(x)) \approx x$ and $G(F(y)) \approx y$ , which forces the two generators to be approximate inverses and therefore rules out constant outputs and arbitrary re-pairing. The critical limitation is that this constraint is invariant to any label-permuting bijection: it enforces invertibility, not meaning, so a translation that swaps content while remaining perfectly reversible pays zero cycle loss. In practice this shows up as residual per-mode collapse (one stripe pattern for every horse), self-adversarial steganography (imperceptible high-frequency codes that let $F$ cheat), and semantic hallucination such as painting zebra texture onto a rider or inventing a lesion in a medical scan.

(1) Two Coupled Mappings, Four Networks: $G$ , $F$ , $D_Y$ , and $D_X$ train jointly, with the adversarial terms fixing the target marginal and nothing else.
(2) Cycle Loss As A Soft Bijection: a bidirectional $\ell_1$ reconstruction penalty weighted by $\lambda = 10$ ties the two generators into an approximate inverse pair.
(3) What It Genuinely Prevents: total collapse is impossible, since a generator that emits one fixed image destroys all input information and $F$ cannot reconstruct $x$ from it.
(4) What It Does Not Constrain: composing $G$ with any measure-preserving permutation of the target domain leaves both losses unchanged, so semantics is unconstrained and only the convolutional inductive bias keeps translations sane.
(5) Self-Adversarial Steganography: the pair can hide a low-amplitude code in the output so reconstruction is near-perfect while the visible translation is wrong, making cycle loss a weak proxy for content preservation.
(6) Deterministic One-To-One Bias: cycle consistency actively penalizes information loss, so many-to-one tasks and geometric or shape changes fail while texture and color remapping succeeds.

Diagram of the CycleGAN objective with a top row showing a real horse photo passing through generator G to a fake zebra and then through generator F back to a reconstruction, a bottom row showing the reverse cycle from a real zebra through F and G, dashed routed paths marking the two L1 cycle losses, and two discriminator boxes in the middle band receiving the fake samples

Figure 1: The full objective in one view. The adversarial terms only ask whether $G(x)$ looks like some member of domain $Y$ , and the cycle terms only ask whether the round trip returns to the start, so no term in the loss ever compares a translated horse against the true zebra version of that same horse.

The published recipe matters because most of the stability comes from implementation choices rather than from the loss on paper. The generators are ResNet encoders with 9 residual blocks at $256 \times 256$ resolution, the discriminators are $70 \times 70$ PatchGAN critics that score local patches instead of the whole image, and the adversarial term uses the least-squares (LSGAN) form rather than the log-likelihood form to avoid vanishing gradients. Discriminators are updated against a buffer of 50 previously generated images, which damps oscillation, and training uses Adam at $2 \times 10^{-4}$ with batch size 1 and a linear learning-rate decay after the first 100 epochs. For photo and painting transfer an extra identity loss weighted at $0.5 \lambda$ penalizes $\|G(y) - y\|_1$ , which is a purely empirical patch that stops the generators from inverting the color palette.

Mathematical Formulation:
$\mathcal{L}_{\mathrm{GAN}}(G, D_Y) = \mathbb{E}_y[(D_Y(y) - 1)^2]$
$+ \mathbb{E}_x[D_Y(G(x))^2]$
$\mathcal{L}_{\mathrm{cyc}} = \mathbb{E}_x \| F(G(x)) - x \|_1$
$+ \mathbb{E}_y \| G(F(y)) - y \|_1$
$\mathcal{L} = \mathcal{L}_{\mathrm{GAN}}(G, D_Y) + \mathcal{L}_{\mathrm{GAN}}(F, D_X)$
$+ \lambda \mathcal{L}_{\mathrm{cyc}}$

Where:

$\mathcal{L}_{\mathrm{GAN}}(G, D_Y)$ is the least-squares adversarial term that $D_Y$ minimizes; the generator instead minimizes $\mathbb{E}_x[(D_Y(G(x)) - 1)^2]$ , which is the non-saturating counterpart.
$x \sim p_X$ and $y \sim p_Y$ are independent draws from the two marginals; there is no joint distribution over pairs anywhere in the objective.
$G$ and $F$ are the two generators, $D_X$ and $D_Y$ the two patch discriminators, all four trained simultaneously.
$\mathcal{L}_{\mathrm{cyc}}$ sums the forward and backward reconstruction errors in $\ell_1$ , chosen over $\ell_2$ because it blurs less.
$\lambda$ is the cycle weight, set to 10 in the paper; raising it sharpens reconstruction and suppresses collapse, but it also makes the generator conservative and can push it toward hidden encodings.

Why The Constraint Is Semantically Blind:
$G' = T \circ G$
$F' = F \circ T^{-1}$
$T_{\#} p_Y = p_Y$
$\mathcal{L}(G', F') = \mathcal{L}(G, F)$

Take any invertible map $T$ on the target domain whose pushforward leaves $p_Y$ unchanged, for example a permutation that sends every image of a single zebra to an image of two zebras and vice versa. Then $G' = T \circ G$ produces samples with exactly the same distribution, so both discriminators are equally fooled, and $F' = F \circ T^{-1}$ still reconstructs the input exactly, so the cycle loss is unchanged. The two solutions are therefore indistinguishable global optima, and the one you actually get is decided by the generator’s limited receptive field, its residual skip structure, and the fact that near-identity mappings are easy to learn. That is a statement about architecture, not about the loss, which is why CycleGAN is reliable for texture and color transfer and unreliable the moment a task requires moving or removing structure.

Two side-by-side panels each showing three boxes of horse-photo content classes on the left and three boxes of zebra-photo content classes on the right; the left panel connects them with parallel content-preserving arrows while the right panel connects them with crossing permuted arrows, and both panels are annotated with identical zero cycle loss and matched marginals

Figure 2: Two mappings between the same two sets. Both are exact bijections and both match the target marginal, so adversarial loss and cycle loss are numerically identical, yet only the left one preserves content. Nothing in the CycleGAN objective distinguishes them, which is the formal root of label flipping and hallucination.

Property	CycleGAN (cycle loss)	CUT (patchwise contrastive)	CycleGAN-Turbo (one-step diffusion)
Content constraint	Round-trip $\ell_1$ reconstruction in both directions	PatchNCE: a patch must match its own location across the translation	Cycle plus perceptual identity on top of a pretrained text-to-image prior
Networks trained	Two generators plus two discriminators	One generator, one discriminator, one small projection head	LoRA adapters on a frozen one-step diffusion backbone
Training cost	Highest of the three: four forward passes per step	Roughly half the memory, single-direction training	Low trainable parameter count, but a large frozen backbone in memory
Output diversity	Deterministic, one output per input by construction	Deterministic, but no invertibility pressure to preserve everything	Deterministic at one step, though the prior supplies plausible detail
Characteristic failure	Hidden encodings, label flipping, refusal to change shape	Weaker global structure retention when the patch head is mistuned	Inherits the prior’s biases and hallucinates plausible but absent detail

August 20, 2026

DL0184 Structure from Motion with Deep Learning

What is Structure from Motion (SfM), and how do deep learning models such as DUSt3R, MASt3R, and VGGT replace traditional keypoint matching and bundle adjustment pipelines?

Answer

Structure from Motion is the problem of recovering both the 3D geometry of a scene (structure) and the camera poses that observed it (motion) from a set of unordered 2D images, up to a global similarity transform. The classical solution is a long pipeline of separately engineered stages: detect keypoints, match descriptors across image pairs, reject outliers with RANSAC on an epipolar constraint, register cameras incrementally with PnP and triangulation, and finally run bundle adjustment, a sparse nonlinear least-squares refinement of all cameras and points against reprojection error. The learned replacements collapse that chain into a regression problem. DUSt3R takes two uncalibrated images and directly predicts dense pointmaps, one 3D point per pixel for both images expressed in the coordinate frame of the first camera, so depth, intrinsics, and relative pose fall out of the output instead of being estimated by geometry. MASt3R adds a dense local-feature head so the same network also emits pixel-accurate correspondences, and VGGT generalizes the idea to many views at once, returning cameras, depth maps, pointmaps, and point tracks for tens or hundreds of frames in a single forward pass with no optimization loop at inference time.

(1) What SfM Actually Solves: recover camera intrinsics and extrinsics plus scene points from images alone, with scale unobservable without an external reference. Everything downstream (MVS, SLAM, NeRF, Gaussian splatting) depends on those poses.
(2) The Classical Chain Is Brittle At Its Joints: keypoints → matching → RANSAC → incremental registration → bundle adjustment. Each stage discards information, and a failure at the front (textureless walls, repeated facades, tiny baselines) cannot be recovered later.
(3) Pointmap Regression Replaces Triangulation: DUSt3R regresses $X^{1,1}$ and $X^{2,1}$ in one shared frame, so the network solves calibration, depth, and relative pose jointly instead of chaining three estimators.
(4) MASt3R Restores Pixel-Level Precision: a 24-dimensional dense descriptor head trained with an InfoNCE matching loss plus fast reciprocal matching gives sub-pixel correspondences that pure 3D regression blurs, and MASt3R-SfM builds a full reconstruction pipeline on top with no RANSAC.
(5) VGGT Removes The Optimizer: alternating frame-wise and global self-attention over all views, with camera and register tokens, yields cameras, depth, pointmaps, and tracks feed-forward in seconds. Bundle adjustment becomes an optional post-hoc refinement rather than the core algorithm.
(6) The Trade-off Is Priors Versus Guarantees: learned models succeed where geometry is degenerate (near-zero baseline, no texture, minimal overlap) but their accuracy is bounded by training data, output is up to scale, and global attention is quadratic in total token count.

Three stacked pipeline rows compared: the classical row runs keypoint detection, pairwise matching over N(N-1)/2 pairs, RANSAC epipolar verification, incremental PnP with triangulation, and bundle adjustment; the DUSt3R and MASt3R row runs an image pair through a shared ViT encoder, two intertwined decoders, pointmap and confidence heads, and a global alignment of pairwise pointmaps in 3D; the VGGT row runs N images through DINOv2 patch tokens with camera tokens, alternating frame-wise and global attention, prediction heads for cameras depth pointmaps and tracks, and an optional bundle adjustment refinement

Figure 1: The same task, three levels of learned integration. The classical row has five hand-designed stages with five distinct failure modes; DUSt3R keeps only a lightweight 3D global alignment over pairwise predictions; VGGT keeps nothing optimization-based at all, and bundle adjustment survives only as an optional refinement.

The conceptual pivot is the pointmap. A depth map is defined only relative to its own camera, so combining two of them requires knowing intrinsics and relative pose first. A pointmap for image $v$ expressed in the frame of camera 1, written $X^{v,1}$ , already lives in a shared space. Once both images are mapped into that space, the classical unknowns are read out rather than solved for: focal length comes from fitting a pinhole model to the ray field of $X^{1,1}$ , relative pose comes from Procrustes alignment or PnP between $X^{2,1}$ and camera 2’s own rays, and correspondence becomes a nearest-neighbour query in 3D rather than a descriptor comparison. That is why DUSt3R needs neither intrinsics nor an initial pose guess, and why it degrades gracefully instead of failing outright when overlap is small.

Diagram showing two pixel grids for images I1 and I2 on the left feeding a network box labelled F of I1 and I2, whose output on the right is a single 3D point cloud drawn in the coordinate frame of camera 1: blue points come from the pixels of I1 and orange points from the pixels of I2, they overlap on a shared floor and wall surface, the camera-1 axes sit at the origin, and a dashed camera-2 frustum marks the pose recovered from the second pointmap rather than supplied as input

Figure 2: Both pointmaps are regressed in one coordinate frame, the frame of camera 1. Because pixels of $I^2$ land directly in that frame, the relative pose is implied by the output rather than required as input, and a match between the two images is just a pair of nearby 3D points.

Classical Objective (Bundle Adjustment):
$E_{BA} = \sum_{(i,j) \in \mathcal{V}} \| \pi_i(X_j) - x_{ij} \|^2$
$\pi_i(X) = K_i (R_i X + t_i)$

Where:

$E_{BA}$ is the total reprojection error, minimized over cameras and points jointly by Levenberg-Marquardt with a Schur-complement trick on the sparse normal equations.
$X_j \in \mathbb{R}^{3}$ is a triangulated scene point and $x_{ij}$ the 2D keypoint where camera $i$ observed it.
$\mathcal{V}$ is the visibility set, that is the surviving inlier matches; every match RANSAC rejected is simply absent, so the objective can only refine, never repair, the front end.
$K_i, R_i, t_i$ are the intrinsics, rotation, and translation of camera $i$ , all unknown and all coupled through the same residuals.

Learned Objective (DUSt3R Pointmap Regression):
$F(I^1, I^2) = (X^{1,1}, X^{2,1}, C^1, C^2)$
$X^{v,1} \in \mathbb{R}^{H \times W \times 3}$
$\ell_i = \| X_i / z - \bar{X}_i / \bar{z} \|$
$\mathcal{L} = \sum_i C_i \ell_i - \alpha \log C_i$

Where:

$F$ is the transformer that consumes the raw pair with no camera parameters, built from a shared ViT encoder and two decoders that exchange information by cross-attention.
$X^{v,1}$ is the pointmap of view $v$ in the frame of camera 1, so $X^{2,1}$ is what encodes the relative pose, and $C^v$ is the per-pixel confidence.
$i$ indexes valid pixels, $\bar{X}_i$ is the ground-truth point, and $z, \bar{z}$ are scale normalizers (mean distance to the origin) that make the loss scale-invariant.
$\alpha > 0$ weights the log-confidence regularizer, letting the network downweight sky, transparent surfaces, and non-overlapping regions instead of hallucinating them with full certainty.
Multi-view reconstruction then optimizes a 3D alignment loss over a graph of pairwise predictions, not a reprojection loss, which is why the global step takes seconds rather than the minutes a full bundle adjustment needs.

VGGT changes the scaling story. Instead of predicting pairs and stitching them, it patchifies all $N$ frames with a DINOv2 backbone, appends per-frame camera tokens and register tokens, and then alternates two attention types for 24 blocks: frame-wise attention that reasons within one image, and global attention that reasons across every token of every image. The first frame’s camera token is given a distinct embedding, which is how the network fixes the output coordinate frame to camera 1 without any positional trick. Heads then decode a 9-dimensional camera parametrization (rotation quaternion, translation, field of view), a DPT depth map, a pointmap, and dense tracking features, all supervised directly.

Feed-Forward Multi-View Prediction (VGGT):
$G(\{I_i\}_{i=1}^{N}) = \{(g_i, D_i, P_i, T_i)\}$
$g_i \in \mathbb{R}^{9}$
$P = (518/14)^2 = 1369$
$N P = 32 \times 1369 = 43808$
$\mathrm{global} : O(N^2 P^2 d)$
$\mathrm{frame} : O(N P^2 d)$

Where:

$g_i$ holds camera $i$ ‘s pose and intrinsics, $D_i$ its depth map, $P_i$ its pointmap, and $T_i$ its tracking features, all produced in one forward pass.
$P$ is the token count per frame at 518-pixel input with 14-pixel patches, and $N P$ the total sequence length that global attention must span.
$d$ is the model width; alternating the cheap frame-wise term with the expensive global term is what keeps hundreds of views tractable, and the quadratic global term is what eventually exhausts GPU memory.

Log-log chart of reconstruction latency in seconds against number of input views for four approaches: classical incremental SfM with exhaustive matching and bundle adjustment rising steeply as N squared, DUSt3R with exhaustive pairwise inference plus global alignment also rising as N squared, MASt3R-SfM with a retrieval-sparsified image graph rising close to linearly, and VGGT feed-forward inference lowest of all but ending at about 200 views where an annotation marks the single-GPU memory wall

Figure 3: Feed-forward inference wins by orders of magnitude at small and medium view counts, but two quadratic terms remain in the field: exhaustive pairing (which retrieval-based graphs in MASt3R-SfM sparsify to roughly $kN$ pairs) and global attention (which sets VGGT’s memory ceiling). VGGT timings follow the published single-GPU measurements; the other curves are illustrative scaling.

Property	Classical SfM (COLMAP-style)	DUSt3R / MASt3R	VGGT
Required input	Images, ideally with EXIF focal priors and good overlap	An uncalibrated image pair, no intrinsics, no pose init	One to hundreds of uncalibrated, unordered images
Core computation	Descriptor matching, RANSAC, and nonlinear least squares	Pairwise transformer inference plus a 3D global alignment	A single forward pass of alternating frame and global attention
Correspondence source	Explicit keypoint descriptors with geometric verification	Nearest neighbours in 3D, plus a dense descriptor head in MASt3R	A tracking head over global features, no matching stage
Scaling in views	Quadratic pairing, plus repeated bundle adjustment as views register	Quadratic in pairs unless a retrieval graph sparsifies it	Quadratic in total tokens, memory-bound rather than time-bound
Dominant failure mode	Hard failure on textureless, repetitive, or low-baseline input	Drift when many pairwise predictions must be fused	Out-of-distribution scenes and a hard memory ceiling on view count
Metric scale	Up to scale unless a known baseline or GPS is supplied	Up to scale by construction (scale-normalized loss)	Up to scale, with scale partly inherited from training priors

August 20, 2026

DL0183 Anchor-Free vs Anchor-Based Detection

How do anchor-free object detectors such as FCOS, CenterNet, and CornerNet differ from anchor-based detectors such as Faster R-CNN and YOLOv3 in box parameterization, label assignment, and training stability?

Answer

The whole difference is what the regression head is measured against. An anchor-based head attaches $k$ prior boxes to every feature-map location and predicts four normalized deltas per prior, so a predicted box only exists relative to a tiled prior whose scales and aspect ratios were chosen in advance from dataset statistics. An anchor-free head has no prior: FCOS predicts the four positive distances from a location to the sides of the object it belongs to, while CornerNet and CenterNet predict heatmap peaks (two corners, or one center) plus a small size and offset regression read off at the peak. Because the reference disappears, the matching rule must change too, so IoU thresholds against priors are replaced by spatial containment plus a per-level size range, and modern detectors replace both with prediction-aware dynamic assignment. Training stability shifts rather than simply improving: anchor-free heads drop the unbounded log-space targets and the anchor hyperparameters, but they need centerness or IoU-aware quality weighting, per-level normalization of the distance targets, and an explicit tie-break for locations that fall inside two objects.

(1) Reference Frame: Anchor-based regresses log-space deltas against a discrete prior set, while anchor-free regresses geometry directly in stride units from a location or a keypoint.
(2) Matching Rule: IoU thresholds are replaced by point-in-box containment, and object scale is routed by FPN level size ranges instead of by anchor size.
(3) Candidate Count: Removing $k=9$ priors per location cuts predictions per level by 9x, which changes the positive/negative ratio the loss has to survive.
(4) Hyperparameter Surface: Scales, aspect ratios, and two IoU thresholds collapse into one set of level ranges plus a center-sampling radius.
(5) Stability Mechanics: Bounded positive distances with a GIoU-style box loss are better conditioned than log deltas, but need per-level scaling or a learnable exponential to keep early gradients sane.
(6) Ambiguity Handling: Two overlapping objects can claim different anchors at the same location, whereas an anchor-free point must be broken by minimum-area assignment, and CenterNet simply collides when two centers land in the same output cell.

Three panels showing the same ground-truth box parameterized three ways: an anchor-based panel with a dashed prior box and an offset arrow to the ground-truth center plus log width and height deltas, an anchor-free panel with a single interior point and four arrows labeled l, t, r, b reaching the four sides, and a keypoint panel with Gaussian peaks at the top-left and bottom-right corners and at the box center with a width-height regression head

Figure 1: The same object, three parameterizations. The anchor head needs a matched prior before its four numbers mean anything, the dense point head predicts four positive distances that are meaningful on their own, and the keypoint head turns detection into peak localization plus a size read-out, so its only remaining prior is the output stride.

Mathematical Formulation:
$t_x = (x - x_a) / w_a$
$t_y = (y - y_a) / h_a$
$t_w = \log(w / w_a)$
$t_h = \log(h / h_a)$

Anchor-Free Distances And Centerness:
$l = (p_x - x_1) / s$
$t = (p_y - y_1) / s$
$r = (x_2 - p_x) / s$
$b = (y_2 - p_y) / s$
$c_x = \min(l,r) / \max(l,r)$
$c_y = \min(t,b) / \max(t,b)$
$c = \sqrt{c_x c_y}$

Where:

$(x,y,w,h)$ is the predicted box and $(x_a,y_a,w_a,h_a)$ the matched anchor, so the four deltas are undefined without an assignment step.
$(t_x,t_y,t_w,t_h)$ are the anchor-based targets; the logarithm makes size ratios additive and keeps targets near zero when the prior already fits, but it is unbounded on both sides for a badly matched prior.
$p = (p_x, p_y)$ is the image-space center of a feature location and $s$ its level stride, typically 8 up to 128 across P3 to P7.
$(x_1,y_1,x_2,y_2)$ are the ground-truth corners; requiring all four of $l, t, r, b > 0$ is exactly the point-in-box positive test that replaces the IoU threshold.
$c \in (0,1]$ is centerness, multiplied into the classification score at inference so boxes regressed from near an object border are down-ranked before NMS.
A point is routed to level $j$ only if $\max(l,t,r,b)$ falls inside that level’s size interval $(m_{j-1}, m_j)$ , which is the anchor-free replacement for choosing anchor scales.

Assignment is where the two families really diverge. Faster R-CNN and YOLOv3 label a prediction by measuring IoU between a fixed prior and the ground truth, which needs a positive threshold, a negative threshold, an ignore band in between, and a rescue rule so every object keeps at least its best anchor. FCOS labels by geometry alone, then narrows the positive set with center sampling (only points within a radius of a few strides from the object center), which raises average positive quality without touching IoU at all. Keypoint methods go further and effectively skip matching: CenterNet splats a Gaussian at the object center on a stride-4 heatmap and treats the single peak cell as the only positive, which is why it needs no NMS but breaks when two centers quantize to the same cell. The lineage since then is a steady handover of the assignment decision from hand-set priors to the model itself: fixed anchors → dense point containment → adaptive statistics in ATSS → prediction-aware costs in SimOTA and TOOD.

Three 12 by 8 feature grids with the same ground-truth box overlaid. The first grid shows dashed anchor boxes of three aspect ratios at one cell and twelve shaded positive cells selected by an IoU threshold. The second grid shades all twenty cells whose centers fall inside the box in light gray and highlights the central three by three block kept by center sampling. The third grid highlights six irregularly placed cells chosen by a prediction-aware cost.

Figure 2: One box, three positive sets. IoU matching scores 9 anchors per cell and keeps a handful, so the imbalance is roughly 1:71 here; center sampling keeps 9 of 96 points with no IoU computation at all; dynamic top-k chooses a variable number of positives from the model’s own cost, which removes thresholds but makes the label set change as training proceeds.

Candidate Count On A RetinaNet-Style FPN (800 x 1024 Input):
$12800 + 3200 + 800 + 208 + 56 = 17064$
$A_{\mathrm{anchor}} = 9 \times 17064 = 153576$
$A_{\mathrm{free}} = 17064$

The count explains most of the practical differences. With about 154k anchors and a few dozen positives per image, an anchor-based one-stage head lives or dies on focal loss or a mined 1:3 negative ratio, and it pays memory and time for an IoU matrix between every anchor and every ground-truth box. Dropping to 17k points removes that matrix, cuts head parameters by the same factor, and makes the box branch predict bounded, strictly positive quantities that pair naturally with an IoU or GIoU loss instead of a smooth-L1 on log deltas. The cost is that scale and ambiguity handling become explicit design choices: level size ranges decide which stride sees an object, and a point inside two boxes is assigned to the smaller-area ground truth, a rule that still degrades on heavily nested or crowded scenes.

Property	Anchor-based (Faster R-CNN, YOLOv3)	Anchor-free dense point (FCOS, ATSS)	Keypoint (CenterNet, CornerNet)
Box parameterization	Four deltas per prior, width and height in log space	Four positive distances to the sides, in stride units	Heatmap peak plus regressed size and sub-pixel offset
Positive assignment	IoU above 0.5 to 0.7, ignore band, best-anchor rescue	Point inside the box, narrowed by center sampling	Exactly one peak cell per object, Gaussian-weighted focal loss
Scale handling	Anchor scales and aspect ratios per level, tuned on the dataset	FPN level size ranges on max(l, t, r, b)	Single high-resolution stride-4 map, no level routing
Candidates per image	About 154k for 9 anchors on P3 to P7	About 17k, one prediction per location	One 200 x 256 heatmap per class
Quality calibration	Objectness or class score, optionally IoU prediction	Centerness or IoU branch multiplied into the score	Peak value itself acts as the confidence
Duplicate removal	NMS, mandatory because many priors fire per object	NMS, since several nearby points stay positive	3 x 3 max-pool peak extraction, no IoU-based NMS
Dominant failure mode	Unusual aspect ratios or tiny objects match no prior, so recall drops silently	Ambiguous points in overlapping boxes, resolved only by min-area	Two centers colliding in one stride-4 cell, or wrong corner grouping

August 20, 2026

DL0182 Semantic, Instance, and Panoptic Segmentation

What are the differences between Semantic, Instance, and Panoptic Segmentation, and when is each task formulation used in practice, for example inside an autonomous-driving perception stack?

Answer

The three tasks differ in what a label is allowed to be. Semantic segmentation assigns every pixel exactly one class out of $K$ and has no notion of object identity, so two touching cars collapse into a single connected “car” region. Instance segmentation does the opposite: it detects and masks each countable object separately with a confidence score, but it only covers thing classes, ignores amorphous stuff such as road, sky, and vegetation, and its masks may overlap each other or leave pixels uncovered. Panoptic segmentation is the union of the two: every pixel receives one pair $(c_p, z_p)$ of class plus instance id, stuff classes get a single segment with no id, things get one segment per object, and the output is a strict partition of the image with no overlaps and no unlabeled pixels except an explicit void region. Which formulation you pick follows the consumer of the output, not fashion. A planner that needs to count and track individual vehicles needs ids, whereas a free-space or sky-replacement module only needs a region mask.

(1) Label Space: semantic returns a class per pixel, instance returns a set of scored binary masks with classes, and panoptic returns a class plus an id per pixel.
(2) Things Versus Stuff: instance segmentation is defined only on countable things; semantic segmentation handles both but cannot separate two instances; panoptic handles both and separates instances.
(3) Overlap Constraint: instance masks are independent and can overlap, while panoptic forces a non-overlapping partition, which means any fusion of two heads must resolve conflicts explicitly.
(4) Metrics Differ In Kind: semantic uses mIoU, instance uses mask AP averaged over IoU thresholds and scores, and panoptic uses PQ at a single fixed matching threshold with no score sweep.
(5) Annotation Cost: stuff-only masks are cheap polygon paint, per-instance boundaries between adjacent same-class objects are the most expensive labels in the dataset.
(6) Practical Selection: use semantic for region questions, instance for counting and tracking, and panoptic when a downstream module needs one consistent scene interpretation per pixel.

Three panels showing the same toy scene of sky, road, two touching cars, and a person labeled three ways: semantic where both cars form one car region, instance where only the two cars and the person get scored masks that slightly overlap while sky and road are ignored, and panoptic where every pixel receives one class plus instance id with the two cars separated by a hard boundary

Figure 1: One scene, three label spaces. The interesting pixels are the ones on the boundary between the two touching cars: semantic segmentation is structurally unable to place that boundary, instance segmentation places it but may let the two masks overlap and says nothing about road or sky, and panoptic segmentation is required to place it and to label every remaining pixel exactly once.

In practice the formulation is chosen by the consumer. Free-space and drivable-area estimation, land-cover mapping from satellite imagery, portrait or sky matting, and organ or tumor delineation are region questions, so semantic segmentation is sufficient and its cheaper labels are a real advantage. Counting, tracking, and grasping are identity questions: counting cells in a microscopy image, tracking each pedestrian across frames, or picking one item out of a bin all require per-object masks, so instance segmentation (or its video extension) is the right task. Panoptic segmentation earns its extra cost when a single downstream consumer must be handed one coherent interpretation of every pixel, which is why it is the natural output format for driving perception and robot scene understanding, where the same map must answer both “is this pixel drivable” and “which vehicle is this”. A useful diagnostic question is whether two adjacent objects of the same class must ever be told apart. If the answer is no, panoptic annotation is money spent on a distinction nobody reads.

Mathematical Formulation:
$f_{\mathrm{sem}}(p) = c_p$
$f_{\mathrm{ins}} = \{(m_j, c_j, s_j)\}_{j=1}^{M}$
$f_{\mathrm{pan}}(p) = (c_p, z_p)$
$\mathrm{mIoU} = \frac{1}{K}\sum_{k=1}^{K}\frac{TP_k}{TP_k + FP_k + FN_k}$
$\mathrm{PQ} = \mathrm{SQ} \times \mathrm{RQ}$
$\mathrm{SQ} = \frac{1}{|TP|}\sum_{(g,q) \in TP}\mathrm{IoU}(g,q)$
$\mathrm{RQ} = \frac{|TP|}{|TP| + \frac{1}{2}|FP| + \frac{1}{2}|FN|}$

Where:

$p$ is a pixel, $c_p$ its class, and $z_p$ its instance id, which is undefined (shared) for stuff classes and unique per object for thing classes.
$K$ is the number of classes, split into disjoint thing and stuff subsets by the dataset definition rather than by the model.
$m_j$ is the $j$ -th predicted binary mask, $c_j$ its class, and $s_j$ its confidence; the $M$ masks are independent, so they may overlap and their union need not cover the image.
$TP$ , $FP$ , and $FN$ in PQ count segments, not pixels, with a ground-truth segment $g$ and a prediction $q$ matched when $\mathrm{IoU}(g,q) > 0.5$ ; that threshold makes the matching provably unique because panoptic segments cannot overlap.
$\mathrm{SQ}$ is segmentation quality, the mean IoU of matched pairs, and $\mathrm{RQ}$ is recognition quality, the F1 score over segments; PQ is reported per class and then averaged, and often split into $\mathrm{PQ}^{\mathrm{th}}$ and $\mathrm{PQ}^{\mathrm{st}}$ .
$TP_k$ in mIoU counts pixels of class $k$ , which is why mIoU is blind to how many objects a class region contains.

Worked Example (One Image):
$\mathrm{SQ} = 3.10 / 4 = 0.775$
$\mathrm{RQ} = 4 / (4 + 1 + 1) = 0.667$
$\mathrm{PQ} = 0.775 \times 0.667 = 0.517$

Four matched segments whose IoUs sum to 3.10 give an SQ of 0.775, and two false positives plus two false negatives each contribute a half count to the RQ denominator, so RQ is 0.667 and PQ lands at 0.517. The decomposition is the practically useful part: a high SQ with a low RQ means the masks are accurate but segments are being missed or hallucinated, which points at the classification and duplicate-removal path, while a high RQ with a low SQ means the right objects are found with sloppy boundaries, which points at output resolution and boundary supervision.

Diagram of panoptic quality computation with a left column of six ground-truth segments and a right column of six predicted segments, arrows joining four matched pairs labeled with IoU values 0.92, 0.81, 0.74 and 0.63, a red dashed pair at IoU 0.38 that fails the threshold and counts as both a false positive and a false negative, one unmatched ground-truth sky segment, one spurious truck prediction, and a side panel computing SQ 0.775, RQ 0.667 and PQ 0.517

Figure 2: PQ is a segment-level bipartite matching followed by a product of two interpretable factors. The pair at IoU 0.38 is the detail that separates candidates who have read the metric from those who have not: it is not a weak match, it is no match at all, and it is charged once as a false positive and once as a false negative.

Property	Semantic	Instance	Panoptic
Output per pixel	One class id	Zero, one, or several scored masks	Exactly one (class, instance id) pair
Stuff classes	Covered	Not part of the task	Covered as one segment per class
Separates same-class objects	No	Yes	Yes, and mandatory
Overlapping masks	Impossible by construction	Allowed and common	Forbidden, conflicts must be resolved
Standard metric	mIoU over pixels	Mask AP swept over IoU and score	PQ = SQ x RQ at IoU above 0.5
Classic architecture	FCN, DeepLab, per-pixel softmax	Mask R-CNN, box then mask head	Panoptic FPN, or mask-classification with queries
Typical use	Free space, land cover, matting, organ delineation	Counting, tracking, robotic grasping	Driving and robot scene understanding, one map for all consumers

August 20, 2026

DL0181 Label Smoothing and Model Calibration

What is Label Smoothing, and how does it affect model calibration, overconfidence, and the trade-off between accuracy and log-likelihood?

Answer

Label smoothing replaces the one-hot target with a mixture of the one-hot vector and the uniform distribution, so the true class receives $1-\epsilon+\epsilon/K$ and every other class receives $\epsilon/K$ . Written this way it is exactly standard cross-entropy plus an $\epsilon$ -weighted KL term pulling the prediction toward uniform, which gives the loss a finite minimizer instead of one that is reached only as the true-class probability approaches 1. That single change removes the pressure that makes logit gaps grow without bound after the argmax is already correct, so the model stops becoming more confident with more training and the classic overconfidence of modern networks largely disappears. The consequences are asymmetric: Expected Calibration Error (ECE) usually drops sharply and top-1 accuracy is flat or slightly better, but the hard-label negative log-likelihood gets worse, because a converged smoothed model deliberately withholds probability mass from the correct class. The Transformer paper states this trade-off explicitly, using $\epsilon=0.1$ and noting that it hurts perplexity while improving accuracy and BLEU.

(1) Target Mixing: the label becomes $(1-\epsilon)y+\epsilon/K$ , which is equivalent to cross-entropy plus $\epsilon\,\mathrm{KL}(u\,\|\,p)$ up to an additive constant.
(2) Bounded Logit Gap: the optimum sits at a finite logit difference, roughly 9.1 nats for $\epsilon=0.1$ and $K=1000$ , so logit norms stop inflating.
(3) Confidence Ceiling: a well-fit smoothed model cannot report top-class confidence above $1-\epsilon+\epsilon/K$ , which is 0.900 at $\epsilon=0.1$ .
(4) Calibration Improves, Then Overshoots: ECE is U-shaped in $\epsilon$ , moving the model from overconfident through calibrated to systematically underconfident.
(5) Log-Likelihood Penalty: the confidence ceiling implies a hard-label NLL floor of $-\log(1-\epsilon+\epsilon/K)$ , about 0.105 nats per example at $\epsilon=0.1$ .
(6) Information Erasure: smoothing equalizes the wrong-class logits, which tightens penultimate-layer clusters and damages knowledge distillation and feature transfer.

Reliability diagram plotting bin accuracy against mean predicted confidence for three training settings: hard labels sit below the perfect-calibration diagonal indicating overconfidence, label smoothing with epsilon 0.1 tracks the diagonal closely and stops at confidence 0.900, and label smoothing with epsilon 0.3 sits above the diagonal indicating underconfidence and stops at confidence 0.700

Figure 1: Reliability curves for the same architecture under three targets. Hard labels fall below the diagonal (confidence exceeds accuracy), $\epsilon=0.1$ tracks the diagonal, and $\epsilon=0.3$ lands above it. Note the truncated horizontal extent: each smoothed curve stops at its confidence ceiling $1-\epsilon+\epsilon/K$ , 0.900 and 0.700 for $K=1000$ . Curves are illustrative of published trends rather than a single benchmark run.

The mechanism is easiest to read off the gradient with respect to the true-class logit, which is $p_y-1$ under hard labels and $p_y-(1-\epsilon+\epsilon/K)$ under smoothing. The hard-label gradient is strictly negative for every finite logit, so optimization keeps pushing the correct logit away from the others long after the prediction is right, and confidence keeps drifting upward while accuracy has already plateaued. The smoothed gradient changes sign once the model reaches the target probability, which pins confidence and caps the logit gap. The wrong-class gradients change too: each incorrect logit is pulled toward the same small target $\epsilon/K$ , so the model is discouraged from expressing that “husky” is much closer to “wolf” than to “airliner”. That equalization is exactly the property Müller and colleagues identified as the reason a label-smoothed teacher makes a worse distillation teacher than a hard-label teacher of identical accuracy.

Mathematical Formulation:
$y^{\mathrm{LS}}_k = (1-\epsilon)\,y_k + \epsilon/K$
$\mathcal{L}_{\mathrm{LS}} = -\sum_{k=1}^{K} y^{\mathrm{LS}}_k \log p_k$
$\mathcal{L}_{\mathrm{LS}} = (1-\epsilon)\mathcal{L}_{\mathrm{CE}} + \epsilon\,\mathrm{KL}(u\,\|\,p) + c$
$p^{\star}_{y} = 1 - \epsilon + \epsilon/K$
$z_y - z_j = \log\left(\frac{K(1-\epsilon)+\epsilon}{\epsilon}\right)$
$\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{N}\left|\mathrm{acc}(B_m) - \mathrm{conf}(B_m)\right|$

Where:

$y \in \{0,1\}^{K}$ is the one-hot label and $y^{\mathrm{LS}}$ the smoothed target over $K$ classes.
$\epsilon \in [0,1)$ is the smoothing strength, $\epsilon=0$ recovering ordinary cross-entropy; $0.1$ is the near-universal default.
$p_k = \mathrm{softmax}(z)_k$ is the predicted probability, $z$ the logits, $y$ the index of the correct class and $j$ any incorrect class.
$u_k = 1/K$ is the uniform distribution, and $c = \epsilon H(u)$ is a constant independent of the parameters, which is why the objective is cross-entropy plus a uniform-KL penalty.
$p^{\star}_{y}$ is the per-example minimizer, giving the finite logit gap in the fifth line; the gap grows only logarithmically as $\epsilon$ shrinks.
$B_m$ is the set of the $N$ validation examples whose top confidence falls in bin $m$ of $M$ (typically $M=15$ ), and ECE is the confidence-weighted gap plotted in Figure 1.

Log-Likelihood Floor At $\epsilon=0.1$ , $K=1000$ :
$p^{\star}_{y} = 0.9 + 0.0001 = 0.9001$
$-\log(0.9001) = 0.1052$
$\exp(0.1052) = 1.111$

The third line is the practical reading for sequence models: a converged smoothed model pays roughly 11% higher perplexity than its own hard-label counterpart would at the same accuracy, purely from the withheld mass, which is why translation systems that smooth at training time are usually evaluated with BLEU or COMET rather than perplexity. Note also that this floor is a property of the optimum, not a hard constraint on the network: a single global temperature fitted on held-out data can sharpen the smoothed logits back and recover most of the lost likelihood. That observation is the reason smoothing is not the right tool if calibrated probabilities are the goal, since post-hoc temperature scaling reaches comparable ECE at zero training cost, one scalar parameter, and no distortion of the wrong-class ranking.

Three side-by-side line charts against smoothing strength epsilon from 0 to 0.4: hard-label negative log-likelihood dips slightly then rises while an analytic floor curve rises monotonically, expected calibration error forms a U shape with a minimum near epsilon 0.1, and top-1 accuracy stays on a flat plateau before declining past epsilon 0.2

Figure 2: The three metrics disagree about the best $\epsilon$ . ECE is U-shaped with a minimum near $0.05$ to $0.1$ , hard-label NLL dips only briefly before the analytic floor $-\log(1-\epsilon+\epsilon/K)$ dominates, and accuracy is flat across a broad plateau. Choosing $\epsilon$ therefore means choosing which metric you are optimizing. Values are illustrative of published sweeps.

Property	Label smoothing	Temperature scaling	Confidence penalty / focal loss
When applied	Training time, changes the targets	Post-hoc, one scalar fitted on a held-out split	Training time, changes the loss on the prediction side
Effect on accuracy	Flat to slightly better; degrades past roughly 0.2	Exactly zero, the argmax is invariant to a positive temperature	Task dependent; focal loss helps mainly under heavy class imbalance
Effect on hard-label NLL	Worse at convergence, floored at $-\log(1-\epsilon+\epsilon/K)$	Directly minimized by the fitting objective, so it improves	Usually worse, for the same mass-withholding reason
Logit ranking preserved	No, wrong-class logits are equalized	Yes, it is a monotone rescaling of all logits	Partially, the entropy term flattens the tail
Main failure mode	Underconfidence at large $\epsilon$ ; weaker distillation teacher and weaker transfer features	One global scalar cannot fix per-class or per-slice miscalibration, and it needs a clean validation split	Extra hyperparameter with little calibration gain over a fitted temperature

August 20, 2026