Category: Medium

DL0132 LLM Serving Latency: TTFT and ITL

How do you measure and reduce Time to First Token (TTFT), Inter-Token Latency (ITL), and latency variance in a production LLM API such as a streaming chat endpoint served with vLLM?

Answer

TTFT is the wall-clock time from request arrival to the first streamed token and ITL is the gap between consecutive tokens after that, and they are separate metrics because they are produced by two phases with two different hardware bottlenecks. Prefill processes the whole prompt in one or a few forward passes and is compute-bound, so $T_{ttft}$ grows roughly linearly with prompt length; decode generates one token per forward pass and is memory-bandwidth-bound, so $T_{itl}$ is floored by the time to stream the weights and KV cache out of HBM. Measurement must happen at the streaming boundary and be reported as a distribution, namely p50, p95 and p99 merged from histograms and bucketed by input and output length, because a p99 computed over a mix of 200-token and 100k-token prompts is meaningless. Reduction is metric-specific: prefix caching and chunked prefill cut TTFT, FP8 quantization and speculative decoding cut ITL, and the variance work is almost entirely about removing things that can stall the decode loop, namely unchunked prefills, KV cache preemption, and queueing above roughly 70% utilization. The metric that actually decides capacity is goodput, the request rate you can sustain while both the TTFT and ITL SLOs hold at p99, not raw tokens per second.

(1) Two Phases, Two Bottlenecks: prefill is compute-bound and sets TTFT, decode is bandwidth-bound and sets ITL, so a change that helps one often does nothing for the other.
(2) Measure Distributions, Not Means: record one TTFT scalar plus an ITL vector of length $N_{out} - 1$ per request, and merge t-digest or HDR histograms across replicas instead of averaging per-replica p99s.
(3) TTFT Is Mostly Queueing Under Load: split server-side spans into $T_{queue}$ and $T_{prefill}$ , because past about 70% utilization the queue term dominates and no kernel optimization will help.
(4) The ITL Tail Is Interference, Not Steady State: p50 ITL reflects the decode step, while p99 ITL reflects the longest single stall, typically a colliding prefill or a preempted request being recomputed.
(5) Knobs Map To Metrics: prefix caching, chunked prefill and admission control for TTFT; quantization, GQA and speculative decoding for ITL; batch caps and prefill-decode disaggregation for variance.
(6) Optimize Goodput: report the sustained rate under both SLOs simultaneously, since raising batch size always trades TTFT and ITL tails for throughput.

Two timeline rows for one decoding request. In the baseline row, queue and prefill produce the first token, then four decode steps are interrupted by one wide unchunked prefill block that stalls the decode loop, creating a large inter-token gap. In the chunked-prefill row, the same prefill is split into six narrow chunks interleaved with decode steps, so every inter-token gap stays small.

Figure 1: TTFT is queue time plus prefill time; ITL is whatever happens between two decode steps of the same request. A single unchunked prefill scheduled into the shared loop stalls every decoding stream at once, which is the dominant source of p99 ITL. Splitting it into token-budgeted chunks bounds the worst stall to one chunk, at a small throughput cost from re-reading weights more often.

Instrument at the streaming boundary rather than inside the model: timestamp request arrival, the first SSE data event, and every subsequent event, then derive TTFT and the ITL vector client-side while the server emits its own queue, prefill and decode spans for the same request ID. The two views disagree in informative ways, because proxy buffering (an nginx deployment with proxy_buffering on) and a slow client event loop both inflate measured ITL with zero GPU involvement. For load testing, drive the endpoint with realistic input and output length distributions, since synthetic fixed-length prompts hide exactly the length-mixing effects that create the real tail; the vLLM benchmark_serving script and NVIDIA GenAI-Perf both report TTFT and ITL percentiles at a fixed request rate, which is the right shape of experiment. The variance checklist beyond scheduling is mundane but produces most of the surprises in practice: KV cache preemption and recompute when cache headroom runs out, ITL growing with batch size as decode drifts toward compute-bound, CUDA graph misses on uncaptured batch shapes, tensor-parallel all-reduce jitter, autoscaler cold starts that load tens of gigabytes of weights, and speculative decoding turning ITL bimodal because accepted-draft steps emit several tokens at once while rejected steps emit one.

Mathematical Formulation:
$T_{e2e} = T_{ttft} + (N_{out} - 1)\, \overline{T_{itl}}$
$T_{ttft} = T_{queue} + T_{prefill}$
$T_{prefill} \approx \frac{2 N_{in} P}{C_{eff}}$
$T_{itl} \geq \frac{B_{w} + B_{kv}}{BW}$
$T_{itl}^{spec} \approx \frac{T_{step}}{\mathbb{E}[n_{acc}]}$
$G = \lambda \cdot \Pr(T_{ttft} \leq S_{t},\ T_{itl} \leq S_{i})$

Where:

$T_{e2e}$ is end-to-end request latency, $T_{ttft}$ the time to first token, and $\overline{T_{itl}}$ the mean inter-token latency (its reciprocal is the perceived tokens per second of one stream).
$T_{queue}$ is scheduler wait time and $T_{prefill}$ the prompt forward pass; separating them is the single most useful piece of instrumentation, because they need opposite fixes.
$N_{in}$ and $N_{out}$ are prompt and completion token counts, $P$ the parameter count, and $C_{eff}$ the achieved (not peak) FLOP/s; the factor 2 counts one multiply and one add per parameter per token.
$B_{w}$ and $B_{kv}$ are the weight and KV cache bytes read per decode step and $BW$ is per-device HBM bandwidth, so this inequality is the bandwidth floor on ITL that no batching removes.
$T_{step}$ is one verification forward pass and $\mathbb{E}[n_{acc}] \geq 1$ the expected accepted tokens per step under speculative decoding, which divides effective ITL but widens its distribution.
$\lambda$ is offered request rate, $S_{t}$ and $S_{i}$ the TTFT and ITL SLO thresholds, and $G$ the resulting goodput, the only capacity number worth putting in a dashboard.

Plugging in real hardware makes the two bottlenecks concrete. A 70B model in BF16 needs roughly 140 GB of weights, so on eight H100s with tensor parallelism each device reads about 17.5 GB per decode step against roughly 3.35 TB/s of HBM, giving a floor near 5 ms per token before all-reduce and KV reads, which is why a single stream tops out around 100 to 150 tokens per second in practice. The same eight GPUs prefill a 2048-token prompt in about $2 \cdot 2048 \cdot 70$ B FLOPs, roughly 287 TFLOPs, which at an achieved 3.2 PFLOP/s is about 90 ms; a 32k-token prompt is 16 times that, which is exactly why long-context traffic destroys a shared TTFT SLO unless prefixes are cached or prefill is disaggregated. Note that FP8 weights halve $B_w$ and therefore the ITL floor, but barely move TTFT, while prefix caching can cut TTFT by an order of magnitude on a 1500-token shared system prompt and does nothing for ITL.

Two line charts against offered request rate. Left panel shows p50 and p99 TTFT in milliseconds rising sharply near capacity with a dotted 1500 ms SLO line and a vertical marker at the crossing rate. Right panel shows p50 and p99 inter-token latency in milliseconds with a dotted 80 ms SLO line crossed at a lower request rate, marking the binding constraint.

Figure 2: Both p50 curves stay flat while the p99 curves bend upward, because queueing and interference hit the tail first. Here the ITL SLO is crossed at a lower rate than the TTFT SLO, so goodput is ITL-bound and the correct response is to cap batch size or disaggregate, not to add prefill compute.

Technique	Metric it moves	What it costs
Prefix / KV cache reuse	TTFT p50 and p99 on shared system prompts, often 5x to 10x	HBM held by cached blocks, eviction policy tuning, cross-tenant isolation concerns
Chunked prefill	ITL p99 (bounds the worst decode stall to one chunk)	Slightly higher TTFT and a few percent lower throughput from repeated weight reads
Prefill-decode disaggregation	Both tails, by giving each phase its own pool and SLO	A KV cache transfer per request, duplicated weights, needs fast interconnect
Speculative decoding	ITL p50, roughly divided by accepted tokens per step	Bimodal ITL, wasted FLOPs on rejection, gains shrink at large batch sizes
FP8 or INT4 weight quantization	ITL floor, by halving or quartering bytes read per step	Accuracy regression that must be evaluated per task, calibration pipeline
Batch cap plus admission control	Latency variance, by refusing to accept work that will miss the SLO	Lower throughput and visible queueing or 429s, so it needs a priority policy

August 17, 2026

DL0131 DoRA: Weight-Decomposed LoRA

Explain DoRA (Weight-Decomposed Low-Rank Adaptation). How does separating weight magnitude from direction improve LoRA fine-tuning stability?

Answer

DoRA reparameterizes every pretrained weight matrix into a magnitude vector and a directional matrix, and then lets LoRA update only the direction. Concretely it writes $W_0 = m \frac{V}{\|V\|_c}$ , where $\|\cdot\|_c$ is the column-wise L2 norm, the trainable vector $m \in \mathbb{R}^{1 \times k}$ is initialized to $\|W_0\|_c$ , and the low-rank product $BA$ is added to the direction before renormalization. The motivation comes from a weight decomposition analysis: when full fine-tuning updates a layer, the magnitude change $\Delta M$ and the direction change $\Delta D$ are negatively correlated (a large rotation with only slight rescaling, or the reverse), while plain LoRA shows a positive, roughly proportional relation, so it can essentially only push both quantities in the same direction. Giving magnitude its own $k$ parameters removes that coupling, and the renormalization makes the gradient reaching the directional component orthogonal to the current direction, which is the classic weight-normalization conditioning effect: the low-rank branch can rotate the weight without simultaneously inflating its scale. In practice this buys a learning pattern that mirrors full fine-tuning, noticeably better accuracy at low rank (the DoRA paper reports about a 3 to 4 point average gain over LoRA on the eight commonsense-reasoning benchmarks with LLaMA-7B, and matching LoRA at half the rank in its ablations), and zero inference overhead, because $m V' / \|V'\|_c$ collapses back into one merged matrix.

(1) Two Degrees Of Freedom Instead Of One: LoRA has a single additive update $W_0 + BA$ ; DoRA splits the same layer into a scalar-per-column scale and a normalized direction, so scale and rotation are optimized by separate tensors.
(2) Exact Identity At Initialization: with $B = 0$ and $m_0 = \|W_0\|_c$ the adapted layer reproduces $W_0$ bit-for-bit, so training starts from the pretrained function with no loss spike.
(3) Gradient Is Projected, Not Just Scaled: differentiating through the column norm inserts the projector $I - V'V'^{\top} / \|V'\|_c^2$ , which removes the radial component of the directional gradient and bounds its effective step.
(4) Learning Pattern Matches Full Fine-Tuning: measured over training, DoRA reproduces the negative $\Delta M$ versus $\Delta D$ slope of full fine-tuning, whereas LoRA’s slope is positive.
(5) Cheapest Where LoRA Hurts Most: the extra magnitude vector costs $k$ parameters per layer, roughly 6% on top of a rank-8 adapter for a $4096 \times 4096$ projection, and pays off most at $r = 4$ to $r = 8$ .
(6) Training Overhead, Not Serving Overhead: the column norm of $W_0 + BA$ must be recomputed each step, which raises training memory; detaching that norm from the backward graph recovers roughly a quarter of the overhead with negligible accuracy change, and inference is unaffected after merging.

Data-flow diagram of a DoRA layer: the frozen pretrained weight is decomposed into a trainable magnitude vector and a frozen normalized direction, the low-rank product BA is added to the direction, the sum is renormalized column-wise, then scaled by the magnitude vector to produce the merged adapted weight

Figure 1: A DoRA layer. Only the magnitude vector $m$ and the LoRA factors $A, B$ receive gradients; the frozen direction is perturbed by $BA$ , renormalized per column, and rescaled by $m$ . Because the final expression is a single matrix, the adapter still merges away at inference.

The stability argument is easier to see from what LoRA cannot express. In LoRA the only knob is the additive term, so any attempt to rescale a column also rotates it, and any attempt to rotate it also changes its norm; the two effects are welded together by a single rank- $r$ product, which is why its measured $(\Delta D, \Delta M)$ points fall on a line with positive slope. Full fine-tuning has $dk$ free parameters and shows the opposite pattern, with subtle magnitude changes accompanying substantial directional change. DoRA recovers that behavior with $k$ extra parameters, and the practical consequences are the ones interviewers care about: lower sensitivity to the learning rate and to the LoRA scaling factor $\alpha / r$ , because a badly scaled low-rank branch is renormalized away instead of blowing up the weight norm, and much flatter accuracy-versus-rank curves, because a rank-4 direction plus a full-rank magnitude still spans useful updates.

Scatter plot of magnitude change versus direction change per layer for full fine-tuning, LoRA, and DoRA with fitted regression lines; full fine-tuning and DoRA have negative slopes while LoRA has a positive slope

Figure 2: Measured magnitude change $\Delta M$ against direction change $\Delta D$ per adapted module. Full fine-tuning and DoRA trace a negative slope, meaning scale and rotation are traded against each other, while LoRA‘s positive slope shows the two are coupled by its single additive term.

Mathematical Formulation:
$W_0 = m_0 \frac{V_0}{\|V_0\|_c}$
$m_0 = \|W_0\|_c$
$W' = m \frac{W_0 + BA}{\|W_0 + BA\|_c}$
$\nabla_{V'}\mathcal{L} = \frac{m}{C}\left(I - \frac{V'V'^{\top}}{C^2}\right)\nabla_{W'}\mathcal{L}$
$\Delta M_t = \frac{1}{k}\sum_{j} |m_t^{j} - m_0^{j}|$
$\Delta D_t = \frac{1}{k}\sum_{j} \left(1 - \cos(V_t^{j}, W_0^{j})\right)$

Where:

$W'$ is the adapted weight actually used in the forward pass, and $W_0 \in \mathbb{R}^{d \times k}$ is the frozen pretrained weight it starts from.
$m \in \mathbb{R}^{1 \times k}$ is the trainable magnitude vector and $V' = W_0 + BA$ the unnormalized direction, with $V_0 = W_0$ and $m_0$ the initialization that makes $W' = W_0$ at step 0.
$B \in \mathbb{R}^{d \times r}$ (initialized to zero) and $A \in \mathbb{R}^{r \times k}$ are the LoRA factors with rank $r \ll \min(d,k)$ .
$j \in \{1,\ldots,k\}$ indexes output columns, $t$ indexes training checkpoints, and $\|\cdot\|_c$ takes the L2 norm of each column independently, returning a $1 \times k$ row vector.
$C = \|V'\|_c$ is that column norm, and $I - V'V'^{\top}/C^2$ is the orthogonal projector that strips the radial part of the gradient, so the directional update is norm-preserving to first order.
$\Delta M_t$ and $\Delta D_t$ are the magnitude and direction change metrics whose correlation is plotted in Figure 2, with $\Delta D_t \in [0,2]$ because it is one minus a cosine similarity.

Property	DoRA	LoRA	Full fine-tuning
Trainable tensors per layer	A, B, and the magnitude vector m	A and B only	The entire weight matrix
Parameters added	r(d + k) + k, about 6% more than LoRA at r = 8, k = 4096	r(d + k)	dk, plus optimizer state for all of it
Magnitude vs direction coupling	Decoupled; negative correlation like full fine-tuning	Coupled through one additive term; positive correlation	Fully free, the reference behavior
Behavior at very low rank	Degrades slowly; competitive at r = 4 to 8	Degrades quickly below r = 8 on harder tasks	Not applicable
Training and serving cost	Extra column-norm recompute each step; merges cleanly, no inference cost	Cheapest to train; merges cleanly, no inference cost	Full optimizer and gradient memory; one checkpoint per task

August 17, 2026

DL0129 SwiGLU, GELU, and ReLU FFNs

Compare SwiGLU, GeLU, and ReLU activation functions in the Feed-Forward Network (FFN) blocks of LLMs, as used in models like Llama 3 and GPT-3.

Answer

All three sit in the same place, namely the two-layer FFN that follows attention in every transformer block, but they differ in smoothness and in whether the nonlinearity is elementwise or multiplicative. ReLU is $\max(0,x)$ : cheapest, produces exact zeros (roughly 90-95% of FFN units are zero on a typical token), but has a kink at the origin and zero gradient for negative inputs, which gives dead units. GELU is $x\Phi(x)$ : smooth, non-monotonic, keeps a small negative response, and was the default through BERT, GPT-2, and GPT-3. SwiGLU is not an elementwise function at all but a gated layer: the up-projection is split into a gate branch passed through SiLU ( $x\sigma(x)$ ) and a value branch left linear, and the two are multiplied elementwise. That third weight matrix is why practitioners shrink the hidden width by $2/3$ to stay parameter- and FLOP-neutral, and the payoff reported by Shazeer’s GLU-variants ablation is a consistent but modest gain, about 0.05 nats of pretraining log-perplexity on T5’s span-corruption objective at matched budget, which is enough that PaLM, Llama, Mistral, and Qwen2 all adopted it. The cost is one extra matmul, one extra saved activation for the backward pass, loss of exploitable sparsity, and a wider activation range that makes low-bit quantization harder.

(1) Two Matrices Versus Three: a dense FFN is $W_{up}$ then $W_{down}$ ; a gated FFN adds $W_g$ , so parameters go from $2 d\, d_{ff}$ to $3 d\, d_{ff}$ at the same width.
(2) The 2/3 Width Rule: to keep the block budget fixed you set $d_{ff} = \frac{8}{3}d$ instead of $4d$ , which is exactly why Llama 2 7B uses 11008 with $d = 4096$ .
(3) Smoothness Changes Optimization: ReLU’s derivative is a step with no gradient for negative inputs, while GELU and SiLU are $C^\infty$ and pass a small negative signal, so units recover instead of dying.
(4) Gating Is A Multiplicative Interaction: the output is a product of two learned projections of the same input, giving a data-dependent, per-channel scaling that no elementwise function can express.
(5) Sparsity Is A ReLU-Only Asset: exact zeros let inference systems skip entire rows of $W_{down}$ ; GELU and SwiGLU produce values that are small but almost never exactly zero.
(6) SwiGLU Costs Memory Traffic And Range: three matmuls plus an extra stored tensor per layer, and the product of two unbounded projections widens the dynamic range that quantization must cover.

Two-panel line chart: left panel plots ReLU, GELU and SiLU over the range minus four to four, showing ReLU flat at zero for negative inputs while GELU and SiLU dip slightly below zero; right panel plots their derivatives, showing ReLU's step from zero to one at the origin versus the smooth GELU and SiLU derivatives that briefly exceed one and go slightly negative

Figure 1: Shape (left) and derivative (right). ReLU’s derivative is a step function with a kink at the origin and no gradient for negative inputs; GELU and SiLU are smooth, non-monotonic, and dip below zero (SiLU’s minimum is $-0.278$ at $x = -1.28$ ), so a unit pushed into the negative region still receives gradient and can come back.

The historical progression is ReLU → GELU → gated GLU variants, and it is worth being honest that only the first step has a clean story. Replacing ReLU with GELU removes the dead-unit problem and gives a slightly better-conditioned loss surface. Replacing an elementwise nonlinearity with a gate is different in kind: the value branch stays linear, and the gate branch modulates it, so the layer computes a second-order interaction of the input with itself. Shazeer’s paper explicitly declines to explain why this helps, attributing the gain to luck rather than theory, and later analyses mostly note that gating gives the block a cheap multiplicative pathway similar in spirit to the gates in an LSTM. What made SwiGLU the production default is that the gain survives at scale, needs no extra hyperparameters, and is free once you apply the $2/3$ width rule. Llama 3 8B keeps the gated form but widens back out to $d_{ff} = 14336$ with $d = 4096$ , a deliberate choice to spend more parameters in the FFN rather than a parameter-neutral swap.

Mathematical Formulation:
$\mathrm{FFN}_{\mathrm{ReLU}}(x) = \max(0, xW_{up})\,W_{down}$
$\mathrm{GELU}(x) = x\,\Phi(x)$
$\mathrm{SiLU}(x) = x\,\sigma(x)$
$\mathrm{SwiGLU}(x) = \mathrm{SiLU}(xW_g) \odot (xW_u)$
$\mathrm{FFN}_{\mathrm{SwiGLU}}(x) = \mathrm{SwiGLU}(x)\,W_{down}$
$d_{ff} = \frac{2}{3}\cdot 4d = \frac{8}{3}d$

Where:

$x \in \mathbb{R}^{d}$ is the post-norm hidden state for one token, and the FFN output has the same dimension $d$ so it can be added back through the residual connection.
$W_{up}, W_g, W_u \in \mathbb{R}^{d \times d_{ff}}$ are up-projections and $W_{down} \in \mathbb{R}^{d_{ff} \times d}$ the down-projection; the dense block uses $W_{up}$ only, the gated block uses both $W_g$ (gate) and $W_u$ (value).
$\Phi$ is the standard normal CDF and $\sigma$ the logistic sigmoid; GELU is often shipped as the tanh approximation $0.5x(1+\tanh(\sqrt{2/\pi}(x+0.044715x^3)))$ , which is why two frameworks can disagree in the last few digits.
$\odot$ is the elementwise product over the $d_{ff}$ hidden channels, the operation that turns a gate into a multiplicative interaction rather than a pointwise map.
$d_{ff}$ is the FFN hidden width; the last line is the parameter-neutral setting, since $3 d \cdot \frac{8}{3}d = 8d^2 = 2 d \cdot 4d$ , and real implementations round the result to a hardware-friendly multiple such as 256.

Architecture diagram with two stacked left-to-right flows: the top flow is a dense FFN with hidden state, one up-projection, an elementwise ReLU or GELU, a down-projection and output, annotated with two d times d_ff parameters; the bottom flow is a gated SwiGLU FFN where the hidden state branches into a SiLU gate path and a linear value path, the two are combined by an elementwise product, then a down-projection and output, annotated with three d times d_ff parameters and the two-thirds width rule

Figure 2: The structural difference is a branch, not a curve. A dense FFN applies one elementwise function between two matrices; a gated FFN splits the up-projection into a SiLU gate path and a linear value path whose elementwise product feeds $W_{down}$ . The third matrix is what forces the $8d/3$ width if the block budget must stay fixed.

Property	ReLU	GELU	SwiGLU
Form	Elementwise, piecewise linear	Elementwise, smooth, non-monotonic	Gated layer: SiLU branch times linear branch
Matrices per FFN	2	2	3, so width drops to 8d/3 to stay budget-neutral
Gradient behaviour	0 or 1; dead units never recover	Smooth, briefly exceeds 1, small negative region	Smooth, plus a gradient path through the gate itself
Exact-zero sparsity	Yes, typically 90-95% of units, exploitable at inference	No; values are small but nonzero	No; near-zero mass only, so row skipping needs a threshold and loses accuracy
Quantization friendliness	Best: bounded below, no multiplicative amplification	Good	Hardest: the product of two unbounded projections widens outliers into W_down
Representative models	Original Transformer, T5 v1.0, OPT	BERT, GPT-2, GPT-3, ViT	PaLM, Llama 2 and Llama 3, Mistral, Qwen2 (Gemma uses the GELU-gated GeGLU)

August 17, 2026

DL0128 Subword Regularization

What is subword regularization, and why can it help machine translation quality?

Answer

Subword regularization is the practice of training a translation model on multiple stochastically sampled segmentations of the same sentence instead of the one deterministic segmentation a tokenizer normally emits. Any subword vocabulary is ambiguous: the word “smaller” can be a single piece, or small plus er, or sm plus all plus er, and a standard tokenizer always returns the same choice, so the model never sees the alternatives. Two mechanisms implement the sampling: unigram-LM sampling (Kudo 2018, shipped in SentencePiece), which draws a path from the segmentation lattice with probability proportional to $P(\mathbf{x})^{\alpha}$ , and BPE-dropout (Provilkov 2020), which keeps the ordinary BPE merge table but skips each merge with probability $p$ during application. The quality gain has three sources: it is free data augmentation over the same parallel corpus, it trains the embeddings of rare pieces that a deterministic segmenter almost never produces, and it makes the encoder robust to the segmentation shifts caused by typos, morphology, and domain change. Reported gains are roughly 1 BLEU on average, shrinking toward noise on large in-domain high-resource benchmarks and reaching 2 BLEU or more on low-resource pairs and out-of-domain test sets. Sampling is a training-time-only device: decoding uses the deterministic segmentation, or optionally averages scores over $k$ sampled segmentations of the source.

(1) Segmentation Is Ambiguous: a fixed vocabulary admits exponentially many segmentations per sentence, and deterministic tokenization silently commits to one of them in every epoch.
(2) Unigram-LM Sampling: SentencePiece samples from the $l$ -best lattice paths with a temperature $\alpha$ , typically $l = 64$ and $\alpha$ near 0.1 to 0.2, where small $\alpha$ is near-uniform and $\alpha \to \infty$ recovers the Viterbi segmentation.
(3) BPE-Dropout: drop each merge operation independently with $p = 0.1$ on both source and target; the vocabulary and merge table are unchanged, so it is a drop-in change to the tokenizer call.
(4) Approximate Marginalization: the model is pushed to score a sentence pair well under any plausible segmentation, which is closer to the true marginal $P(Y \mid X)$ than a single-path proxy.
(5) Robustness Beats In-Domain Accuracy: the largest wins appear on low-resource, morphologically rich, noisy, and out-of-domain data, exactly where the test-time segmentation drifts from the training-time one.
(6) Costs Are Real: average sequence length grows, tokenization moves into the data loader and cannot be cached, and convergence is slower in wall-clock terms for the same number of updates.

Segmentation lattice for the word smaller: boundary nodes under the characters, with three bracketed paths drawn above them, one spanning the whole word as a single piece, one splitting into small plus er, and one splitting into sm plus all plus er, each annotated with its unigram probability and the sampling weight at alpha equal to 0.2

Figure 1: One word, three legal paths through the segmentation lattice. Deterministic tokenization always returns the top path, so the -er morpheme boundary is never shown to the encoder; sampling at $\alpha = 0.2$ returns the top path only about half the time and spends the rest of the probability mass on the finer decompositions.

The reason the same trick helps two different segmenters is that both are inverting a latent variable the corpus never labels. A unigram language model gives an explicit probability to each path, so its lattice can be sampled cleanly and the temperature $\alpha$ gives a continuous dial from “uniform over the lattice” to “deterministic”. BPE has no probability model at all, only an ordered merge list, so BPE-dropout injects randomness procedurally instead: skipping a merge with probability $p$ produces a shorter piece and therefore a longer, more character-like segmentation, and stacking many independent skips over a long word yields a wide distribution of segmentations. Both settings need a knob that is gentle: too much randomness pushes segmentation toward characters, inflating sequence length, wasting attention on re-learning spelling, and destroying the frequency statistics that made the vocabulary useful in the first place.

Mathematical Formulation:
$P(\mathbf{x}) = \prod_{i=1}^{M} p(x_i)$
$P_{\alpha}(\mathbf{x} \mid X) \propto P(\mathbf{x})^{\alpha}$
$\mathcal{L}(\theta) = \mathbb{E}_{\mathbf{x}, \mathbf{y}}\left[\log P(\mathbf{y} \mid \mathbf{x}; \theta)\right]$
$P(Y \mid X) = \sum_{\mathbf{x}} P(\mathbf{x} \mid X)\, P(Y \mid \mathbf{x})$
$\hat{Y} = \arg\max_{Y} \frac{1}{k} \sum_{j=1}^{k} \log P(Y \mid \mathbf{x}_j)$

Where:

$X$ and $Y$ are the raw source and target sentences, while $\mathbf{x} = (x_1, \ldots, x_M)$ and $\mathbf{y}$ are particular segmentations of them into vocabulary pieces.
$p(x_i)$ is the unigram probability of piece $x_i$ , estimated by EM when the SentencePiece unigram vocabulary is built, and $M$ is the number of pieces on that path.
$\alpha$ is the smoothing exponent of the sampling distribution: $\alpha \to 0$ gives a near-uniform draw over the $l$ -best lattice paths and $\alpha \to \infty$ collapses to the single Viterbi path, so deterministic tokenization is the limiting case.
The expectation in $\mathcal{L}(\theta)$ is taken over $\mathbf{x} \sim P_{\alpha}(\cdot \mid X)$ and $\mathbf{y} \sim P_{\alpha}(\cdot \mid Y)$ , approximated by one fresh sample per example per epoch rather than by an inner loop.
$P(Y \mid X)$ is the true segmentation-marginalized translation probability that the training objective approximates; the sum over $\mathbf{x}$ is intractable, which is exactly why sampling is used.
$k$ is the number of source segmentations used in optional n-best decoding, where $k = 1$ with the Viterbi path is the standard, cheapest inference setting.

Two-panel line chart: left panel shows the entropy in bits of the segmentation sampling distribution decreasing from near-maximum to near zero as alpha grows from 0.02 to 3 for a three-path and a four-path lattice; right panel shows the expected number of pieces per word falling toward one over the same alpha range, both panels marking the default alpha near 0.2

Figure 2: The temperature $\alpha$ controls exactly two things that matter. Left: the entropy of the sampling distribution, which is how much augmentation diversity the model receives per epoch. Right: the expected pieces per word, which is the sequence-length tax you pay for that diversity. Beyond roughly $\alpha = 1.5$ both collapse to the deterministic Viterbi behaviour and the method does nothing.

Property	Deterministic BPE / unigram	Unigram-LM sampling	BPE-dropout
Randomness source	None; one path per sentence forever	Draw from the l-best lattice paths of a unigram LM	Independently skip each merge while applying the merge table
Knob and typical value	Vocabulary size only	l = 64, alpha near 0.1 to 0.2	p = 0.1 on both sides, lower for high-resource pairs
Segmenter changes	n/a	Needs a probabilistic unigram vocabulary, not a merge list	None; reuses an existing BPE model unchanged
Main cost	Rare pieces stay under-trained	Lattice sampling per example; longer average sequences	Re-merging per example; length inflation grows fast with p
Inference	Same path as training	Viterbi path, or average log-scores over k sampled sources	Standard BPE with dropout disabled

August 17, 2026

DL0126 Contrastive Learning Pairs

What is contrastive learning, and what are positive and negative pairs?

Answer

Contrastive learning trains an encoder by comparison instead of by prediction of a label: it pulls the embeddings of things that should mean the same thing together and pushes everything else apart. A positive pair is two views of the same underlying content (two augmentations of one photo in SimCLR, an image and its caption in CLIP, two nearby audio segments in CPC), and a negative pair is an anchor paired with content assumed to be different (any other item in the batch). The standard objective, InfoNCE, turns this into a softmax classification problem: given an anchor, identify its single positive among one positive and $K$ negatives, using scaled cosine similarity as the logit. Positives define what the representation should be invariant to, and negatives are what prevents the trivial solution where the encoder maps every input to the same vector, a failure called representational collapse. The whole design problem of a contrastive system is therefore the pair-construction policy: a positive that is too easy teaches nothing, and a negative that is secretly a positive actively teaches the wrong thing.

(1) Positives Encode The Invariance You Want: whatever transformation you apply to build a positive pair is exactly the factor the encoder learns to discard, so color jitter buys color invariance and destroys any task where color is the signal.
(2) Negatives Are The Anti-Collapse Term: without a repulsive term the constant map $f(x) = c$ is a global minimum of the attraction loss, so negatives provide the uniformity pressure that spreads embeddings over the hypersphere.
(3) InfoNCE Is A $K+1$ -Way Classification: one positive logit in the numerator, the positive plus all negatives in the denominator, which is why the loss is bounded below by $\log(K+1)$ under a random encoder.
(4) Temperature Sets Hard-Negative Focus: small $\tau$ (0.05 to 0.1 is typical) concentrates almost all repulsive gradient on the few most similar negatives, while large $\tau$ treats all negatives nearly equally.
(5) Negative Count Is A Systems Problem: SimCLR needs batch sizes of 4096 or more to get enough in-batch negatives, MoCo decouples them with a momentum-encoder queue, and CLIP trained with a 32768 global batch across many GPUs.
(6) False Negatives Are The Main Bias: in-batch negatives assume every other sample is semantically different, which is false on class-imbalanced or deduplicated-poorly data, and the loss then pushes apart items that should be close.

Pairs do not have to come from augmentation. The general recipe is: find a cheap source of known agreement and treat everything else as disagreement. Augmentation gives agreement between two crops of one image; multimodal alignment gives agreement between an image and the alt-text scraped alongside it; temporal or spatial context gives agreement between adjacent frames, patches, or sentences; and labels give agreement between any two samples of the same class, which is what SupCon exploits to allow many positives per anchor. The negatives are almost always just the rest of the batch, because sampling them explicitly is expensive and in-batch negatives come for free with the forward pass already computed.

Two-panel diagram: left panel shows one source image producing two augmented views labeled anchor and positive while two other images in the batch produce negative views; right panel shows the same points on a unit circle with an attraction arrow from anchor to positive and dashed repulsion arrows pushing the negatives away

Figure 1: Pair construction and its geometric effect. Each anchor has exactly one positive (the other view of the same source) and $2N-2$ negatives (all views of the other images in the batch); on the unit hypersphere the loss is a single attractive force toward the positive balanced against repulsive forces from every negative.

Mathematical Formulation:
$s(u,v) = \frac{u^\top v}{\|u\|\,\|v\|}$
$\ell_i = -\log \frac{\exp(s_{i,i^+}/\tau)}{\sum_{k \neq i} \exp(s_{i,k}/\tau)}$
$\mathcal{L} = \frac{1}{2N}\sum_{i=1}^{2N} \ell_i$
$I(u;v) \geq \log K - \mathcal{L}_{\mathrm{InfoNCE}}$

Where:

$s(u,v)$ is the cosine similarity used as the logit, computed on L2-normalized embeddings $u = f(v)$ produced by the encoder (and, in SimCLR, a discarded projection head).
$\ell_i$ is the per-anchor NT-Xent loss and $\mathcal{L}$ its average over all $2N$ views of an $N$ -image batch.
$i$ indexes the anchor view, $i^{+}$ its unique positive, and $k$ ranges over every other view, so the denominator holds 1 positive and $2N-2$ negatives.
$\tau > 0$ is the temperature; the gradient weight on negative $k$ is its softmax probability, so shrinking $\tau$ sharpens that distribution onto the hardest negatives.
$I(u;v)$ is the mutual information between the two views and $K$ the number of negatives, giving the standard InfoNCE lower bound: the bound saturates at $\log K$ , which is one reason large batches help.

Log-x line chart of the cumulative share of repulsive gradient carried by the hardest fraction of negatives, for temperature 0.05, 0.1 and 0.5 over 1024 negatives; the low-temperature curve rises almost vertically showing that the hardest one percent of negatives absorbs most of the gradient

Figure 2: Temperature decides which negatives matter. At $\tau = 0.05$ the hardest 1% of negatives absorbs most of the repulsive gradient, making training an implicit hard-negative miner that is also maximally sensitive to false negatives; at $\tau = 0.5$ the pressure is spread nearly uniformly and the embedding space stays smoother but less discriminative.

Aspect	Augmentation (SimCLR, MoCo)	Multimodal (CLIP, ALIGN)	Supervised (SupCon)
Anchor	One augmented view of an image	An image embedding	A labeled sample
Positive	A second augmentation of the same image (exactly one)	The paired caption text (exactly one, symmetric loss both directions)	Every other sample sharing the label (many per anchor)
Negatives	All 2N-2 other views, or a momentum queue of 65k stale keys	All other captions in the global batch (32k in CLIP)	Only samples with a different label, so false negatives vanish
Invariance learned	To the augmentation family you chose (crop, color, blur)	To modality and phrasing, giving a shared image-text space	To everything within a class, which can over-collapse fine detail
Main failure mode	Augmentation removes task-relevant signal; shortcut solutions from crop statistics	Noisy or generic captions produce weak positives; batch size dominates cost	Needs labels, so it is not self-supervised and inherits label noise

August 17, 2026

DL0121 Open-Loop vs Closed-Loop Validation

Explain why open-loop validation metrics (e.g., MSE on offline trajectories) often fail to correlate with closed-loop success rates in physical robotics or driving environments.

Answer

Open-loop evaluation replays a logged trajectory and asks the policy to reproduce the expert action at states the expert visited, with ground-truth history fed back in at every step, so each prediction error is scored and then discarded. Closed-loop execution feeds every error into the next observation, so the policy’s own state distribution drifts off the demonstration manifold and errors compound: the classic behavior-cloning result turns a per-step error $\epsilon$ into an excess cost of up to $T^2 \epsilon$ over a $T$ -step episode, a bound that is already vacuous for a 200-step driving episode at $\epsilon = 0.01$ . Two further effects break the correlation before compounding even starts. MSE is mean-seeking: its minimizer is the conditional mean $\mathbb{E}[a \mid s]$ , and averaging the two valid modes of “swerve left or swerve right around the obstacle” produces a straight-line action that is optimal under the metric and a collision in the world. And metric mass is not risk mass: most logged frames are trivial lane-following, so average displacement error is dominated by how well a model extrapolates its own velocity, which is why an MLP fed only ego status with no perception input at all scored competitively on nuScenes open-loop L2 while being worthless as a planner.

(1) Covariate Shift: open loop measures the loss under $d_{\pi^*}$ , the expert’s state distribution, while success is determined under $d_{\pi}$ , the policy’s own induced distribution; the two diverge as soon as the policy acts.
(2) Compounding Error: teacher forcing caps the deviation at one step’s worth of error, whereas rollout integrates it, giving the quadratic-in-horizon gap that a single-step regression number cannot express.
(3) Mode Averaging: a squared-error objective on a multimodal action distribution returns an interpolation of the modes, which is frequently the one infeasible action available.
(4) Long-Tail Mismatch: safety outcomes are decided by a fraction of a percent of frames (cut-ins, occluded pedestrians, contact-rich grasps), and those frames contribute almost nothing to a dataset-averaged MSE.
(5) Non-Reactive Logs: logged agents never yield, brake, or negotiate, so open loop cannot score any behavior whose correctness depends on how the world responds to the ego.
(6) Shortcut Exploitation: ground-truth history leaks the answer; extrapolating the logged ego velocity minimizes displacement error without any scene understanding, and that shortcut vanishes the moment the policy controls its own history.

Two panels: left shows a logged expert trajectory with short prediction-error arrows at each sampled state and the state reset to the log after every step; right shows the same per-step error accumulating into a rollout that curves away from the dashed expert reference into off-distribution states

Figure 1: Open-loop scoring resets the policy onto the logged state after every prediction, so a bounded per-step error stays bounded; closed-loop rollout feeds each error into the next observation, and the visited states leave the training distribution where the policy has no guarantees at all.

The theory is unusually clean here. Ross and Bagnell showed that supervised imitation with per-step loss $\epsilon$ under the expert distribution admits an excess cost that grows as $T^2 \epsilon$ , because a mistake at step $t$ can put the agent in a state where it makes mistakes for all remaining $T - t$ steps; on-policy correction such as DAgger restores the linear $T \epsilon$ rate precisely by collecting labels on $d_{\pi}$ . The practical consequence is that ranking two policies by offline $\epsilon$ tells you almost nothing about their closed-loop ordering, since the multiplier between them differs by a factor of $T$ and depends on recovery behavior that the offline data never contains. Codevilla and colleagues measured this directly on vision-based driving models and found offline prediction error to be a weak predictor of on-road driving quality, and the nuPlan and NAVSIM benchmarks were built specifically because open-loop leaderboard position stopped tracking closed-loop driving score.

Mathematical Formulation:
$\epsilon = \mathbb{E}_{s \sim d_{\pi^*}}\left[\ell(s, \pi(s))\right]$
$J(\pi) - J(\pi^*) \leq T^2 \epsilon$
$\lVert d_{\pi} - d_{\pi^*} \rVert_1 \leq 2 T \epsilon$
$\pi_{\mathrm{mse}}(s) = \mathbb{E}[a \mid s]$
$\pi_{\mathrm{mse}}(s) = 0.5 a_L + 0.5 a_R$

Where:

$\epsilon$ is the offline per-step error that an open-loop MSE actually reports, and $\ell$ is the per-state surrogate loss (squared action error, displacement error).
$\pi$ is the learned policy, $\pi^*$ the expert, and $J$ the closed-loop episode cost, with per-step cost bounded in $[0, 1]$ .
$d_{\pi^*}$ and $d_{\pi}$ are the state distributions induced by the expert and by the policy; open loop samples the first, deployment samples the second.
$T$ is the episode horizon in control steps, the multiplier that an offline metric never sees; the bound becomes vacuous once $T^2 \epsilon \geq T$ .
$a_L$ and $a_R$ are two equally valid expert modes (pass left, pass right) at the same state $s$ ; their MSE-optimal average drives straight into the obstacle.
Required condition for the bounds: the offline data is drawn from $d_{\pi^*}$ with no on-policy correction, which is exactly the assumption behind pure behavior cloning.

Log-scale chart of excess closed-loop cost against episode length T for a fixed per-step error of 0.01: the offline metric is a flat line at 0.01, on-policy correction grows linearly as T times epsilon, and behavior cloning grows quadratically as T squared times epsilon

Figure 2: A single offline number $\epsilon$ is consistent with wildly different closed-loop outcomes, because the horizon $T$ is the multiplier and it is invisible to the metric; on-policy data collection is what changes the exponent from 2 to 1.

Property	Open-loop replay	Closed loop, log-replay agents	Closed loop, reactive agents or hardware
States visited	Expert distribution only	Policy distribution, but in a world frozen to the log	Policy distribution with a world that responds to it
Error feedback	None; state is reset each step	Ego error compounds; other agents do not react	Full two-way feedback including other agents
Typical metric	Action MSE, ADE/FDE, L2 at 1/2/3 s	Route completion, collision rate, comfort sub-scores	Task success rate, interventions per kilometer or per trial
Cost per evaluation	One forward pass per frame, fully parallel, seconds	Sequential rollout, hundreds of scenarios, minutes to hours	Highest; wall-clock hardware time or heavy sim agents
Main blind spot	Recovery, mode collapse to the mean, ego-status shortcuts	False collisions from behind, merges and nudges scored unfairly	Sim-to-real gap in sensing, or low statistical power on real hardware

August 17, 2026

DL0118 VLA vs VLM

What is a VLA model and how does it differ from a VLM?

Answer

A VLA (Vision-Language-Action) model is a policy that maps camera images plus a natural-language instruction directly to robot actions, typically a short sequence of end-effector or joint deltas plus a gripper command, emitted at a fixed control rate. A VLM (Vision-Language Model) maps the same image-plus-text input to text tokens. Architecturally the two are close relatives: almost every modern VLA (RT-2, OpenVLA, $\pi_0$ ) starts from a pretrained VLM backbone and is fine-tuned on teleoperated demonstration trajectories, either by discretizing each action dimension into vocabulary tokens or by attaching a continuous action expert head. The real difference is not the encoder, it is everything downstream of it: the output lives in a continuous, embodiment-specific action space, the model runs inside a closed feedback loop where its own outputs change the next observation, and errors therefore compound over the rollout instead of being independent per query. That single fact drives the different data (robot demos, not web image-text pairs), the different latency budget (tens of milliseconds, not seconds), and the different metric (physical task success rate, not benchmark accuracy).

(1) Output Space: a VLM produces a distribution over a discrete text vocabulary; a VLA produces a vector in $\mathbb{R}^{d}$ per timestep, usually predicted as an action chunk covering the next $H$ control steps.
(2) Closed Loop Versus Open Loop: a VLA’s action changes the world and therefore its own next input, so covariate shift makes behavior-cloning error grow roughly with the square of the horizon; a VLM answer is scored once and never fed back through a robot.
(3) Training Data: web-scale image-caption and VQA corpora for the VLM, versus expensive teleoperated trajectories (Open X-Embodiment aggregates roughly one million episodes) with synchronized proprioception for the VLA.
(4) Action Representation Is A Design Choice: RT-2 and OpenVLA quantize each dimension into 256 bins and reuse rarely-used text tokens; $\pi_0$ instead trains a flow-matching action expert that emits continuous chunks at 50 Hz.
(5) Latency Is A Correctness Constraint: a 2 s VLM response is fine, but a controller starved of fresh actions produces jerky or unsafe motion, so chunk horizon and inference time must be budgeted together.
(6) Embodiment Coupling: VLM weights transfer across any image; VLA action heads are tied to a specific DoF count, camera mount, and control convention, which is why cross-embodiment training is an active research problem.

Diagram showing a shared ViT plus LLM backbone taking an RGB observation and a language instruction, branching into a text decoder head producing answer tokens for a VLM and an action expert head producing a 50 by 7 action chunk executed by a robot at 50 Hz, with a feedback arrow returning the new observation to the encoder

Figure 1: Both models share the same perception stack; the VLA replaces or augments the text head with an action head and runs inside the loop observe → predict chunk → execute → observe, so its own predictions determine the next input distribution.

The token-budget arithmetic explains why action chunking is universal. A 7-DoF arm predicted one step at a time at 50 Hz would need a full autoregressive forward pass every 20 ms, which no 3B-parameter backbone can sustain. Predicting a chunk of $H = 50$ actions amortizes one forward pass across a full second of motion, at the price of running open loop within the chunk. Shorter chunks mean tighter feedback and better disturbance rejection but more compute and more jitter at chunk boundaries; longer chunks are smoother but blind to anything that happens mid-chunk. Naive autoregressive decoding of 350 discrete action tokens is also slow, which is exactly the bottleneck that continuous action experts and frequency-domain tokenizers were built to remove.

Mathematical Formulation:
$p_\theta(y_{1:T} \mid I, \ell) = \prod_{t=1}^{T} p_\theta(y_t \mid y_{1:t-1}, I, \ell)$
$a_{t:t+H-1} \sim \pi_\theta(\cdot \mid o_t, s_t, \ell)$
$k_j = \mathrm{round}\left(\frac{a_j - a_{\min}}{a_{\max} - a_{\min}}(B-1)\right)$
$N_{tok} = H \cdot d = 50 \cdot 7 = 350$
$t_{infer} \leq H / f_{ctrl}$
$50 / 50\ \text{Hz} = 1\ \text{s}$
$J(\pi_\theta) - J(\pi^{*}) = O(\epsilon T^{2})$

Where:

$y_{1:T}$ are the text tokens a VLM emits for image $I$ and instruction $\ell$ ; the factorization is the only thing the VLA keeps unchanged.
$a_{t:t+H-1} \in \mathbb{R}^{H \times d}$ is the action chunk, $o_t$ the current camera observation, and $s_t$ the proprioceptive state that a VLM never receives.
$d$ is the action dimension (7 for a 6-DoF pose delta plus gripper) and $H$ the chunk horizon in control steps.
$k_j \in \{0, \ldots, B-1\}$ is the discrete bin for dimension $j$ , with $B = 256$ and $[a_{\min}, a_{\max}]$ set from per-dimension training quantiles so outliers do not collapse the resolution.
$f_{ctrl}$ is the control frequency and $t_{infer}$ the policy latency; the required deployment condition is that a new chunk arrives before the previous one is exhausted.
$\epsilon$ is the per-step imitation error, $T$ the rollout length, and $J$ the task cost; the quadratic bound is the classical behavior-cloning compounding result.

Log-scale chart of accumulated error against rollout length, comparing a linear curve for independent per-query error and a quadratic curve for closed-loop behavior cloning, with the quadratic curve 400 times higher at 400 steps

Figure 2: A VLM’s mistakes are independent per query and accumulate linearly; a VLA’s mistakes move the robot off the demonstration distribution, so the worst-case cost grows as $O(\epsilon T^{2})$ and a 1% per-step error is fatal over a 400-step manipulation.

Property	VLA (Vision-Language-Action)	VLM (Vision-Language Model)
Output	Continuous action chunk, typically 50 steps by 7 dimensions, as bin tokens or a flow-matching head	Text tokens from a fixed vocabulary of roughly 32k to 256k entries
Extra inputs	Proprioception, gripper state, often multiple synchronized camera views	Images and text only
Training data	Teleoperated demonstrations (Open X-Embodiment scale is about 1M episodes), usually co-trained with web data to keep semantics	Billions of web image-text pairs plus instruction tuning
Latency budget	Chunk must arrive before the previous one runs out, so tens to a few hundred milliseconds	Seconds; streaming hides most of it from the user
Evaluation	Physical or simulated rollout success rate over many trials, with high variance and slow iteration	Static benchmark accuracy or preference scores, reproducible offline
Dominant failure mode	Compounding covariate shift, unrecoverable states, embodiment mismatch, control jitter	Hallucination and grounding errors, recoverable by re-prompting

August 17, 2026

DL0117 3D Attention vs Frame Pooling

Compare 3D convolutional and space-time attention modules (e.g., TimeSformer) against frame-level pooling for temporal token aggregation in video encoders. When is each the right choice?

Answer

Both families turn $T$ frames of $N$ patch tokens into one clip representation, and they differ only in where temporal information is allowed to mix. Frame-level pooling runs a purely spatial encoder per frame and then averages or attention-pools the $T$ frame vectors (frames → per-frame ViT → pool → head), so no token ever sees another frame and the aggregation is permutation invariant over time. 3D modules mix earlier: a 3D convolution gives each token a local spatiotemporal receptive field of size $k_t$ , joint space-time attention lets all $NT$ tokens attend to each other, and TimeSformer’s divided space-time attention factorizes that into a temporal MSA over the same spatial position across frames followed by a spatial MSA inside each frame. The cost separation is the first thing to state in an interview: joint attention is $O(N^2T^2)$ in the pair count while divided is $O(N^2T + NT^2)$ , a ratio of $NT/(N+T)$ that reaches roughly 64x at 96 frames. The accuracy separation depends almost entirely on whether the label actually depends on frame order: on scene-biased Kinetics-400 dropping temporal attention costs about a point, while on Something-Something V2 the same ablation costs roughly 23 points.

(1) Pooling Is Order-Blind By Construction: mean or max pooling over frame embeddings is symmetric, so “opening a door” and “closing a door” produce the identical clip vector no matter how good the image backbone is.
(2) 3D Convolution Buys Locality Cheaply: cost is linear in $T$ , but the temporal receptive field grows only $k_t - 1$ frames per layer, so long-range order needs depth or a slow/fast dual pathway.
(3) Joint Attention Is Global But Quadratic: with ViT-B at 8 frames the token sequence is $196 \times 8 = 1568$ , and every added frame inflates the attention matrix quadratically.
(4) Factorization Is The Practical Default: divided space-time attention beat both space-only and joint attention in the TimeSformer ablations while being far cheaper than joint, which is why factorized variants dominate video ViTs.
(5) Benchmark Bias Decides The Verdict: appearance-biased datasets reward a strong image backbone, and temporally-ordered datasets punish any aggregator that discards order.
(6) Pooling Keeps System Properties Attention Destroys: per-frame embeddings can be cached, indexed, and streamed independently, which is why large-scale video retrieval still ships CLIP-style pooled encoders.

Three grids of tokens arranged as spatial patches by frames; in the first panel a query token connects only to tokens in its own frame, in the second it connects to its own frame plus the same spatial position in all frames, and in the third it connects to every token in the clip

Figure 1: The three aggregation schemes differ only in the attended set of a query token. Space-only attention plus pooling never crosses a frame boundary, divided space-time adds a one-dimensional temporal pass over the same spatial position, and joint attention connects all $NT$ tokens at quadratic cost.

A subtlety that separates mid from senior candidates is that divided attention is not simply “cheaper joint attention”. Its temporal MSA only compares a patch with the same spatial coordinate in other frames, so a fast-moving object that shifts several patches between frames is matched indirectly, through the spatial MSA that follows. That works because the two passes alternate at every block, but it is also why divided attention benefits from higher frame rates and larger patch strides, and why 3D convolutions with a spatial kernel remain competitive on motion-heavy, short-horizon tasks. Practically, inflating an image-pretrained ViT into a divided model requires zero-initializing the temporal projection so the network starts as an exact image model and the pretrained features survive the first epochs.

Mathematical Formulation:
$z = \frac{1}{T}\sum_{t=1}^{T} f(x_t)$
$C_{pool} = O(N^2 T D)$
$C_{3D} = O(k_t k_s^2 N T D^2)$
$C_{joint} = O(N^2 T^2 D)$
$C_{div} = O(N^2 T D + N T^2 D)$
$\frac{C_{joint}}{C_{div}} = \frac{NT}{N + T}$
$\frac{196 \cdot 96}{196 + 96} \approx 64$

Where:

$z$ is the clip embedding and $f$ the frozen or fine-tuned per-frame encoder applied to frame $x_t$ ; because the sum is symmetric, $z$ is unchanged by any permutation of the frames.
$t \in \{1, \ldots, T\}$ indexes frames, $N$ is the number of patch tokens per frame ( $196$ for ViT-B at $224 \times 224$ with patch 16), and $D$ is the model width.
$C_{pool}$ , $C_{joint}$ , and $C_{div}$ are the attention costs of space-only, joint space-time, and divided space-time blocks; all three exclude the identical per-token MLP term.
$k_t$ and $k_s$ are the temporal and spatial kernel sizes of a 3D convolution, whose cost is linear in $T$ but whose temporal receptive field after $L$ layers is only about $L(k_t - 1) + 1$ frames.
The ratio $NT/(N+T)$ holds whenever $T$ is at least 2; it grows toward $N$ as $T$ grows, so the saving is bounded above by the token count per frame.

Grouped bar chart of top-1 accuracy for space-only, joint space-time, and divided space-time attention on Kinetics-400 and Something-Something V2, showing a small gap on Kinetics and a very large gap on Something-Something

Figure 2: TimeSformer ViT-B ablations at 8 frames. Removing temporal mixing costs only 1.1 points on Kinetics-400, whose classes are largely identifiable from a single frame, but 22.9 points on Something-Something V2, where the label is defined by the direction of motion.

Property	Frame-level pooling	3D convolution (I3D, X3D, SlowFast)	Divided space-time attention
Temporal receptive field	None inside the encoder; one symmetric average at the end	Local, grows by k_t minus 1 frames per layer	Global over all T frames from the first block
Cost scaling in T	Linear, and trivially parallel across frames	Linear, with a constant factor of k_t	Linear plus a small quadratic term N T squared
Sensitive to frame order	No; shuffled clips give identical embeddings	Yes, within the local window	Yes, with temporal position embeddings
Image pretraining transfer	Perfect; the backbone is unchanged	Via kernel inflation and rescaling	Strong if the temporal projection is zero-initialized
Per-frame embedding caching	Yes; embeddings are reusable across clips and queries	No; features depend on neighboring frames	No; every block mixes across the whole clip
Typical failure	Collapses on reversible actions and counting tasks	Misses long-horizon structure without deep stacks	Weak on fast motion that leaves the shared patch column
Where it wins	Retrieval, tagging, zero-shot with image-text encoders	Short motion-heavy clips on constrained hardware	Order-sensitive recognition over dozens of frames

August 17, 2026

DL0115 Visual Chain-of-Thought

Explain how Chain-of-Thought (CoT) prompting with visual cropping (Visual CoT) enhances multi-step spatial reasoning in complex visual problem-solving.

Answer

Text-only CoT helps a vision-language model because it lets the model spend more tokens on intermediate reasoning, but it cannot help with detail the vision encoder never tokenized. A CLIP ViT-L/14 at $336 \times 336$ produces a fixed $24 \times 24$ grid of 576 patch tokens, so a $1500 \times 1500$ photograph is downsampled by $4.5$ x before a single transformer layer runs and a 60-pixel license plate lands inside less than one patch. Visual CoT closes that gap by making localization an explicit step in the reasoning chain: the model first emits a bounding box for the region the sub-question depends on, that crop is re-encoded at the encoder’s native resolution, the new visual tokens are appended to the context alongside the original global tokens, and only then does the model answer. The chain becomes localize → crop → re-encode → reason, and it can iterate, which is exactly what multi-step spatial questions need: “what does the sign to the left of the red truck say?” decomposes into finding the truck, resolving left of in the global view, then reading text at a magnification where the glyphs actually exist. The VisCoT work trains this behavior with 438k question-answer pairs carrying intermediate bounding-box annotations, and reports its largest gains on text-rich and small-object splits, precisely where fixed-resolution encoding destroys the evidence.

(1) Perception Bottleneck, Not Reasoning Bottleneck: on high-resolution inputs the failure is usually that the target occupies a fraction of a patch, so extra reasoning tokens cannot recover it.
(2) Crop As A Resolution Amplifier: re-encoding a $w \times h$ crop at $S \times S$ raises the sampling density on that region by $WH/(wh)$ without changing the encoder.
(3) Global Tokens Must Be Retained: a crop destroys the frame of reference, so relational predicates such as “left of” or “third from the top” require keeping the original 576 tokens in context.
(4) Errors Compound Multiplicatively: the answer is correct only if localization and reading are both correct, so a 0.9 localization accuracy caps a two-hop chain at 0.81.
(5) It Needs Grounding Supervision: unlike text CoT, the intermediate step is a box, so it comes from bbox-annotated data, a detector tool, or a localization reward rather than from prompting alone.
(6) Sequential Cost: each hop is an extra forward pass over the encoder plus a longer prefill, which trades latency for accuracy in a way tiling does not.

Left-to-right pipeline: a 1500 by 1500 image and question are globally encoded into 576 tokens, the model predicts a bounding box, the crop is re-encoded at 336 by 336 into another 576 tokens, and the model reasons over 1152 tokens to produce an answer, with a loop back to the localization step for multi-step questions and a dashed path showing the global tokens retained in context

Figure 1: Localization becomes an explicit reasoning step: the crop is re-encoded at native resolution and concatenated with the global tokens, so each hop of a multi-step spatial question is answered at the magnification it needs while the frame of reference survives.

The cost profile is what makes this a design decision rather than a free win. Visual CoT buys resolution with sequential compute: two or more encoder passes and a growing prefill, so at fixed batch size it roughly doubles time-to-first-token, and every hop is a place where a bad box silently poisons everything downstream. Tiling approaches such as AnyRes buy resolution with parallel compute instead, encoding a fixed grid of sub-images in one pass, which is friendlier to serving but pushes the visual context to 2,880 tokens or more and makes the language model’s $O(N^2)$ attention the new bottleneck. The interesting property of the crop-based chain is that it is adaptive: it spends its extra tokens only on the region the question actually depends on, which is why it scales to $4$ K screenshots and document scans where uniform tiling would need dozens of tiles.

Mathematical Formulation:
$N = (S/p)^2$
$N = (336/14)^2 = 576$
$\rho = \frac{W H}{w h}$
$\rho = \frac{1500 \cdot 1500}{150 \cdot 150} = 100$
$p_{\mathrm{task}} = p_{\mathrm{loc}} \cdot p_{\mathrm{read}}$
$p_{\mathrm{task}} = 0.90 \cdot 0.90 = 0.81$

Where:

$N$ is the number of visual tokens per encoder pass, $S$ the encoder’s square input side, and $p$ the patch size; $S = 336, p = 14$ gives the familiar 576 tokens.
$W \times H$ are the original image dimensions and $w \times h$ the predicted crop, both in pixels, with the crop constrained to lie inside the image.
$\rho$ is the areal sampling gain on the cropped region, so the linear magnification is $\sqrt{\rho} = 10$ for the numbers above.
$p_{\mathrm{loc}}$ is the probability the emitted box actually contains the target and $p_{\mathrm{read}}$ the probability of reading it correctly once magnified; $p_{\mathrm{task}}$ is end-to-end accuracy for a two-hop chain.
For a $k$ -hop chain the same argument gives $p_{\mathrm{task}} = \prod_{i=1}^{k} p_i$ , and total visual tokens grow as $(k+1)N$ because the global view is kept.

Log-scale bar chart of how many ViT patches cover a 60 by 60 pixel target inside a 1500 by 1500 image: under one patch for a single 336 by 336 encode with 576 tokens, about four patches for AnyRes 2 by 2 tiling with 2880 tokens, and about 92 patches for a re-encoded 150 by 150 Visual CoT crop with 1152 tokens

Figure 2: A 60-pixel target inside a $1500 \times 1500$ image gets under one patch from a single low-resolution pass and about four from $2 \times 2$ tiling, but roughly 92 patches after a crop is re-encoded, at half the token cost of tiling.

Property	Visual CoT (crop and re-encode)	Fixed single low-res pass	AnyRes tiling
How fine detail is obtained	Model predicts a bbox, that crop is re-encoded at native encoder resolution	Not obtained; anything below one patch is destroyed at encode time	Image is split into a fixed grid of tiles, each encoded at full resolution
Visual tokens at 1500×1500	1,152 for one hop (576 global + 576 crop)	576	2,880 for four tiles plus a global view
Encoder passes per answer	Two or more, strictly sequential	One	One, tiles batched in parallel
Patches on a 60 px target	About 92	Under 1	About 4
Supervision required	Bbox-annotated chains, a detector tool, or a localization reward	None beyond standard instruction tuning	None; purely an input-packing change
Main failure mode	A wrong box yields a confident answer about the wrong region	Small text, gauges, and distant objects are simply invisible	Quadratic attention growth, and tile borders cut objects in half

August 17, 2026

DL0114 Chain-of-Thought Prompting

What is Chain-of-Thought prompting and why does it improve reasoning?

Answer

Chain-of-Thought (CoT) prompting makes a language model emit intermediate reasoning steps before its final answer, either through few-shot exemplars whose demonstrations contain worked solutions (Wei et al., 2022) or through a zero-shot trigger phrase such as “Let’s think step by step” (Kojima et al., 2022). The gains are large on multi-step problems: on GSM8K, 8-shot CoT lifted PaLM 540B from 17.9% to 56.9%, and the same trigger took GPT-3 (text-davinci-002) from 10.4% to 40.7% with no weight updates. Two mechanisms explain why. The first is serial compute: a transformer with $L$ layers can apply only $O(L)$ sequential operations before it must commit to the next token, so a task needing more sequential steps than the depth provides is simply not representable in one forward pass; every generated token reruns the whole stack, turning the context window into an external scratchpad that raises effective serial depth from $L$ to $L(T+1)$ . Theory backs this up: constant-depth transformers with a polynomially long chain can simulate any polynomial-time computation, while the same model restricted to direct answers cannot. The second mechanism is factorized conditioning: instead of sampling one high-entropy jump from question to answer, the model decomposes the problem into a product of low-entropy conditionals, each conditioned on the already-written steps, which keeps it on the distribution of solution traces seen in pretraining. CoT is not universally free money: it is emergent with scale, and later meta-analysis shows the gains concentrate on math, symbolic, and logical tasks rather than on knowledge or commonsense retrieval.

(1) Two Elicitation Modes: few-shot CoT shows worked examples and controls format tightly, while zero-shot CoT appends one trigger sentence and costs almost no prompt tokens.
(2) Depth Becomes Time: each scratchpad token buys another full pass through the network, so the model trades latency and tokens for sequential computation it structurally lacked.
(3) Easier Conditionals: the chain rewrites one hard prediction as many easy ones, and each step reads all earlier steps as ordinary context.
(4) Emergent, Not Universal: below roughly 10B parameters, CoT often produces fluent but invalid chains and can score below direct prompting.
(5) Chains Are Not Guaranteed Faithful: models shift answers when a prompt carries a bias cue while the written rationale never mentions it, so a chain is an output artifact, not an audit trail.
(6) Cheap Ensembling On Top: because chains are stochastic, sampling $k$ of them and majority-voting the final answers (self-consistency) pushed PaLM 540B on GSM8K to 74.4%.

Two-panel diagram: direct prompting sends the question through one L-layer forward pass to the answer, while chain-of-thought generates T scratchpad tokens, each of which reruns the full L-layer stack before the answer token

Figure 1: Direct prompting caps sequential computation at the network depth $L$ ; CoT reuses the same weights once per generated token, giving $L(T+1)$ sequential steps and letting step $t$ read every earlier step as context.

The costs are as concrete as the benefits. Chains are typically 100 to 400 decode tokens on grade-school math, so a CoT query can cost an order of magnitude more decode passes and time-to-last-token than a direct answer, which matters for anything user-facing. Errors also propagate: the decoder never backtracks, so a slip in the first arithmetic step is carried through every later step, and accuracy behaves roughly like the product of per-step accuracies. Few-shot CoT is additionally prompt-sensitive, since exemplar choice, ordering, and even the formatting of the equations move accuracy by several points. These properties are exactly why the field moved from prompting toward verifying or training the chain rather than just requesting it.

Mathematical Formulation:
$p(a \mid q) = \sum_{z} p(a \mid q, z)\, p(z \mid q)$
$p(z \mid q) = \prod_{t=1}^{T} p(z_t \mid q, z_{1:t-1})$
$D_{\text{direct}} = L$
$D_{\text{CoT}} = L \cdot (T + 1)$
$\hat{a} = \arg\max_{a} \sum_{i=1}^{k} \mathbf{1}[a_i = a]$

Where:

$q$ is the prompt (question plus any exemplars), $a$ the final answer span, and $z = z_{1:T}$ the generated rationale, a latent variable the model writes into its own context.
$t \in \{1, \ldots, T\}$ indexes chain tokens; $T$ is the chain length and $L$ the number of transformer layers.
$D$ counts the sequential layer applications available before the answer is committed, which is the resource CoT actually adds.
$k$ is the number of sampled chains in self-consistency, $a_i$ the answer extracted from sample $i$ , and $\mathbf{1}[\cdot]$ the indicator used for majority voting.
Required condition: the sum over $z$ is intractable, so plain CoT approximates it with a single greedy or sampled $\hat{z}$ , and self-consistency approximates it with $k$ Monte Carlo samples at temperature $T_s > 0$ .

Grouped bar chart of GSM8K solve rate for PaLM at 8B, 62B and 540B parameters under standard prompting and 8-shot chain-of-thought, with the CoT advantage appearing only at 62B and growing at 540B

Figure 2: CoT is an emergent ability: at 8B parameters the chains are fluent but wrong and buy nothing, while at 540B they roughly triple the GSM8K solve rate. Values are approximate figures reported for PaLM by Wei et al. (2022) and Wang et al. (2023).

Property	Zero-shot CoT	Few-shot CoT	CoT + self-consistency
How it is elicited	One trigger sentence appended to the question	A handful of exemplars that show the full reasoning trace	Same CoT prompt sampled k times, majority vote on the extracted answer
Extra prompt tokens	Roughly 10	Hundreds to a few thousand, and it consumes context	Unchanged prompt, but k independent decodes
Reported GSM8K result	10.4% to 40.7% on text-davinci-002	17.9% to 56.9% on PaLM 540B, 8-shot	74.4% on PaLM 540B with 40 sampled chains
Main weakness	Format drift: the model may skip steps or answer immediately	Sensitive to exemplar choice, order, and formatting	k times the decode cost; needs a comparable, extractable final answer
Best fit	Quick baseline and open-ended chat traffic	A fixed task where output format must be stable	High-value math or code queries where accuracy beats latency

August 17, 2026