Author: admin

DL0200 BERT GPU Underutilization

What can cause a BERT-based model to struggle with GPU underutilization, how do you detect CPU-side preprocessing bottlenecks, and what optimizations (pinned memory, ONNX export, TensorRT, INT8 quantization) would you recommend?

Answer

A BERT forward pass at batch 32 and sequence length 128 is a few milliseconds of dense GEMM work, so the GPU is almost never the slow part of a badly performing BERT service. Underutilization comes from three separable causes: the host-side pipeline (Python tokenization, collation, and synchronous host-to-device copies that block the training or serving loop), wasted GPU work (padding every request to max_length, batch size 1 traffic, or fragmented small kernels whose launch overhead exceeds their compute), and framework overhead (eager-mode PyTorch dispatching hundreds of tiny kernels per layer instead of fused ones). Detection is a matter of separating those causes rather than staring at the single nvidia-smi utilization number, which reports only whether any kernel was resident during a sampling window and happily shows 90% while the SMs are mostly idle. The fix ladder is ordered by cost: first hide the host work with fast tokenizers, multiple dataloader workers, pinned memory plus non_blocking copies and prefetch; then remove padding waste with length bucketing and dynamic batching; then export to ONNX for graph fusion and constant folding; then build a TensorRT engine in FP16; and only last apply INT8 post-training quantization, which is the only step in the list that can change your accuracy.

(1) Host-Bound Pipeline: a Python BertTokenizer plus per-batch collation can cost 15-25 ms while the forward pass costs 6 ms, so a serial loop pins utilization near 25% no matter how fast the GPU is.
(2) Synchronous Copies: a transfer from pageable host memory cannot be overlapped with compute, because the driver must stage it through an internal pinned buffer; pinned memory plus non_blocking=True is what makes DMA overlap legal.
(3) Padding Waste: attention cost scales as $O(L^2)$ , so padding a batch whose mean length is 27 tokens out to 128 burns roughly 79% of the tokens on [PAD], which looks like high utilization and low throughput.
(4) Kernel Launch Overhead: eager BERT-base issues on the order of a thousand kernels per forward pass; at batch 1 the launch and layout overhead, not the math, sets latency.
(5) Detection By Elimination: loop over one cached pre-tokenized tensor. If throughput jumps, the bottleneck was host-side; if it does not, profile kernels with Nsight Systems and look at gaps between them.
(6) Optimize In Cost Order: dataloader and padding fixes are free and safe, ONNX and TensorRT FP16 are numerically near-lossless, and INT8 is the only step that requires a calibration set and an accuracy regression gate.

Two stacked timelines in milliseconds. The upper pair shows a serial dataloader where the CPU tokenizes for 18 ms while the GPU row is idle, then the GPU computes for 6 ms while the CPU waits, repeating so the GPU is idle three quarters of the time. The lower pair shows three CPU worker lanes tokenizing in staggered 18 ms windows offset by 6 ms each, after an 18 ms prefetch warm-up, feeding a GPU row of back-to-back 6 ms forward passes with no gaps.

Figure 1: The same model and the same GPU, two pipelines. In the serial loop the GPU waits for tokenization and the utilization ceiling is $6/(18+6) = 25\%$ . With three prefetching workers writing into pinned buffers, host work is fully hidden behind compute after an 18 ms warm-up and the GPU runs back to back. No kernel was made faster here; only the schedule changed.

How To Detect A Host-Side Bottleneck: start by distrusting the utilization percentage. nvidia-smi samples whether at least one kernel was resident, not how many SMs were busy, so query DCGM fields such as SM activity and SM occupancy, or run nvidia-smi dmon and watch the memory-controller column: a host-bound job shows low SM activity and near-zero copy activity in the gaps. Then run the decisive experiment, which takes ten minutes: tokenize one batch once, keep it on the device, and loop the forward pass. That measures your true kernel-only throughput, and the gap between it and end-to-end throughput is exactly your host overhead. Confirm the cause with torch.profiler (look for aten::copy_, cudaStreamSynchronize, and dataloader wait time) or an Nsight Systems timeline, where a host-bound run appears as short kernel clusters separated by long empty stretches on the CUDA stream. Common culprits found this way: the slow Python tokenizer instead of the Rust BertTokenizerFast, num_workers=0, tokenizing inside the request handler on the same thread that owns the GPU, a stray .item() or .cpu() inside the loop forcing a sync every step, and logging metrics per step.

Mathematical Formulation:
$U_{\mathrm{serial}} = \frac{t_{\mathrm{gpu}}}{t_{\mathrm{cpu}} + t_{\mathrm{gpu}}}$
$U_{\mathrm{ovl}} = \frac{t_{\mathrm{gpu}}}{\max(t_{\mathrm{cpu}}/W,\; t_{\mathrm{gpu}})}$
$C \propto B\,(L^2 d + L d^2)$
$w = 1 - \bar{L}/L_{\max}$

Where:

$U$ is the fraction of wall-clock time the GPU spends executing kernels, capped at 1, with $U_{\mathrm{serial}}$ for a blocking loop and $U_{\mathrm{ovl}}$ for an overlapped prefetch pipeline.
$t_{\mathrm{cpu}}$ is per-batch host time (tokenize, collate, copy) and $t_{\mathrm{gpu}}$ is per-batch device time for the forward, or forward plus backward when training.
$W$ is the number of dataloader workers or preprocessing processes; overlap only holds if their output lands in pinned memory and the copy is issued as non_blocking.
$C$ is the dominant FLOP count per batch with $B$ the batch size, $L$ the padded sequence length, and $d$ the hidden width; the $L^2 d$ term is attention and the $L d^2$ term is the projections and feed-forward.
$w$ is the fraction of computed tokens that are padding, with $\bar{L}$ the mean true length and $L_{\max}$ the padded length; $w$ understates the true waste whenever the $L^2$ term dominates.

Worked Numbers For One Batch:
$U_{\mathrm{serial}} = \frac{6}{18 + 6} = 0.25$
$U_{\mathrm{ovl}} = \frac{6}{\max(18/3,\; 6)} = 1.0$
$w = 1 - \frac{27}{128} = 0.79$

Three workers are enough to hide 18 ms of host work behind a 6 ms forward pass, and the same arithmetic tells you when adding workers stops helping: once $t_{\mathrm{cpu}}/W$ drops below $t_{\mathrm{gpu}}$ , extra workers only add memory pressure and page-cache churn. The padding figure is the other half of the story. Sorting a serving queue by length into buckets, or using dynamic padding to the longest member of each batch, typically recovers most of that 79% at zero accuracy cost, and it is strictly better than buying a larger GPU to compute attention over [PAD] tokens faster.

Bar chart of throughput in sequences per second for five BERT-base inference configurations at batch 32 and sequence length 128 on a single mid-range GPU: PyTorch eager FP32 with a serial tokenizer at 210, PyTorch eager FP32 with three workers and pinned memory at 320, ONNX Runtime FP32 with a fused graph at 480, TensorRT FP16 at 1150, and TensorRT INT8 with post-training calibration at 1900.

Figure 2: An illustrative optimization ladder for BERT-base at batch 32, sequence 128, on one mid-range GPU. Each rung removes a different bottleneck: host stalls first, then kernel launch and layout overhead through graph fusion, then arithmetic precision. Only the last rung can move your evaluation metric, which is why it ships behind an accuracy gate.

Property	PyTorch eager	ONNX Runtime	TensorRT (FP16 / INT8)
Graph optimization	None by default; op-by-op dispatch	Constant folding, LayerNorm and GELU fusion, attention fusion	Full autotuned kernel selection plus fused multi-head attention
Variable sequence length	Free; any shape runs	Dynamic axes supported, some re-optimization per new shape	Needs declared optimization profiles; shapes outside them fail or fall back
Build and deploy cost	Zero; ship the checkpoint	One export step, portable artifact	Minutes of engine build, plus a calibration pass for INT8
Portability	Runs anywhere PyTorch runs	CPU, GPU, and other execution providers	Engine is tied to GPU architecture and TensorRT version
Accuracy risk	Baseline	Numerically equivalent in FP32	FP16 usually within noise; INT8 needs a regression gate
Dominant failure mode	Launch overhead and host stalls dominate at small batch	Unsupported op forces a subgraph back to a slow path	Engine rebuild on driver or hardware change; activation outliers break INT8

Code Implementation:

# 1) hide host work: fast tokenizer, workers, pinned memory, prefetch
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True)

def collate(batch):
    # dynamic padding to the longest item, not to max_length
    return tok([b["text"] for b in batch], padding=True,
               truncation=True, max_length=128, return_tensors="pt")

loader = torch.utils.data.DataLoader(
    ds, batch_size=32, collate_fn=collate,
    num_workers=4, pin_memory=True, persistent_workers=True,
    prefetch_factor=4)

for batch in loader:
    batch = {k: v.to("cuda", non_blocking=True) for k, v in batch.items()}
    with torch.autocast("cuda", dtype=torch.float16):
        out = model(**batch)      # no .item() / .cpu() inside the loop

# 2) measure kernel-only throughput to prove where the time goes
cached = {k: v.cuda() for k, v in collate(ds[:32]).items()}
torch.cuda.synchronize(); t0 = time.perf_counter()
for _ in range(200):
    with torch.no_grad(), torch.autocast("cuda", dtype=torch.float16):
        model(**cached)
torch.cuda.synchronize()
print("device-only seq/s:", 200 * 32 / (time.perf_counter() - t0))

# 3) export and build: ONNX -> TensorRT with dynamic shape profiles
# torch.onnx.export(model, tuple(cached.values()), "bert.onnx",
#                   dynamic_axes={"input_ids": {0: "B", 1: "L"}}, opset_version=17)
# trtexec --onnx=bert.onnx --fp16 --int8 --calib=calib.cache \
#         --minShapes=input_ids:1x16 --optShapes=input_ids:32x128 \
#         --maxShapes=input_ids:64x256 --saveEngine=bert.plan

August 21, 2026

DL0199 Sensitive Data Memorization Remediation

Imagine you’ve trained a model with unusually strong memorization of sensitive data. What steps would you take next?

Answer

Handle it as a privacy incident with a measurement problem at its core, not as a modeling curiosity. The first hour is containment: stop the rollout, pin and lock every checkpoint derived from the suspect run, and bring legal and privacy owners in immediately, because notification clocks (for example the 72-hour clock in GDPR Article 33) start from awareness rather than from confirmation. Then quantify before you touch anything, since “the model memorizes” is not an actionable statement until you have a per-record extraction rate under an adversarial prompt distribution, a canary exposure number, and a membership-inference AUC. Diagnosis almost always lands on the same few causes: near-duplicate records, personal data that survived scrubbing because it arrived through production logs, or too many epochs over a small sensitive shard. Only then do you pick the cheapest remediation tier that clears an agreed risk gate, escalating output guard → targeted unlearning → retrain with deduplication and DP-SGD, and you verify with adversaries you did not design the fix against.

(1) Contain Before You Explain: Freeze the rollout, revoke external checkpoint and API access, and preserve prompts, completions, and data manifests so the incident can be scoped later.
(2) Quantify With Adversarial Probes: Report extraction rate under prefix, divergence, and template attacks plus canary exposure in bits, not a handful of prompts that happened to leak.
(3) Duplication Is The Usual Root Cause: A record repeated a dozen times is far more extractable than a record seen once, so a near-duplicate audit on the flagged records comes before any algorithmic fix.
(4) Use A Cost-Ordered Remediation Ladder: Output filters buy hours, targeted unlearning buys weeks, and only retraining with deduplication and DP-SGD yields a formal per-record bound.
(5) Verification Must Be Independent: Re-measure on held-out canaries and an adaptive red team, check utility regressions, and probe for relearning, because suppressed knowledge often returns after light fine-tuning.
(6) Deletion Obligations Propagate: Embeddings, indexes, eval logs, distilled students, and downstream fine-tunes all inherit the memorization, so the remediation plan must enumerate every derived artifact.

Serpentine incident-response flow: top row contains contain, quantify and diagnose stages; the middle row is an escalating remediation ladder read right to left from output guard to targeted unlearning to full retrain with deduplication and differential privacy, with escalate arrows between tiers; the bottom row runs verify, monitor and close out

Figure 1: The playbook, ordered by cost rather than by ambition. Each escalation step is triggered by a failed risk gate, not by preference, and the loop only closes when independent probes and a documented data lineage both hold.

The measurement phase decides everything downstream, so it deserves real engineering. Insert canaries (synthetic secrets with known formats and known insertion counts) into future runs, and for the current model estimate exposure by ranking the true secret against a candidate set of same-format alternatives under the model’s own likelihood. Complement this with black-box extraction rate: run a large, adversarially generated prompt set and count verbatim recoveries of records you know are in the training set. Two subtleties trip teams up. A single leaked record is enough to be reportable, so the metric that matters is a confidence bound on the leak probability rather than a point estimate. And zero observed hits is not zero risk: with no hits in $N$ probes, the 95% upper bound on the per-probe rate is about $3/N$ , which is why a 200-prompt smoke test can never clear a production gate.

Mathematical Formulation:
$\mathrm{exposure}(s) = \log_2 |\mathcal{R}| - \log_2 \mathrm{rank}(s)$
$\mathrm{rank}(s) = |\{ r \in \mathcal{R} : \mathcal{P}(r) \leq \mathcal{P}(s) \}|$
$\hat{e} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[ g(p_i) = s_i ]$
$\hat{e} = 0 \Rightarrow p_{95} \approx 3/N$
$\Pr[M(D) \in S] \leq e^{\epsilon} \Pr[M(D') \in S] + \delta$

Where:

$s$ is the sensitive string under test and $\mathcal{R}$ the set of same-format candidates it is ranked against, so $\log_2 |\mathcal{R}|$ (30 bits for a 9-digit-scale secret) is the maximum exposure.
$\mathcal{P}(\cdot)$ is the model’s perplexity of a candidate; $\mathrm{rank}(s) = 1$ means the true secret is the model’s single most likely completion.
$\hat{e}$ is the empirical extraction rate, $p_i$ the $i$ -th adversarial prompt, $g$ the decoding procedure, and $N$ the number of probes.
$p_{95}$ is the 95% upper confidence bound on the per-probe leak probability, which is the number a risk gate should be written against.
$M$ is the training mechanism, $D$ and $D'$ two datasets differing in one record, and $(\epsilon, \delta)$ the differential privacy budget that bounds any single record’s influence on the released weights.

Log-scale chart of canary exposure in bits against the number of times a sensitive record appears in the training corpus, showing a baseline curve that saturates near 30 bits after a few dozen duplicates, a slower curve for deduplication plus loss masking, and a nearly flat curve for DP-SGD training, with a shaded region above 20 bits marking where a single greedy decode recovers the secret

Figure 2: Why the duplicate audit comes first. Under ordinary training, roughly 13 near-duplicates of one record are enough to push exposure past 20 bits, where a single greedy decode recovers it; mitigations such as span masking only shift the curve right, while DP-SGD flattens it at the cost of long-tail accuracy.

Remediation tier	3a. Output-side guard	3b. Targeted unlearning	3c. Retrain (dedup + DP-SGD)
What it changes	Decoding and post-processing only; weights untouched	Weights, via a forget-set objective on the flagged records	The data pipeline and the training objective, from a clean checkpoint
Time to deploy	Hours	Hours to days	Days to weeks
Compute cost	Negligible, plus per-token filter latency	Roughly 0.1 to 1 percent of the original training FLOPs	A full training budget, plus DP-SGD clipping overhead
Guarantee	None; only block-list coverage of known strings	Empirical and benchmark-dependent; reversible by relearning	Formal per-record bound with a stated epsilon and delta
Utility cost	Near zero, aside from false blocks on common strings	Measurable drift on neighboring, legitimate knowledge	Largest on rare classes and long-tail facts
Dominant failure mode	Paraphrase, translation, or encoding walks around the match	The fact is still latent; only the surface form is suppressed	Epsilon accounting is void if one record recurs under many users

August 21, 2026

DL0198 TF-IDF Vectorization

What is TF-IDF vectorization, how does it combine term frequency and inverse document frequency to weight words, and what are its limitations compared to embedding-based representations?

Answer

TF-IDF maps each document to a vector whose length equals the vocabulary size, where coordinate $t$ holds a weight that rises with how often term $t$ occurs in that document and falls with how many documents in the corpus contain it. The term-frequency factor is local and per-document, usually damped as $1 + \log f_{t,d}$ so that ten occurrences do not count ten times as much as one. The inverse document frequency factor is a global corpus statistic, $\log(N/n_t)$ , which pushes function words such as “the” toward zero and lifts rare, discriminative terms. Multiplying the two produces the classic “frequent here, rare elsewhere” signal, and L2 normalization turns a plain dot product into cosine similarity. The result is extremely sparse (typically over 99.9% zeros), exactly interpretable per term, and requires no training beyond counting. Its weaknesses are structural rather than fixable by tuning, because every term is its own orthogonal dimension, so “car” and “automobile” share zero similarity, word order and negation are discarded, and any term outside the fitted vocabulary is silently dropped.

(1) Two Factors, Two Scopes: tf is computed from one document, idf from the whole corpus, and the product is what makes a term both present and distinctive.
(2) Sublinear Damping: raw counts overweight repetition, so $1+\log f_{t,d}$ or BM25 saturation caps the reward for saying the same word again.
(3) IDF Is Fitted, Not Computed Per Query: the $n_t$ statistics are frozen at fit time, so corpus drift, duplicated documents, and tiny corpora all distort the weights.
(4) Normalization Removes Length Bias: without L2 (or pivoted) normalization long documents dominate every ranking simply by containing more words.
(5) Sparse Means Cheap And Exact: an inverted index serves lexical matches on CPU in milliseconds, and high idf makes SKUs, error codes, and legal citations very precise.
(6) The Structural Limits: orthogonal vocabulary dimensions mean no synonymy or paraphrase, bag-of-words means no order or negation, and out-of-vocabulary terms contribute nothing at all.

Mathematical Formulation:
$w_{t,d} = \mathrm{tf}(t,d)\cdot \mathrm{idf}(t)$
$\mathrm{tf}(t,d) = 1 + \log f_{t,d}$
$\mathrm{idf}(t) = \log \frac{N}{n_t}$
$\mathrm{idf}_{s}(t) = \log \frac{1+N}{1+n_t} + 1$
$\hat{w}_{d} = w_{d} / \|w_{d}\|_2$
$\mathrm{sim}(q,d) = \hat{w}_{q}^{\top}\hat{w}_{d}$

Where:

$w_{t,d}$ is the unnormalized weight of term $t$ in document $d$ , and $\hat{w}_{d}$ is the final L2-normalized document vector.
$f_{t,d}$ is the raw count of $t$ in $d$ , defined only for $f_{t,d} \geq 1$ ; terms with zero count keep weight zero, which is what makes the vector sparse.
$N$ is the number of documents in the fitted corpus and $n_t$ the document frequency, the number of documents containing $t$ , so $1 \leq n_t \leq N$ .
$\mathrm{idf}_{s}$ is the smoothed variant used by scikit-learn, which adds one to both counts to tolerate unseen terms and adds a constant so a term present everywhere still keeps a small nonzero weight.
$q$ is the query treated as a short pseudo-document, so retrieval scoring is just a cosine between two sparse vectors over shared nonzero coordinates.

Worked IDF Example (natural log, one million documents):
$\mathrm{idf}(\mathrm{the}) = \log(10^6/10^6) = 0$
$\mathrm{idf}(\mathrm{car}) = \log(10^6/10^5) = 2.30$
$\mathrm{idf}(\mathrm{sarcoidosis}) = \log(10^6/50) = 9.90$

The spread of those three numbers is the whole mechanism. A stopword is annihilated without any hand-written stoplist, a common content word keeps a moderate weight, and a domain-specific term is worth roughly four common words in the same document. That also exposes the fragility: a term appearing in exactly one document receives the maximum weight even when it is a typo, an OCR error, or a hash, which is why production pipelines cap the vocabulary with min_df and max_df thresholds instead of trusting idf alone.

Two panels: on the left, inverse document frequency plotted against document frequency on a log x-axis for a corpus of ten thousand documents, comparing the plain log of N over n_t against the smoothed scikit-learn variant, with annotations at the rare-term and stopword ends; on the right, three term-frequency transforms plotted against raw term count, showing the raw linear count leaving the panel, the sublinear one-plus-log curve, and BM25 saturation flattening near two point two

Figure 1: The two factors behave very differently. IDF decays logarithmically in document frequency and reaches zero for a term present in every document, while the tf transform decides how much repetition is rewarded: raw counts grow without bound, $1+\log f_{t,d}$ gives a tenfold count only 3.3 times the weight of a single occurrence, and BM25 saturation is asymptotically bounded by $k_1+1$ .

BM25 is the natural upgrade inside the sparse family, replacing the log-tf term with an explicitly saturating form plus document-length normalization, and it usually beats plain TF-IDF on ranking while keeping the same inverted index. What neither can do is bridge vocabulary mismatch. In a TF-IDF space the coordinates for “car”, “automobile”, and “voiture” are mutually orthogonal, so a query using one word scores exactly zero against a document using another, no matter how many synonyms both share in meaning. Embedding-based representations solve this by construction, mapping text into a dense $\mathbb{R}^{d}$ with $d$ typically between 384 and 1024, where distributional training places related words and paraphrases close together and sub-word tokenization guarantees that nothing is out of vocabulary. The price is real: an encoder needs pretraining plus contrastive fine-tuning, similarity becomes uninterpretable, and dense retrievers can drift badly out of domain, which is exactly where exact lexical matching on an identifier still wins.

Left panel shows a six-term by five-document TF-IDF matrix as a grayscale heatmap with numeric cell values, where the row for the stopword the is all zeros and the rows for car and automobile never share a nonzero column; right panel sketches a dense embedding space as unit arrows from the origin, with car, automobile, and vehicle pointing in nearly the same direction, insurance rotated away, and banana pointing into a different quadrant

Figure 2: Left, a TF-IDF matrix is mostly zeros and each term owns its own axis, so the “car” and “automobile” rows never overlap and their cosine similarity is exactly 0. Right, a dense encoder places the same synonyms at a small angle, which is what recovers paraphrase recall while giving up the per-term readability of the sparse weights.

Property	TF-IDF / BM25 (sparse lexical)	Dense embeddings (bi-encoder)
Dimensionality	Vocabulary size, 50k to 1M, over 99.9% zeros per document	Fixed 384 to 1024 dense floats, fully populated
Fitting cost	One counting pass, no gradients, minutes on CPU	Pretraining plus contrastive fine-tuning on paired data
Synonyms and paraphrase	None: distinct surface forms are orthogonal dimensions	Captured: near-duplicate meanings land at small angles
Rare exact strings	Strong: high idf makes SKUs and error codes highly selective	Weak: rare identifiers are blurred into nearby token directions
Unseen terms	Dropped silently at transform time, contributing nothing	Always representable through sub-word tokenization
Order and negation	Lost, unless n-grams are added at a cost in dimensionality	Partly encoded by the contextual attention layers
Serving	Inverted index, CPU, millisecond lookups, trivial updates	ANN index such as HNSW or IVF, GPU for encoding queries
Dominant failure mode	Vocabulary mismatch gives zero recall on paraphrased queries	Out-of-domain drift returns topically close but wrong documents

August 21, 2026

DL0197 Continuous Incremental Online Learning

What is continuous learning after model deployment, and how do incremental learning and online learning differ in their handling of new data streams and catastrophic forgetting?

Answer

Continuous learning is the practice of keeping a deployed model current as the serving distribution drifts, by updating its parameters from the live stream instead of freezing the artifact that passed offline evaluation. The umbrella covers three regimes that differ in how much data one update sees and how often it ships: periodic batch retraining on the accumulated corpus, incremental learning that resumes from the current weights on a new chunk or task, and online learning that takes one low-learning-rate step per example or micro-batch and never revisits it. Incremental learning still controls the update: it can mix a replay sample of old data into every batch, run a few epochs over the chunk, and gate the result behind a shadow evaluation, so forgetting is a tunable quantity. Online learning gives up that control by construction, because each example is seen once, in arrival order, with no shuffling and no held-out replay, which makes the gradient sequence strongly non-i.i.d. and makes catastrophic forgetting the default rather than the exception. The core tension in both is the stability-plasticity dilemma: enough plasticity to absorb today’s traffic, enough stability to keep yesterday’s competence.

(1) Why Forgetting Happens: gradients computed only on new data are free to move weights that encoded the old distribution, since nothing in the loss references it. Breaking the i.i.d. sampling assumption of SGD is the mechanism, not model capacity.
(2) Incremental Learning Batches The Stream: updates operate on a chunk, task, or day of data, so replay ratios, epoch counts, and evaluation gates are all design knobs.
(3) Online Learning Streams One Pass: a single step per example, bounded memory, and no revisits, which buys seconds-level freshness and pays with instability and high variance.
(4) Three Families Of Mitigation: rehearsal (replay buffers, generative replay), regularization (EWC, distillation from the previous checkpoint), and parameter isolation (adapters, LoRA branches, masks).
(5) Forgetting Is Measurable: report per-task retention and average accuracy from the accuracy matrix, never a single aggregate on fresh traffic, which hides collapse on older slices.
(6) The Dual Failure Is Loss Of Plasticity: a model updated too conservatively for months stops learning at all, so stability alone is not the objective.

Diagram with a live traffic box on the left feeding three horizontal lanes: batch retraining on weeks of shuffled logged data with full offline evaluation and weekly deploys, incremental learning on a new chunk plus a replay sample with a few epochs from current weights and hourly to daily shadow-gated deploys, and online learning taking a single low-learning-rate step per example with learning-rate clipping, canary and rollback guardrails shipping in seconds

Figure 1: The gradient step is identical in all three lanes. What changes is how much data one update sees, how often it ships, and what protects the old distribution: shuffling over the full corpus, an explicit replay sample plus a shadow gate, or nothing but a small learning rate and a rollback button.

In production the choice is usually driven by label latency rather than by an appetite for novelty. Recommendation and ad ranking get feedback in seconds and genuinely benefit from near-online updates, so they run streaming updates on the embedding tables and the last layers while the backbone is refreshed on a slower incremental schedule. Fraud, credit, and medical models often wait days or weeks for a trustworthy label, so a per-example update would be trained on noise; incremental daily or weekly chunks with a replay mix are the safer default. The practical recipe that survives contact with traffic is rarely exotic: keep a stratified replay buffer covering old slices, mix roughly 5 to 20 percent of every batch from it, use a small re-warmed learning rate, and gate every candidate on a fixed regression suite of historical slices before promotion. Parameter isolation is attractive when tasks are known and separable, because a per-task adapter cannot be overwritten, but it grows parameters linearly in the number of tasks and needs task identity at inference unless the routing is learned.

Mathematical Formulation:
$\theta_t = \theta_{t-1} - \eta_t \nabla \ell(\theta_{t-1}; z_t)$
$B_t = \alpha B^{new}_t + (1-\alpha) B^{replay}_t$
$\mathcal{L}(\theta) = \mathcal{L}_{new}(\theta) + \lambda \Omega(\theta)$
$\Omega(\theta) = \tfrac{1}{2}\sum_i F_i (\theta_i - \theta^{*}_i)^2$
$A_K = \frac{1}{K}\sum_{j=1}^{K} a_{K,j}$
$f_j = \max_{k} a_{k,j} - a_{K,j}$

Where:

$\theta_t$ are the parameters after the $t$ -th update, $\eta_t$ the step size, and $z_t = (x_t, y_t)$ the arriving example; the pure online case uses each $z_t$ exactly once.
$B_t$ is the batch actually used at step $t$ , built from fresh data $B^{new}_t$ and a buffer sample $B^{replay}_t$ , with replay ratio $1-\alpha$ typically in the 0.05 to 0.20 range.
$\mathcal{L}_{new}$ is the loss on the incoming chunk and $\Omega$ a stability penalty anchored at the previous checkpoint $\theta^{*}$ , with $\lambda$ trading plasticity for retention.
$F_i$ is the diagonal Fisher information for parameter $i$ under the old task, so weights the old task depended on move less; setting $F_i = 1$ reduces the penalty to plain L2-to-previous.
$a_{k,j}$ is accuracy on task $j$ measured after training through task $k$ , with $j$ and $k$ indexing the $K$ chunks seen so far.
$A_K$ is average accuracy over everything seen and $f_j$ the forgetting on task $j$ , the drop from its best-ever value to its current one.

Figure 2: Sequential updates without rehearsal do not degrade gracefully, they collapse: task 1 loses 69 accuracy points by task 10. A 5 percent replay buffer recovers most of the gap to joint retraining at a fraction of the compute, while a stability penalty alone lands in between because it also suppresses learning on the new chunk.

Property	Batch retraining	Incremental learning	Online learning
Data per update	The full accumulated corpus, shuffled	One chunk, day, or task plus a replay sample	One example or micro-batch, in arrival order
Passes over data	Many epochs, i.i.d. sampling holds	A few epochs inside the chunk, old data only via buffer	Exactly one, no revisits
Freshness	Days to weeks behind traffic	Hours to a day behind	Seconds to minutes behind
Forgetting exposure	None by construction, old data is in every epoch	Bounded and tunable through replay ratio and penalty weight	High, the update sees only the current regime
Main safeguard	Full offline evaluation before promotion	Replay buffer, EWC or distillation, shadow eval on old slices	Small clipped learning rate, canary traffic, instant rollback
Dominant failure	Staleness under drift, and rising retraining cost	Buffer becomes unrepresentative, or the penalty freezes learning	Label noise and feedback loops steer the model within hours
Good fit	Slow drift, regulated or audited models	New domains, languages, or product surfaces arriving over time	Fast implicit feedback such as clicks, prices, or trending content

August 21, 2026

DL0196 Reward Model Overfitting

How do you prevent the reward model from overfitting in RLHF, and what role do data diversity, ensemble reward models, and uncertainty estimation play?

Answer

A reward model (RM) is a learned proxy for human judgement fitted on a finite set of preference pairs, so it fails in two distinct ways that need different defenses. The first is ordinary statistical overfitting: held-out pairwise accuracy typically plateaus in the 65-75% band and then degrades, which is why InstructGPT-era practice is a single epoch at a small learning rate, since a second epoch mostly memorizes annotator noise. The second and more damaging failure is over-optimization, where held-out accuracy still looks acceptable but the policy drifts into a region the RM never saw labeled, so the proxy score climbs while true human preference falls. The defenses are therefore spread across the whole loop rather than concentrated in the RM training script. Diverse prompts and on-policy responses widen the region where the RM is actually supervised, an ensemble of independently seeded RMs converts the unsupervised region into a measurable disagreement signal, and a pessimistic aggregate such as mean minus $\lambda$ times the spread turns that signal into a reward the policy cannot profit from. A KL penalty to the frozen reference policy plus periodic relabeling of fresh on-policy samples keeps the optimizer inside the RM’s zone of competence.

(1) Two Different Failures: Classical overfitting is measured on held-out preference pairs, while over-optimization is a Goodhart effect visible only when the RM is used as an RL objective. Fixing one does not fix the other.
(2) Train Short, Regularize Hard: One epoch, learning rates around $10^{-5}$ or below, early stopping on held-out pairwise accuracy, plus label smoothing or a margin term and an auxiliary language-modeling loss to keep the backbone from collapsing.
(3) Data Diversity Sets The Support: Prompts from real traffic, red-teaming, and synthetic edge cases, with responses sampled from several policy checkpoints and temperatures, so the RM is supervised where the policy will actually go.
(4) Multiple Annotators Bound The Ceiling: Human agreement of roughly 70-80% on subtle pairs caps achievable accuracy, so a model chasing 90% training accuracy is fitting noise, not preference.
(5) Ensembles Estimate Epistemic Uncertainty: $K = 3$ to $5$ members whose pretraining seeds differ disagree most off-distribution, which is exactly where reward hacking begins.
(6) Pessimism Turns Uncertainty Into Reward: Optimizing a lower confidence bound or the worst-of- $K$ score delays the over-optimization peak instead of merely detecting it.
(7) Close The Loop: Iterative rounds that relabel on-policy samples attack the root cause, distribution shift, rather than its symptom.

Loop diagram: a diverse prompt pool feeds response sampling from many policy checkpoints and temperatures, which feeds human preference labeling with two to three annotators per pair, which trains an ensemble of K reward models with independent pretrain seeds, whose outputs are aggregated into a mean and a spread, converted into a conservative reward equal to the mean minus lambda times the spread, and used by a PPO or GRPO policy update with a KL penalty, with a dashed feedback edge sending fresh on-policy responses back for relabeling each round

Figure 1: Where each defense sits in the loop. Diversity widens the labeled support, $K$ independently seeded reward models turn unlabeled regions into a measurable spread $s$ , and the conservative reward makes the policy pay for that spread. The dashed feedback edge is the only mechanism that actually removes distribution shift rather than penalizing it.

Data diversity is the highest-leverage lever because it changes the support of the training distribution rather than the fit on a fixed support. Preference sets collected from one prompt source and one policy checkpoint produce an RM that is accurate on that slice and arbitrary elsewhere, and the policy will find “elsewhere” within a few thousand PPO steps. Two diversity axes matter most: the prompt distribution, which should mix production traffic, adversarial red-team prompts, and long-tail synthetic cases, and the response distribution, which should include samples from several checkpoints, several temperatures, and deliberately balanced lengths so the RM does not learn that longer equals better. Multi-annotator labeling on a subset gives the agreement rate, and that rate is the practical accuracy ceiling: driving training accuracy far past it is a direct measurement of memorization.

Mathematical Formulation:
$\Delta_\theta = r_\theta(x, y_w) - r_\theta(x, y_l)$
$\mathcal{L}_{\mathrm{RM}} = -\log \sigma(\Delta_\theta)$
$\bar r(x,y) = \frac{1}{K}\sum_{k=1}^{K} r_k(x,y)$
$s^2(x,y) = \frac{1}{K}\sum_{k=1}^{K}(r_k - \bar r)^2$
$r_{\mathrm{LCB}} = \bar r - \lambda s$
$J(\pi) = \mathbb{E}_{\pi}[r_{\mathrm{LCB}}] - \beta D_{\mathrm{KL}}$

Where:

$\Delta_\theta$ is the score margin between the preferred response $y_w$ and the rejected response $y_l$ for prompt $x$ , and $\sigma$ is the logistic function, so $\mathcal{L}_{\mathrm{RM}}$ is the Bradley-Terry negative log-likelihood.
$\theta$ are the RM parameters: a pretrained or SFT backbone with a scalar head, usually trained for one epoch so $\mathcal{L}_{\mathrm{RM}}$ never approaches zero.
$r_k$ is the score of ensemble member $k \in \{1,\ldots,K\}$ , and $\bar r$ their mean, which is also a mild variance reduction on the reward itself.
$s(x,y)$ is the ensemble spread, used as an epistemic uncertainty proxy: it stays small on the labeled support and grows on responses no annotator ever judged.
$\lambda \geq 0$ is the pessimism coefficient, commonly in the range $0.5$ to $2$ ; $\lambda = 0$ recovers plain mean aggregation.
$\beta$ is the KL coefficient (typically $0.01$ to $0.05$ ) and $D_{\mathrm{KL}} = D_{\mathrm{KL}}(\pi \,\|\, \pi_{\mathrm{ref}})$ the divergence from the frozen reference policy, which bounds how far the policy may travel from the RM’s support.

Two-panel chart. Left panel: reward model training accuracy on preference pairs climbing from 0.69 toward 0.98 over four epochs while held-out validation accuracy peaks near 0.73 at one epoch and then declines, with an early-stopping marker at one epoch. Right panel: as policy drift measured by the square root of KL increases, the proxy reward from the reward model rises monotonically, the gold human reward peaks around square root KL of nine and then falls, and a conservative lower-confidence-bound reward reaches a later and higher peak around thirteen

Figure 2: Left: held-out pairwise accuracy peaks near one epoch while training accuracy keeps climbing, which is why single-epoch training with early stopping is the default. Right: the same RM used as an RL objective. The proxy score rises monotonically while the gold reward peaks and then declines, and pessimistic aggregation pushes that peak further out. Curves are schematic and follow the functional form fitted by Gao et al.

The right panel is the empirically important one, and its shape is well described by the over-optimization scaling law, where the gold reward is a function of the drift $d = \sqrt{D_{\mathrm{KL}}}$ rather than of training steps.

Over-Optimization Law:
$R_{\mathrm{gold}}(d) = d\,(a - b \log d)$
$d = \sqrt{D_{\mathrm{KL}}(\pi \,\|\, \pi_{\mathrm{ref}})}$

Here $a$ and $b$ are fitted constants that improve with RM size and with the amount of preference data, so a larger RM trained on more diverse comparisons does not merely score better, it peaks later. Ensembling contributes on two fronts. Averaging $K$ members reduces variance in the reward itself, which stabilizes advantage estimates, and the spread $s$ provides the only cheap in-loop signal that the policy has left the supervised region. In practice the disagreement metric is worth logging even if you never subtract it: a sudden rise in mean $s$ over sampled rollouts is an early warning that arrives before the gold reward turns over. Subtracting it, as in conservative or worst-of- $K$ optimization, costs a little early progress because uncertain but genuinely good responses are also penalized, and buys a substantially later peak. The critical implementation detail is that members must be diverse in their errors: fine-tuning five heads from one pretrained checkpoint with different shuffles yields correlated mistakes, so the spread underestimates true uncertainty precisely on hacked outputs.

Property	Data diversity and relabeling	Ensemble of K reward models	Pessimistic aggregation
What it fixes	Removes the unsupervised region the policy would exploit	Measures that region and reduces reward variance	Makes exploiting that region unprofitable
Mechanism	Widens the support: prompt sources, checkpoints, temperatures, fresh on-policy labels	Spread across members as an epistemic uncertainty proxy	Optimize a lower confidence bound or the worst-of-K score
Typical setting	Relabel every RLHF round; 2 to 3 annotators on a calibration subset	K = 3 to 5, independent pretrain seeds plus bootstrapped data	Pessimism coefficient roughly 0.5 to 2, tuned on gold evaluations
Cost	Human annotation budget and pipeline latency per round	K times training and K times reward-inference compute	Slower early progress; uncertain-but-good responses are penalized
Fails when	Annotators share a bias, such as preferring length or confident tone	Members share a pretrained backbone, so errors correlate	Spread is miscalibrated, so the penalty blocks progress or misses hacks

August 21, 2026

DL0195 Non-Max Suppression

What is Non-Max Suppression (NMS) in object detection, how does it work with confidence scores and IoU thresholds, and what are its limitations and alternatives such as Soft-NMS and DIoU-NMS?

Answer

NMS is the greedy post-processing step that turns a detector’s dense candidate set into a short, non-redundant list of boxes, and it normally runs independently per class. Boxes below a score floor are dropped first, the survivors are sorted by confidence, the highest-scoring box $M$ moves to the output list, and every remaining box whose IoU with $M$ exceeds a threshold $N_t$ is deleted. The loop then repeats on whatever is left: sort → pick the max → suppress → repeat. The algorithm rests on two assumptions, that confidence ranks localization quality and that a high IoU means two boxes describe the same object. Both fail in crowds, where the best box on a partially occluded neighbour can sit above $N_t$ against the winner and is silently erased. Soft-NMS replaces deletion with a monotonic score decay, and DIoU-NMS subtracts a normalized center distance from the IoU so that concentric duplicates are punished harder than offset neighbours.

(1) Greedy Per-Class Loop: NMS is not an optimizer, it is a sorted sweep, so the first mistake it makes is permanent because a deleted box never re-enters the pool.
(2) Two Thresholds, Not One: a score floor (often 0.001 for AP evaluation and 0.25 for a shipped product) plus the IoU threshold $N_t$ in the 0.5 to 0.7 range, with a top- $k$ cap in front to bound the cost.
(3) Confidence Is The Ranking Key: classification score and box quality are only loosely correlated, so NMS can keep a confident but sloppy box and delete a precise one, which is why IoU-aware or centerness-weighted scores improve NMS without touching the algorithm.
(4) Cost: an $O(n \log n)$ sort plus up to $O(n^2)$ pairwise IoU tests, which is why production stacks cap candidates and run a batched CUDA kernel.
(5) Dominant Failure Mode: a single global $N_t$ must serve sparse and crowded regions of the same image, so lowering it kills recall in crowds and raising it floods sparse regions with duplicates.
(6) Alternatives: Soft-NMS (linear or Gaussian decay), DIoU-NMS (center-distance penalty), Weighted and Cluster-NMS, Matrix NMS, and NMS-free detectors that learn one-to-one assignment instead.

Two side-by-side panels showing a heavily occluded pair of pedestrians. The left panel draws four candidate boxes with confidence scores 0.94 and 0.81 on the first person and 0.72 and 0.55 on the second, with the pairwise IoU values against the top box listed. The right panel shows the result of greedy NMS at IoU threshold 0.5: the 0.94 and 0.55 boxes are kept as solid outlines while the 0.81 duplicate and the 0.72 box on the second person are drawn dashed and grey as suppressed.

Figure 1: Greedy NMS on a heavily occluded pair. The 0.81 box is a genuine duplicate and should go, but the 0.72 box is the detector’s best evidence for the second person, and its IoU of 0.53 with the winner crosses the 0.5 threshold, so hard NMS erases it. What remains for that person is a poorly localized 0.55 box, and Gaussian Soft-NMS would instead keep the 0.72 box at a decayed 0.41.

Mathematical Formulation:
$M = \arg\max_{b_i \in \mathcal{B}} s_i$
$\mathrm{IoU}(M, b_i) = |M \cap b_i| / |M \cup b_i|$
$s_i \leftarrow s_i \cdot \mathbf{1}[\mathrm{IoU}(M, b_i) \leq N_t]$
$s_i \leftarrow s_i \, (1 - \mathrm{IoU}(M, b_i))$
$s_i \leftarrow s_i \exp(-\mathrm{IoU}(M, b_i)^2 / \sigma)$
$\mathrm{DIoU}(M, b_i) = \mathrm{IoU} - \rho^2 / c^2$

Where:

$\mathcal{B}$ is the current candidate set for one class, $b_i$ a box in it with score $s_i$ , and $M$ the current maximum-score box that is moved to the output list $\mathcal{D}$ .
$N_t$ is the IoU threshold; line 3 is hard NMS, which zeroes (deletes) any box overlapping $M$ more than $N_t$ .
Line 4 is linear Soft-NMS, applied only when $\mathrm{IoU}(M, b_i) > N_t$ ; line 5 is Gaussian Soft-NMS, applied to every remaining box with no threshold at all.
$\sigma$ controls the decay width, typically $\sigma = 0.5$ ; a smaller $\sigma$ makes Soft-NMS behave more like hard NMS.
$\rho$ is the Euclidean distance between the two box centers and $c$ the diagonal of the smallest box enclosing both, so $\rho^2 / c^2$ lies in $[0, 1)$ and is scale-invariant.
$i$ indexes the surviving candidates, and the loop terminates when $\mathcal{B}$ is empty or every score falls under the final report floor.

Worked Decay For The Erased Box:
$\mathrm{IoU}(M, b) = 0.53 > N_t = 0.5$
$0.53^2 / 0.5 = 0.5618$
$0.72 \times \exp(-0.5618) = 0.41$

Hard NMS maps that box to 0, Soft-NMS maps it to 0.41, and the practical difference is whether a report threshold of 0.3 still shows the second person. This is also the clearest way to see why Soft-NMS buys roughly 1 to 2 AP on MS COCO with no retraining: average precision rewards a correctly located box even at low confidence, because a decayed detection is ranked below the confident ones and only costs precision after all the good detections are already counted. The same property is a liability in a live product, since nothing is ever deleted and the output list keeps every candidate at some non-zero score, so a final score floor and a top- $k$ cap become mandatory rather than optional.

Line chart with IoU against the kept box on the horizontal axis from 0 to 1 and the score multiplier applied to the overlapping box on the vertical axis from 0 to 1. Hard NMS is a step function that holds at 1 until IoU 0.5 and then drops to 0. Linear Soft-NMS holds at 1 until 0.5 and then falls linearly to 0 at IoU 1. Gaussian Soft-NMS with sigma 0.5 decays smoothly from 1 and reaches about 0.14 at IoU 1, with an annotation marking that IoU 0.53 keeps 57 percent of the score.

Figure 2: Every NMS variant is just a different score multiplier as a function of IoU. Hard NMS is a discontinuous step at $N_t$ , which is what makes a 0.499 and a 0.501 overlap have completely different fates. Soft-NMS swaps the step for a monotonic decay, so the cliff at the threshold disappears and the ranking, rather than a hard rule, decides what appears in the final list.

DIoU-NMS attacks a different weakness: IoU alone cannot tell a concentric duplicate from a genuinely different object. Two boxes that share a center almost certainly describe the same thing, while two boxes with the same IoU but far-apart centers are much more likely to be neighbouring instances, so the criterion becomes IoU minus the squared center distance normalized by the enclosing diagonal. The catch is that same normalization. For two tall, near-identical pedestrian boxes offset by a fraction of their width, $\rho^2 / c^2$ is on the order of 0.01, so DIoU-NMS behaves almost exactly like hard NMS on the case in Figure 1. It helps most where scale or center offset differs substantially, which is why it is usually reported together with the DIoU regression loss rather than as a standalone fix for crowds.

Two panels each showing a large kept box M with a candidate box. In the left panel the candidate is a smaller box concentric with M, both centers coincide, IoU is 0.25, the center distance is zero and DIoU stays 0.25 so the box is suppressed at a 0.20 threshold. In the right panel the candidate is the same size as M but shifted horizontally, IoU is again 0.25, the center distance is 3.6 with the dashed enclosing box diagonal marked, the penalty is 0.10 and DIoU falls to 0.15 so the box is kept.

Figure 3: Both candidates have identical IoU 0.25 with the kept box, so hard NMS treats them identically. DIoU-NMS separates them: the concentric box keeps DIoU 0.25 and is deleted, while the offset box pays a 0.10 penalty and survives at 0.15 under a 0.20 threshold. Because the penalty is divided by the enclosing-box diagonal, it only bites when the center offset is large relative to the pair’s extent.

Property	Hard NMS	Soft-NMS	DIoU-NMS
Rule on an overlapping box	Delete if IoU exceeds Nt	Multiply the score by a linear or Gaussian decay, never delete	Delete if IoU minus the normalized center distance exceeds the threshold
Hyperparameters	Nt only, usually 0.5 to 0.7	Decay form, sigma near 0.5, plus a mandatory final score floor	Threshold plus the implicit distance normalization (sometimes an exponent beta)
Output size	Bounded and small	Every candidate survives with some score, so top-k is required	Bounded, slightly larger than hard NMS
Crowded scenes	Worst case, deletes true neighbours outright	Best AP gain, roughly 1 to 2 AP on MS COCO with no retraining	Helps when centers differ; near no effect for side-by-side boxes of equal size
Cost	Sort plus pairwise IoU, fast batched CUDA kernel	Same order but no early deletion, so more IoU work and a longer output list	Hard NMS plus one center-distance term per pair
Typical use	Default in Faster R-CNN and YOLO inference paths	Benchmark AP, crowd and dense-object settings, offline pipelines	Shipped with the DIoU/CIoU loss family, for example YOLOv4

The deeper limitation is structural rather than parametric: NMS is a non-differentiable, hand-tuned rule bolted onto a learned model, so the network is never trained to produce exactly one box per object. That is what NMS-free detectors remove. DETR-style models use Hungarian one-to-one matching during training, so duplicate suppression becomes a learned property of the decoder, and YOLOv10 keeps a one-to-many head for training signal while using a one-to-one head at inference, eliminating the NMS stage and its latency variance entirely.

August 21, 2026

DL0194 Sensor Degradation Feature Extractor

How do you detect and handle sensor degradation (lens distortion, rain noise, sensor misalignment) dynamically within a deep feature extractor pipeline, for a production perception stack such as a robotaxi’s camera-LiDAR fusion model or a Valeo-style surround-view ADAS?

Answer

Degradation is not handled by making the backbone bigger. It is handled by a cheap monitoring path that runs beside the feature extractor and a policy layer that changes how the extractor’s outputs are consumed. Three monitor families are affordable per frame: referenceless image-quality and soiling heads on the raw or early-feature tensor, feature-space drift statistics such as a Mahalanobis distance between the current channel-mean vector and the training reference, and geometric self-consistency residuals such as cross-sensor reprojection error, which is the only signal that separates a genuine extrinsic shift from bad weather. Detection alone is worthless, so those signals drive three mitigations: a quality gate that reweights per-sensor features before fusion, FiLM-style conditioning that lets the shared trunk adapt its normalization to the measured degradation, and an operational-design-domain (ODD) fallback that reduces speed, disables a fused output, or triggers a wiper and an online extrinsic re-estimation. The pipeline therefore reads capture → ISP → undistort → backbone → gate → fusion → heads, with the monitors tapping the first three stages and writing only into the gate.

(1) Separate Detection From Correction: a small monitor head that answers “how bad is this input” is far easier to train and validate than a backbone expected to be silently invariant to everything. It also gives you an auditable signal to log.
(2) Three Complementary Signals: pixel-level quality catches soiling and droplets, feature drift catches global appearance shift like rain veiling or blooming, and reprojection residuals catch calibration error. No single one covers all three failure classes.
(3) Geometry For Geometry Faults: a 0.5 degree extrinsic yaw drift leaves every image looking perfectly clean, so appearance-based monitors are blind to it; only cross-sensor residuals or photometric alignment expose it.
(4) Gate The Fusion, Not The Backbone: down-weighting a corrupted branch with normalized weights $w_m$ is a one-line change at inference and needs no retraining, whereas swapping backbone weights per weather condition doubles the validation surface.
(5) Train For Dropout, Not Just For Rain: corruption augmentation plus random modality dropout is what makes a gated fusion model usable when a branch is masked, otherwise the fused head has never seen a zeroed input.
(6) Absolute Scores Plus An Abstention Path: a softmax gate can only express relative trust, so a separate calibrated absolute quality score must be able to declare that all sensors are bad and hand control to the ODD layer.

Architecture diagram with three sensor lanes for front camera, side camera, and LiDAR, each passing through a per-sensor encoder and a quality head, all feeding a tall degradation gate box that computes fusion weights and FiLM conditioning, then a feature fusion block, then detection heads and a degradation-state and fallback block, with an online monitor box at the bottom sending dashed arrows to the gate and back to the encoder for recalibration

Figure 1: The monitors sit outside the critical path and write only into the gate. An alarm changes the fusion weight $w_m$ , the FiLM conditioning, and the declared ODD, but it never edits backbone weights at runtime, which keeps the deployed model bit-identical and the failure behaviour testable.

Each monitor has a different latency and a different false-alarm profile. The soiling and quality head is a per-tile classifier over an early feature map, typically under 1 ms on an embedded accelerator, and it is the only monitor fast enough to drive a physical actuator such as a nozzle or a heater. Feature drift is computed from the channel means of a mid-level tensor against a reference mean and covariance collected on clean data, then smoothed by a CUSUM accumulator so that a single dark frame does not raise an alarm while a sustained shift does within a few frames. Reprojection residuals need matched features across overlapping fields of view or LiDAR points projected into the image, so they run at a lower rate over a sliding window of several seconds, which is acceptable because extrinsic drift from thermal expansion or a curb strike is either slow or a step change that persists. Lens distortion sits between these cases: an intrinsics change makes straight lines curve and inflates residuals even for a single camera, and the correct response is to re-estimate the undistortion look-up table rather than to touch the network, because a CNN trained on rectified images treats a mis-rectified frame as out-of-distribution geometry.

Two stacked time-series panels over 300 camera frames. The top panel shows feature-drift distance rising from about 1.15 to 3.5 during a shaded rain burst between frames 90 and 170 while the reprojection residual stays near 0.35 pixels, then a second shaded region after frame 200 where the reprojection residual steps to about 2.5 pixels while feature drift stays low, with a dashed one-pixel residual threshold. The bottom panel shows the CUSUM statistic on feature drift staying at zero, then sawtoothing above a threshold of five during the rain burst with alarm markers.

Figure 2: The two monitors have orthogonal signatures. Rain moves the feature-drift statistic and leaves the reprojection residual untouched, while a mount shift moves the residual by 2.5 px without disturbing appearance statistics at all. Reading only one monitor guarantees you misdiagnose one of the two faults.

Quality-Gated Fusion:
$q_m = \sigma(g_{\phi}(F_m))$
$\hat{F}_m = \gamma(q_m) \odot F_m + \beta(q_m)$
$w_m = \frac{\exp(a_m + \log q_m)}{\sum_j \exp(a_j + \log q_j)}$
$F = \sum_m w_m \hat{F}_m$

Where:

$F$ is the fused feature tensor consumed by the task heads, and $F_m$ is the raw feature map of sensor $m$ , with $\hat{F}_m$ its degradation-conditioned version.
$q_m \in [0,1]$ is the absolute quality score produced by a small monitor head $g_{\phi}$ with logistic output $\sigma$ ; it is supervised by synthetic corruption labels and calibrated on held-out real degraded clips.
$\gamma(\cdot)$ and $\beta(\cdot)$ are the FiLM scale and shift vectors, and $\odot$ is channel-wise multiplication.
$a_m$ is a content-dependent attention logit from the ordinary fusion module, so the gate combines what is informative with what is trustworthy.
$w_m$ sums to 1 over sensors $m$ , which is why a low but uniform $q$ across all sensors must be caught by the raw $q_m$ values rather than by the weights.

Runtime Degradation Monitors:
$d_t^2 = (\mu_t - \mu_0)^{\top} \Sigma_0^{-1} (\mu_t - \mu_0)$
$S_t = \max(0, S_{t-1} + d_t - k)$
$r_{ij} = \| u_i - \pi(T_{ij} X_j) \|_2$

Where:

$d_t$ is the Mahalanobis drift at frame $t$ between the current channel-mean vector $\mu_t$ of a mid-level feature map and the clean-data reference $\mu_0$ with covariance $\Sigma_0$ .
$S_t$ is the CUSUM statistic with slack $k$ , which is set just above the clean-condition mean of $d_t$ ; an alarm is raised when $S_t > h$ and $S_t$ is then reset to 0.
$r_{ij}$ is the reprojection residual in pixels for correspondence $(i,j)$ , where $u_i$ is the observed image point, $X_j$ the 3D point from LiDAR or a second camera, $T_{ij}$ the extrinsic transform, and $\pi$ the projection using current intrinsics.
A robust percentile of $r_{ij}$ above roughly 1 px sustained over a window indicates extrinsic or intrinsic drift rather than matching noise, and triggers online recalibration.

The handling policy has to be trained for, not bolted on. A gate that can zero a camera branch is only safe if the fused head saw randomly dropped modalities and heavy corruption augmentation during training, otherwise masking a branch pushes the fusion layer into a region it never visited and accuracy collapses harder than with the corrupted input left in place. Augmentation alone is also insufficient, because it buys average-case robustness while a gate buys graceful worst-case behaviour: at high rain severity the model that can lean on LiDAR keeps far more of its mAP than the model that must average a clean point cloud with a veiled image. The cost is roughly one mAP point in clean weather, from the gate occasionally distrusting a good camera, plus the engineering burden of calibrating $q_m$ so that the gate does not permanently learn to ignore a sensor after a single bad deployment week.

Line chart of mAP versus rain and spray severity from level zero to five for three configurations: a clean-trained fusion baseline falling from 58 to 19, the same model with corruption augmentation falling from 58 to 31, and augmentation plus quality gating with LiDAR fallback starting slightly lower at 57 and falling only to 40, with an annotation noting the gate down-weights the camera branch at high severity

Figure 3: Corruption augmentation flattens the curve, but only the quality gate changes the shape of the tail, because it can stop trusting the camera entirely. The 1-point clean-weather cost at severity 0 is the price of that option, and it is the number a reviewer should ask you for.

Degradation	Detection signal	Where it runs	Runtime mitigation
Lens soiling, droplets	Per-tile soiling mask, loss of high-frequency energy	Early feature map, per frame, under 1 ms	Actuate nozzle or heater, mask affected tiles, lower that camera’s weight
Rain, spray, fog veiling	Feature-drift CUSUM on channel statistics, quality head score	Mid-level tensor, per frame with a few-frame delay	FiLM conditioning, shift fusion weight toward LiDAR and radar
Intrinsics or distortion drift	Straight-line curvature, single-camera reprojection residual	Sliding window of seconds, off the critical path	Re-estimate the undistortion look-up table before the backbone
Extrinsic misalignment	Cross-sensor residual above 1 px, LiDAR edge to image edge offset	Sliding window, low rate, host CPU acceptable	Online extrinsic correction, disable geometric fusion if outside bound
Full blockage or frozen stream	Frame hash repetition, entropy collapse, timestamp gap	Driver layer, before the network	Drop the modality using dropout-trained fusion, reduce the declared ODD

August 21, 2026

DL0193 DINOv3 Frozen Backbone Zero-Shot Transfer

Why can DINOv3 perform zero-shot dense segmentation and classification using a completely frozen backbone without task-specific fine-tuning, and what properties of its self-supervised pretraining enable this transfer?

Answer

DINOv3 is pretrained with a purely discriminative self-distillation objective that supervises two granularities at once: an image-level DINO loss on the CLS token and a patch-level iBOT masked-latent loss on individual patch tokens. Because the target is another view’s latent assignment rather than a caption, patch tokens are never squeezed through a text bottleneck, so the $P \times d$ patch grid emerging from the last block is already a linearly separable semantic map rather than an intermediate code that only a trained decoder can read. The property that keeps this true at 7B parameters and 1.689B curated images is Gram anchoring: late in training DINOv3 adds a loss that matches the student’s patch-token Gram matrix to that of an earlier EMA teacher, pinning the patch-to-patch similarity structure while global features keep improving. Register tokens remove the high-norm artifact patches that otherwise pollute attention maps, and axial RoPE with frequency jittering makes the same weights usable at resolutions far above the training crop. The result is that read-out becomes non-parametric: k-NN on the CLS token classifies, and cosine similarity between patch embeddings segments, with no gradient ever entering the backbone. A single linear layer on the frozen features reaches roughly 88% ImageNet-1k top-1 and around 56 mIoU on ADE20k, which is why “frozen backbone plus tiny head” is a competitive deployment strategy rather than a baseline.

(1) Two Granularities Of Supervision: the DINO term shapes the CLS token for image-level tasks while the iBOT term shapes individual patch tokens, so dense and global capability are trained jointly instead of one being a by-product.
(2) No Text Bottleneck: contrastive image-text training pushes patch information toward one caption-level summary, whereas latent self-distillation leaves per-patch detail intact, which is exactly what dense matching consumes.
(3) Gram Anchoring Preserves Dense Structure: constraining the $P \times P$ similarity matrix, not the absolute features, lets representations keep drifting globally while patch-similarity geometry stays sharp.
(4) Artifact-Free High-Resolution Maps: register tokens absorb the global-information outliers that would otherwise appear as bright background patches, and jittered axial RoPE keeps features stable when inputs are upscaled.
(5) Read-Out Is Non-Parametric: cosine similarity to a handful of reference patches or a simple clustering of patch tokens yields masks with zero trained parameters, so “zero-shot” here is literal.
(6) Scale, Curation, Then Distillation: a 7B teacher on curated web data is distilled into ViT-S/B/L and ConvNeXt students that inherit the dense property, so the same recipe ships at deployable sizes.

Left-to-right diagram: an input image at any resolution passes through 16x16 patch embedding with four register tokens and jittered axial RoPE into a frozen ViT-7B/16 that receives no gradients, producing a CLS token and P patch tokens, which feed three read-out paths: k-NN or linear probe on the CLS token for ImageNet classification, patch-to-patch cosine similarity for zero-shot masks with no trained head, and a lightweight linear or DPT head for ADE20k segmentation and NYUv2 depth

Figure 1: One frozen backbone, three read-outs. The only task-specific parameters live to the right of the token boundary, and the middle path has none at all: it is pure cosine similarity between patch embeddings, which is only meaningful because Gram anchoring protected that similarity structure during pretraining.

The interesting failure this design fixes is temporal. Train DINO plus iBOT long enough at 7B scale and the two granularities diverge: image-level metrics such as ImageNet linear probing keep improving, while patch-level metrics (segmentation with a frozen linear head, video tracking, patch-similarity coherence) peak relatively early and then degrade. Patch tokens slowly become carriers of global, image-level evidence, so neighbouring patches on the same object stop being each other’s nearest neighbours and background patches start matching foreground ones. Monitoring only ImageNet accuracy hides this completely, which is why the effect went unaddressed for so long. Gram anchoring intervenes on the second-order statistic rather than the features: the student must reproduce the teacher’s patch-patch cosine matrix, so it is free to rotate, rescale, or refine the embedding space while the relational geometry that dense tasks read out is held in place. Meta reports the anchoring loss switched on in a late phase of training, using a Gram teacher whose features are computed at higher resolution and downsampled, which sharpens the target further.

Dual-axis schematic line chart over 1.2 million training iterations: the ImageNet-1k linear probe top-1 curve rises monotonically from about 76 to 88 percent, while the ADE20k frozen linear mIoU curve rises to about 56 by 0.22 million iterations, decays to about 48 by 1.0 million, then jumps back above 56 after a dash-dot vertical line marking where Gram anchoring is switched on

Figure 2: Schematic shape of the reported trend. Global and dense quality diverge under long self-supervised training, so a model that looks strictly better on classification can be strictly worse for zero-shot dense read-out. Gram anchoring recovers the dense curve without sacrificing the global one.

Mathematical Formulation:
$\mathcal{L} = \mathcal{L}_{\mathrm{DINO}} + \mathcal{L}_{\mathrm{iBOT}} + \lambda \mathcal{L}_{\mathrm{Gram}}$
$G_S = \hat{X}_S \hat{X}_S^{\top}$
$\mathcal{L}_{\mathrm{Gram}} = \| G_S - G_T \|_F^2$
$P = (H/16) \times (W/16)$
$s_{pq} = \hat{x}_p^{\top} \hat{r}_q$
$\hat{y}_p = \arg\max_q s_{pq}$

Where:

$\mathcal{L}_{\mathrm{DINO}}$ is the image-level cross-view self-distillation loss on the CLS token and $\mathcal{L}_{\mathrm{iBOT}}$ the patch-level masked-latent loss, with $\lambda$ weighting the anchoring term.
$\hat{X}_S \in \mathbb{R}^{P \times d}$ holds the student’s $L_2$ -normalized patch features and $\hat{X}_T$ the same for the Gram teacher, an earlier EMA checkpoint evaluated at higher resolution and downsampled.
$G_S, G_T \in \mathbb{R}^{P \times P}$ are the patch-patch cosine similarity matrices, so the loss constrains relations between patches and not the patch vectors themselves.
$P$ is the number of patch tokens for an $H \times W$ input at patch size 16, and $d$ the model width; the anchoring loss therefore costs $O(P^2 d)$ per view.
$\hat{x}_p$ is the normalized embedding of test patch $p$ and $\hat{r}_q$ a normalized reference embedding, taken from an annotated patch, a class prototype, or a cluster centroid.
$\hat{y}_p$ is the predicted label for patch $p$ ; the read-out has no learned weights, and the backbone is evaluated with $\lambda$ and all pretraining machinery discarded.

Two side-by-side grayscale heatmaps of cosine similarity to a single query patch marked by a red square inside a large elliptical object; the left map is sharp with high similarity confined to the object and near-zero elsewhere including on a nearby distractor object, while the right map is blurred and noisy with spurious high-similarity patches scattered across the background

Figure 3: What one row of $G$ looks like in image space. Zero-shot dense segmentation is nothing more than thresholding or arg-maxing maps like these, so map sharpness is the capability: the right-hand panel, typical of dense degradation, produces masks that bleed into the background even though the model’s classification accuracy is higher.

Two practical consequences follow. First, “zero-shot” in this setting means no text alignment: DINOv3 can tell you that patch $p$ belongs to the same thing as your reference patch, but it cannot be queried with the string “traffic cone” the way SigLIP or CLIP can, so open-vocabulary work still needs a language-aligned encoder or a small aligner trained on top. Second, freezing is what makes the model economical to operate: one forward pass produces features that many heads share, features can be cached, and there is no risk of catastrophic forgetting when a new task arrives.

Property	Frozen DINOv3	Frozen CLIP / SigLIP	Supervised fine-tuning
Pretraining signal	Self-distillation on images only, image-level plus patch-level	Image-text contrastive or sigmoid loss on alt-text pairs	Task labels, one dataset at a time
Patch-token quality	High and explicitly protected by Gram anchoring	Weaker: detail is compressed toward the caption-level summary	High for the trained task, unreliable off-task
Zero-shot dense masks	Yes, by cosine matching or clustering of patch tokens	Poor without a dense-adaptation stage	Not applicable, the head defines the label set
Text queries	None, references must be visual examples	Native open-vocabulary querying	Fixed closed vocabulary
Adaptation cost	Cache features once, train a linear or DPT head in minutes	Cheap for classification, expensive for dense tasks	Full backbone gradients per task, plus a new checkpoint
Dominant failure mode	Domains far from curated web imagery, and no semantics for text prompts	Blurred patch features and prompt sensitivity	Catastrophic forgetting and per-task serving cost

August 20, 2026

DL0192 Depth Anything 3 Cross-View Depth Estimation

How does Depth Anything 3 enable cross-view interaction for consistent multi-view monocular depth estimation, and what architectural changes distinguish it from its single-view predecessors?

Answer

Depth Anything 3 keeps the monocular recipe of its predecessors and changes essentially one thing inside the network: the scope of self-attention. Each of the $N$ input views is patchified by the same plain DINOv2 encoder, the per-view token sequences are concatenated into a single sequence, and the stack then alternates between within-view attention, where a query sees only its own view and fine monocular detail survives, and cross-view attention, where every token attends to every token of every view and correspondence, relative pose, and a common scale are learned. No fusion module, cross-attention adapter, or cost volume is introduced. The pretrained attention weights are simply given a wider window, so a single-view input degenerates exactly to the original monocular model, which is why multi-view capability does not cost single-image quality. The second change is the output: instead of the per-image affine-invariant disparity of Depth Anything V1 and V2, DA3 predicts a depth-ray target per view (a depth map plus a ray map), and depth along predicted rays back-projects to one point cloud and one set of camera poses under a single global scale. That one target replaces the multi-head, multi-task output of VGGT-style geometry transformers, and the reported gains over VGGT are roughly 44% on camera pose accuracy and 25% on geometric accuracy, while monocular depth still improves over DA2.

(1) Attention Scope, Not A New Module: cross-view interaction is implemented by concatenating view tokens and letting the existing self-attention layers run over the union, so no modality-specific or view-specific parameters are added.
(2) Interleaved Within-View And Cross-View Layers: within-view layers protect high-frequency monocular detail, cross-view layers enforce geometric agreement, and the two are alternated through the stack.
(3) Input-Adaptive Degeneration: at $N = 1$ the cross-view layer is numerically identical to the single-view layer, so the model is a strict superset of its monocular predecessor rather than a compromise.
(4) Single Depth-Ray Target: one head predicts depth plus a ray map per view; depth along rays → point cloud, and camera pose is read out of the ray field instead of a dedicated pose head.
(5) One Global Scale For The Whole Set: normalization is fitted once over all views rather than a free scale and shift per image, which is precisely what removes per-frame flicker and non-overlapping point clouds.
(6) Plain Backbone, Teacher-Student Data: a vanilla DINOv2 transformer with a DPT-style dense head is enough; the accuracy comes from the target and from teacher-student pseudo-labelling, not from architectural specialisation.
(7) The Price Is Quadratic: a cross-view layer costs $N$ times a within-view layer, so view count, not image resolution, becomes the dominant memory term.

Architecture diagram: three input views feed one shared DINOv2 patch embedding that concatenates N times 1369 tokens into a single sequence, which passes through a within-view attention block and then a cross-view attention block interleaved over M blocks, then a shared DPT-style dense head that emits a depth map and a ray map per view, which are fused into one point cloud and camera poses in a shared frame under a single global scale

Figure 1: One encoder, two attention scopes, one target. The only cross-view machinery is the wider attention window, and the only output is a depth map plus a ray map per view, from which the point cloud and the camera poses are derived rather than predicted by separate heads.

It helps to look at the attention mask directly. With $N$ views of $L$ tokens each, a within-view layer is a block-diagonal mask: $N$ independent $L \times L$ blocks, exactly what a monocular model computes, repeated in parallel. A cross-view layer fills in the off-diagonal blocks, and those off-diagonal entries are the whole mechanism, because a token on a wall corner in view 3 can now match the same corner in view 1 and inherit its depth ordering. Because the projection matrices are unchanged, the same weights serve both scopes and the model never has to learn a separate matching operator. The cost of filling those blocks is the reason view count dominates the budget, and it is also the reason non-overlapping views buy nothing: the off-diagonal blocks exist, but there is no correspondence for them to find, so the relative scale between two disjoint clusters of views stays unconstrained.

Two 3-by-3 block attention masks for three views with eight tokens each: the left mask has only the three diagonal within-view blocks filled and the six off-diagonal blocks marked masked, totalling 192 token pairs; the right mask has the diagonal within-view blocks plus all six off-diagonal cross-view blocks filled, totalling 576 token pairs

Figure 2: The same layer, two masks. Single-view models compute only the block diagonal; DA3 fills the off-diagonal blocks, and cross-view correspondence lives entirely there. The pair count grows from $N L^2$ to $(NL)^2$ , a factor of exactly $N$ .

Mathematical Formulation:
$Z = [\,X_1; X_2; \ldots; X_N\,]$
$A_{\mathrm{within}}(v) = \mathrm{softmax}(Q_v K_v^{\top}/\sqrt{d})V_v$
$A_{\mathrm{cross}} = \mathrm{softmax}(QK^{\top}/\sqrt{d})V$
$P_v(u) = o_v + d_v(u)\, r_v(u)$
$s = \mathrm{median}_{v,u}\, \| P_v(u) \|$
$C_{\mathrm{within}} = N L^2$
$C_{\mathrm{cross}} = N^2 L^2$

Where:

$X_v \in \mathbb{R}^{L \times d}$ holds the tokens of view $v$ and $Z \in \mathbb{R}^{NL \times d}$ is the single concatenated sequence the transformer actually sees.
$Q_v, K_v, V_v$ are the projections restricted to one view, while $Q, K, V$ are the same projections applied to all of $Z$ ; the weights are shared, only the scope differs.
$u$ indexes pixels, $v \in \{1,\ldots,N\}$ indexes views, $L = (H/p)(W/p)$ is tokens per view for patch size $p$ , and $d$ is the model width.
$d_v(u)$ is the predicted depth and $(o_v, r_v(u))$ the predicted ray map (origin and unit direction) expressed in a shared frame, so $P_v(u)$ is a 3D point in that frame and the camera pose follows from fitting $r_v$ .
$s$ is a single global scale estimated jointly over all views and pixels, replacing the per-image scale and shift used by affine-invariant monocular training.
$C_{\mathrm{within}}$ and $C_{\mathrm{cross}}$ count attention token pairs per layer, so their ratio is $N$ and the KV cache of a cross-view layer grows linearly in $N$ .

Token Budget At 518 Pixels And Patch Size 14:
$L = 37 \times 37 = 1369$
$N L = 32 \times 1369 = 43808$
$C_{\mathrm{within}} = 32 \times 1369^2 \approx 6.0 \times 10^{7}$
$C_{\mathrm{cross}} = 43808^2 \approx 1.92 \times 10^{9}$

Thirty-two views at a modest resolution already put nearly 44k tokens in one sequence, and a single cross-view layer touches about 1.9 billion token pairs against 60 million for a within-view layer. This is why the interleaving ratio is a real design knob rather than a detail: every cross-view layer you insert buys consistency and pays $N$ times the attention cost, and it is why the practical deployment question for any-view geometry models is not accuracy but how many views fit on the device.

Log-scale line chart of attention token pairs per layer versus number of input views from 1 to 64, with a dashed blue line for a within-view layer growing linearly as N times L squared and a solid orange line for a cross-view layer growing quadratically as N L squared, annotated at N equals 32 with 43808 tokens and 1.92 billion versus 60 million pairs, and a note that the two curves coincide at N equals 1

Figure 3: Consistency is not free. A within-view layer scales linearly in view count while a cross-view layer scales quadratically, and the two curves meet at $N = 1$ , which is the formal statement of the input-adaptive property that keeps monocular quality intact.

The target change matters as much as the attention change. A per-image affine-invariant prediction is ambiguous by construction: two frames of the same room can be individually excellent and still disagree by a factor of two in scale, so stitching them produces a doubled wall. Fitting one scale over the whole view set turns depth from a per-image ranking problem into a set-level geometry problem, and the ray map supplies the missing piece by encoding where each pixel’s viewing ray points in the shared frame. Camera intrinsics and extrinsics then fall out of the ray field by a least-squares fit rather than from a separate pose head, which is the concrete sense in which DA3 collapses a multi-task output into a single one.

Property	Depth Anything V1 / V2	VGGT	Depth Anything 3
Input	One image, independently per frame	A set of images in one forward pass	1 to N views, optionally with known poses
Cross-view mechanism	None; consistency is a post-processing problem	Alternating frame-wise and global attention with dedicated camera tokens	Interleaved within-view and cross-view self-attention, no new parameters
Prediction target	Affine-invariant relative disparity	Separate heads for camera, depth, point map, tracking	A single depth-ray target per view
Scale handling	Free scale and shift per image	Set-level, anchored to the first camera	One global scale fitted over the whole view set
Camera pose	Not produced	Predicted by a dedicated camera head	Read out of the predicted ray map
Backbone	DINOv2 ViT with a DPT dense head	ViT with specialised camera and register tokens	Plain DINOv2 ViT, no architectural specialisation
Dominant failure mode	Temporal flicker and misaligned point clouds across frames	Multi-task head interference and heavy memory	Quadratic cost in N; needs genuine overlap between views

August 20, 2026

DL0191 GCN, GraphSAGE, and GAT

How do Graph Convolutional Networks (GCN), GraphSAGE, and Graph Attention Networks (GAT) differ in their neighborhood aggregation strategies, and what are the trade-offs in transductive versus inductive settings?

Answer

All three layers do the same thing at a high level: build a new vector for node $i$ as a weighted combination of its neighbors’ vectors, then apply a linear map and a nonlinearity. The real difference is where the mixing coefficient comes from and how the neighborhood is enumerated. GCN fixes the coefficient from the graph alone as symmetric degree normalization $1/\sqrt{\tilde{d}_i \tilde{d}_j}$ , applied to the full neighborhood in one sparse matrix product over the entire graph. GraphSAGE replaces the full neighborhood with a fixed-size random sample and a permutation-invariant aggregator (mean, max-pool, or LSTM), and keeps the node’s own vector in a separate concatenated slot instead of averaging it away. GAT makes the coefficient content-dependent: a small shared scoring vector reads both endpoint features, a softmax over each node’s neighborhood turns those scores into weights $\alpha_{ij}$ , and several attention heads are run in parallel. Transductive versus inductive is then mostly a property of the training procedure and of what the coefficients depend on, not of the layer equation itself.

(1) GCN Weights Are Structural And Frozen: $c_{ij}$ depends only on the two degrees, so a high-degree hub is deliberately down-weighted and no coefficient is ever learned. This is a strong, cheap prior when features are weak and the graph is homophilous.
(2) GraphSAGE Weights Are Uniform Over A Sample: the mean aggregator gives every sampled neighbor $1/|S(i)|$ , and the sample size is the knob that bounds compute rather than the node’s true degree.
(3) Self Vector Handling Differs: GCN and GAT fold the node into the same sum through a self-loop, while GraphSAGE concatenates it and gives it its own block of the weight matrix, which preserves the node’s own signal at depth.
(4) GAT Weights Are Learned From Features: $\alpha_{ij}$ can vary across edges of identical degree, which is what lets the layer ignore a noisy neighbor. The price is $K|E|$ stored coefficients and gradients flowing through the attention scores.
(5) Transductive Is A Training Choice, Not A Law: the published GCN is trained full-batch on a single normalized adjacency, so a new node changes degrees and the normalization; GraphSAGE was designed minibatch-first so the same weights run on an unseen node by sampling its fan-out.
(6) Depth Costs Differently: full-neighborhood expansion grows as roughly $\bar{d}^{\,L}$ per target node, while sampling caps it at $\prod_l S_l$ at the cost of estimator variance.

Three side-by-side panels showing the same target node with four neighbors A, B, C, D. In the GCN panel the incoming arrow thickness follows fixed degree-based coefficients 0.32, 0.26, 0.18 and 0.15 with a self-loop coefficient 0.20. In the GraphSAGE panel only A and C are sampled with equal weight 0.50, B and D are drawn as dashed gray unsampled nodes, and the target keeps its own vector through concatenation. In the GAT panel the coefficients are learned attention values 0.09, 0.46, 0.11, 0.19 with self attention 0.15, so arrow thickness no longer tracks degree.

Figure 1: One neighborhood, three weighting rules. GCN reads its coefficients off the degrees, so the sparsest neighbor A gets the largest share and the hub D the smallest. GraphSAGE throws away two neighbors and splits the mass uniformly over what survives, keeping the self vector in a separate concatenated slot. GAT lets features decide, so neighbor B dominates with $\alpha = 0.46$ despite having a middling degree.

Mathematical Formulation:
$h_i' = \sigma\big(\textstyle\sum_{j \in \tilde{N}(i)} c_{ij} W h_j\big)$
$c_{ij} = 1 / \sqrt{\tilde{d}_i \tilde{d}_j}$
$a_i = \mathrm{AGG}(\{h_j : j \in S(i)\})$
$h_i' = \sigma(W \, [\, h_i \,\|\, a_i \,])$
$e_{ij} = \mathrm{LeakyReLU}(a^{\top}[W h_i \,\|\, W h_j])$
$\alpha_{ij} = \mathrm{softmax}_j(e_{ij})$
$h_i' = \sigma\big(\textstyle\sum_{j \in \tilde{N}(i)} \alpha_{ij} W h_j\big)$

Where:

$h_i \in \mathbb{R}^{d}$ is the incoming representation of node $i$ and $h_i'$ the layer output; $\sigma$ is the elementwise nonlinearity and $W$ the shared linear map.
$\tilde{N}(i)$ is the neighborhood including the self-loop, and $\tilde{d}_i$ is the corresponding degree, so $c_{ij}$ is the fixed GCN coefficient that never changes during training.
$S(i) \subseteq N(i)$ is the sampled fan-out of fixed size $S_l$ at layer $l$ , and $\mathrm{AGG}$ is a permutation-invariant reducer (mean, elementwise max over an MLP, or LSTM over a shuffled order).
$\|$ is concatenation, so the GraphSAGE weight matrix has shape $d' \times 2d$ and the node’s own features get an independent parameter block.
$a \in \mathbb{R}^{2d'}$ is the shared attention vector and $e_{ij}$ the raw score; the softmax is taken over $j \in \tilde{N}(i)$ so that $\sum_j \alpha_{ij} = 1$ per node.
GAT runs $K$ heads whose outputs are concatenated in hidden layers and averaged in the output layer, multiplying both parameter count and stored coefficients by $K$ .

The cost picture follows directly. A full-batch GCN layer is $O(|E| d + |V| d^2)$ and needs the whole feature matrix plus every layer’s activations resident, which is why it stops fitting long before the graph itself does. GAT adds $O(K |E|)$ scores and their gradients on top of the same sparse pattern, so its memory is edge-bound rather than node-bound. GraphSAGE instead bounds a minibatch: with batch size $B$ and fan-outs $(S_1, \ldots, S_L)$ , the computation graph holds a number of node instances that is completely independent of $|V|$ . That is the property that makes web-scale graph learning practical, and it is the mechanism behind Pinterest’s PinSage, which trains on a graph of billions of nodes by expanding only short sampled neighborhoods per target.

Minibatch Fan-Out Budget:
$N_{mb} = B \prod_{l=1}^{L} S_l$
$25 \times 10 = 250$
$N_{mb} = 512 \times 250 = 128000$

Log-scale line chart of node instances in the computation graph for a single target node versus GNN depth from 1 to 4 layers. Full-neighborhood expansion at average degree 100 rises from 100 to about 101 million, full-neighborhood expansion at average degree 20 rises from 20 to about 168 thousand, and sampled fan-out 25 then 10 per layer rises only from 25 to about 28 thousand. A dashed horizontal line marks a graph of 2.4 million nodes, which the average-degree-100 curve crosses between three and four layers.

Figure 2: Depth is what makes full-neighborhood aggregation intractable. On a graph with average degree 100, a 3-hop receptive field already touches about a million node instances per target and a 4-hop field exceeds the graph size, meaning most of the graph is re-visited for a single prediction. A fixed fan-out of 25 then 10 keeps the same depth at roughly 2,800 instances, trading exactness for a cost that no longer depends on $|V|$ .

Property	GCN	GraphSAGE	GAT
Neighbor weight	Fixed 1 / sqrt(d_i d_j)	Uniform 1 / \|S(i)\|, or max-pool / LSTM reducer	Learned alpha_ij per head, softmax over the neighborhood
Weight depends on	Graph structure only	Sample size only	Endpoint features, so it changes as the model trains
Self representation	Self-loop term inside the same sum	Concatenated, own block of the weight matrix	Self-loop with its own attention coefficient
Published training regime	Full-batch over one normalized adjacency	Minibatch with fixed per-layer fan-out sampling	Full-batch on citation graphs, sampled variants for large graphs
Unseen node at inference	Degrees and normalization must be recomputed; the transductive recipe assumes the test nodes were present during training	Designed for it: sample the fan-out and run one forward pass	Works if the new node has features, since attention is edge-local
Per-layer cost	O(\|E\| d + \|V\| d^2), memory scales with the whole graph	O(B x prod S_l x d) per batch, independent of \|V\|	K x O(\|E\| d), plus K\|E\| stored coefficients and their gradients
Dominant failure mode	Over-smoothing beyond two or three layers; hubs are dampened by construction	Sampling variance and unstable embeddings; fan-out explodes at three or more layers	Edge-bound memory; static attention ranking, which GATv2 fixes

August 20, 2026