Category: Hard

DL0089 RLHF: Human Feedback Training

What is Reinforcement Learning from Human Feedback?

Answer

Reinforcement Learning from Human Feedback (RLHF) is a fine-tuning procedure that optimizes a language model against a learned model of human preferences instead of against a hand-written loss. It exists because the behaviors we actually want (helpful, honest, non-toxic, well-formatted answers) have no differentiable definition: next-token cross-entropy on scraped text rewards imitating the corpus, not satisfying a user. The standard recipe has three stages: supervised fine-tuning on demonstrations, training a reward model on human comparisons of pairs of responses, then optimizing the policy with an RL algorithm such as PPO against that reward, with a KL penalty pinning the policy near its starting point. Humans never grade individual tokens; they rank whole responses, and the reward model is the mechanism that turns those sparse rankings into a dense per-sequence training signal. InstructGPT is the canonical demonstration: labelers preferred outputs from the 1.3B InstructGPT model to those from the 175B pretrained GPT-3 base model despite roughly 100x fewer parameters, so the gain came from the objective rather than from scale.

(1) Three-Stage Recipe: SFT → reward model → RL fine-tuning, where each stage consumes a different kind of human data (demonstrations, then comparisons, then only prompts).
(2) Preferences Are Cheaper Than Demonstrations: asking “which of these two answers is better?” is faster and gives higher inter-annotator agreement than asking someone to write the ideal answer, and the Bradley-Terry model converts those binary comparisons into a scalar reward.
(3) The KL Anchor Is Load-Bearing: the reward being optimized is a learned proxy, so the objective subtracts $\beta$ times the KL divergence to the SFT reference policy; remove it and the policy walks off-distribution into text the reward model scores highly and humans hate.
(4) Reward Hacking Is The Default Failure: proxy reward keeps rising while true human preference peaks and then declines, which is why production pipelines collect fresh comparisons on the current policy’s own samples rather than training once on a frozen dataset.

Three-stage RLHF pipeline diagram: supervised fine-tuning on prompts plus demonstrations produces the reference policy, a reward model is trained on pairwise preference labels with the Bradley-Terry loss, and PPO optimizes the policy on prompts only against the frozen reward model minus a beta-weighted KL term, with a dashed path showing the SFT policy also serving as the KL anchor

Figure 1: The three stages and the data each one needs. Stage 3 requires no new labels, only prompts, because the reward model has absorbed the human judgments, and the SFT checkpoint serves double duty as initialization and as the KL reference.

The reward model is a copy of the transformer with the language-modeling head replaced by a scalar head, trained so that the preferred response scores higher than the rejected one. Because only differences of rewards appear in the loss, the reward scale and offset are unidentifiable, which is why RLHF implementations whiten or normalize rewards per batch before computing advantages. In stage 3 the policy samples completions for a prompt, the frozen reward model scores each one, and PPO takes a clipped policy-gradient step; the KL term is usually implemented as a per-token penalty folded into the reward rather than as a hard constraint. A naive PPO setup keeps four networks resident (policy, reference, reward model, value head), so memory and orchestration cost is the practical reason many teams reach for a direct preference method instead.

Mathematical Formulation:
$\mathcal{L}_{RM} = -\log \sigma(r_\phi(x,y_w) - r_\phi(x,y_l))$
$R(x,y) = r_\phi(x,y) - \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)}$
$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta}[R(x,y)]$

Where:

$\mathcal{L}_{RM}$ is the Bradley-Terry pairwise loss and $\sigma$ is the logistic function, so the reward model is trained as a binary classifier over response pairs.
$x$ is a prompt from the dataset $\mathcal{D}$ , and $y_w$ and $y_l$ are the human-preferred and rejected responses to it.
$r_\phi$ is the learned scalar reward model with parameters $\phi$ ; only reward differences are identified, so its absolute scale is arbitrary.
$\pi_\theta$ is the policy being optimized and $\pi_{\mathrm{ref}}$ is the frozen SFT policy; $\beta > 0$ sets how far the policy may drift, with typical values near $0.01$ to $0.1$ .
$R(x,y)$ is the KL-shaped reward actually fed to PPO, and $J(\theta)$ is the on-policy objective, with the expectation taken over responses sampled from the current policy.

Chart of reward versus KL divergence from the reference policy: the proxy reward model score rises monotonically with KL while the gold human preference score rises, peaks around 12 nats, and then declines, with a marker at the peak

Figure 2: Illustrative overoptimization curve: the reward model score climbs without bound as the policy drifts, while measured human preference peaks and falls. $\beta$ and early stopping exist to keep training left of that peak.

Aspect	PPO-based RLHF	DPO	GRPO
Separate reward model	Yes, trained and frozen	No, the policy is its own implicit reward	Yes, or a programmatic verifier
Sampling	On-policy generation every step	Offline, fixed preference pairs	On-policy groups of responses per prompt
Networks in memory	Four: policy, reference, reward, value	Two: policy and reference	Three: no value network, group mean is the baseline
Main failure mode	Reward hacking plus brittle PPO tuning	Stale data, drifts off the current policy distribution	Noisy advantages when a whole group scores alike
Best fit	Large budget, fresh labels, general helpfulness	Fixed preference dataset, limited compute	Reasoning tasks with checkable answers

August 15, 2026

DL0087 QLoRA: 4-bit Quantized LoRA

What is QLoRA, and how does it combine 4-bit quantization with LoRA?

Answer

QLoRA (Dettmers et al., 2023) is a fine-tuning recipe that keeps the entire base model frozen in 4-bit precision and trains only LoRA adapters in 16-bit on top of it, so the memory that would have held gradients and optimizer state for billions of parameters simply never gets allocated. During the forward pass each 4-bit weight block is dequantized to bf16 on the fly, used for the matrix multiply, and discarded; the backward pass dequantizes again to propagate gradients through the frozen weights into the adapters, which are the only tensors that receive updates. Three components make the 4-bit base workable: the NF4 data type, which places its 16 levels at quantiles of a normal distribution rather than uniformly, double quantization of the per-block scaling constants, and paged optimizers that offload optimizer state through unified memory when a long-sequence step spikes. The headline result was fine-tuning a 65B model on a single 48 GB GPU while matching 16-bit LoRA quality on instruction-following benchmarks. Note that QLoRA quantizes the frozen base, never the adapters or the gradients, so all learning still happens in bf16.

(1) NF4 (4-bit NormalFloat): pretrained weights within a block are approximately zero-centered Gaussian, so NF4 spaces its levels at quantiles of a standard normal instead of uniformly, which is information-theoretically optimal for that assumption and beats 4-bit float or int at equal bit width.
(2) Blockwise Absmax Scaling: weights are quantized in blocks of 64 with one absmax constant per block, which localizes outliers so a single large weight cannot crush the resolution of an entire tensor.
(3) Double Quantization: those constants are themselves quantized to 8-bit with a second-level fp32 scale per 256 constants, cutting the metadata overhead from 0.5 to about 0.127 bits per parameter (roughly 3 GB on a 65B model).
(4) Only Adapters Train: gradients pass through the frozen 4-bit weights but are stored only for $A$ and $B$ , so optimizer state scales with the adapter rank, not with model size.
(5) Paged Optimizers: optimizer states live in NVIDIA unified memory and are paged to host RAM during transient spikes, which is what keeps a 33B or 65B single-GPU run from OOM-ing on a long batch.

Block diagram of a QLoRA linear layer: input X feeds a frozen NF4 base weight with its double-quantized absmax constants, which is dequantized to bf16 for the matmul, while a parallel bf16 LoRA A and B path scaled by alpha over r is added to produce output Y, with dashed gradient arrows returning only into A and B

Figure 1: One QLoRA linear layer. The NF4 weight and its quantization constants are read-only, the bf16 copy exists only for the duration of the matmul, and the dashed path shows that gradients terminate at the low-rank adapters.

Mathematical Formulation:
$Y = X\,\mathrm{dq}(W_4) + \frac{\alpha}{r}\,X A B$
$\mathrm{dq}(W_4)_{ij} = c_i\,z_{q_{ij}}$
$c_i = \mathrm{absmax}(W_i)$
$b = 4 + \frac{8}{64} + \frac{32}{64 \cdot 256}$
$b \approx 4.127\ \text{bits per parameter}$

Where:

$Y$ is the layer output and $X$ the bf16 input activation; both stay in 16-bit throughout.
$W_4$ is the frozen base weight stored as NF4 indices, and $\mathrm{dq}$ is the dequantization that reconstructs a bf16 tile just before the matmul.
$i$ indexes the blocks of 64 weights, $q_{ij} \in \{0,\ldots,15\}$ is the stored 4-bit code, and $z_k$ are the fixed NF4 levels, obtained from normal quantiles and rescaled to $[-1, 1]$ with an exact zero.
$c_i$ is the per-block absmax constant; double quantization stores it in 8 bits with one fp32 scale per 256 constants.
$A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times k}$ are the trainable bf16 adapters of rank $r$ (the paper uses $r = 64$ on every linear layer), scaled by $\alpha / r$ ; $B$ starts at zero so the layer initially reproduces the quantized base model.
$b$ is the effective storage cost per base parameter: 4 bits of payload plus 0.125 bits of 8-bit constants plus 0.002 bits of second-level scales.

Horizontal stacked bar chart of training memory for a 7B model: full fine-tuning totals 112 GB with 14 GB weights, 14 GB gradients and 84 GB optimizer states; 16-bit LoRA totals 16.6 GB; QLoRA totals 6.2 GB with a 3.6 GB NF4 base

Figure 2: Illustrative state memory for a 7B model with rank-64 adapters on all linear layers. LoRA removes the optimizer and gradient bulk, and QLoRA then shrinks the remaining frozen weights from 14 GB to 3.6 GB; activation memory is excluded.

Aspect	QLoRA (NF4 base)	16-bit LoRA	Full Fine-Tuning
Base weight storage	NF4, about 4.13 bits per parameter with double quantization	bf16, 16 bits per parameter	bf16 weights plus an fp32 master copy
Trainable parameters (7B)	About 160M in bf16 at rank 64 on all linear layers	Identical adapter count	All 7B parameters
State memory (7B)	About 6 GB	About 17 GB	About 112 GB, needs sharding across GPUs
Step time	Slowest: dequantization runs in both forward and backward	Fastest of the two adapter methods	Highest total compute and communication
Instruction-tuning quality	Matches 16-bit LoRA when adapters cover every linear layer	Reference point for adapter tuning	Preferred with very large in-domain corpora or heavy domain shift
Deployment	Adapter cannot fold into NF4 losslessly; merge into the 16-bit base, then requantize	Merge into bf16 weights for zero added latency	One merged checkpoint per task

August 15, 2026

DL0079 Scaling Laws Size vs Data

How do scaling laws relate model size to training data requirements?

Answer

Neural scaling laws say that test loss falls as a power law in both parameter count $N$ and training tokens $D$ , plus an irreducible floor, so the two are not independent knobs: a fixed compute budget $C \approx 6ND$ forces a trade between them. Kaplan et al. (2020) fit the loss surface and concluded that most extra compute should go into parameters, which is why GPT-3 used 175B parameters on only about 300B tokens (roughly 1.7 tokens per parameter). Hoffmann et al. (2022, “Chinchilla”) re-ran the sweep with properly decayed learning-rate schedules over more than 400 runs and found the compute-optimal frontier scales both quantities at roughly the same rate, $N_{opt} \propto C^{0.5}$ and $D_{opt} \propto C^{0.5}$ , giving the well-known heuristic of about 20 tokens per parameter. Under that law a 70B model trained on 1.4T tokens beat a 280B model trained on 300B tokens at equal compute. The practical answer to the interview question is therefore: data requirements grow roughly linearly with model size along the compute-optimal frontier, and any deviation from that ratio should be justified by a constraint the loss law does not model, such as a finite corpus or the cost of serving.

(1) Joint Power Law: loss decomposes into an entropy floor plus one shrinking term per resource, $L = E + AN^{-\alpha} + BD^{-\beta}$ , so starving either resource leaves its term dominant and no amount of the other resource can fix it.
(2) Compute Couples The Two: with $C \approx 6ND$ FLOPs (2 for the forward pass and 4 for the backward pass per parameter per token), choosing $N$ fixes $D$ , which is why the question is always “what split”, never “how big”.
(3) IsoFLOP Minima Define The Frontier: sweeping $N$ at fixed $C$ traces a shallow U-shaped curve; the locus of minima across budgets is the compute-optimal frontier, and its slope in log-log space is the exponent you actually care about.
(4) Kaplan Versus Chinchilla: the earlier exponents ( $N_{opt} \propto C^{0.73}$ ) came from a fixed-length cosine schedule truncated early and from parameter counts that excluded embeddings, both of which penalize the long-token runs and bias the fit toward oversized models.
(5) Optimal Does Not Mean Correct: the law optimizes training loss per training FLOP; it says nothing about inference cost, so serving-heavy models are deliberately overtrained far past 20 tokens per parameter.

The empirical procedure behind these numbers is worth stating precisely, because interviewers probe it. You pick a grid of compute budgets, and for each budget you train several models of different widths and depths, always to the token count that exhausts exactly that budget, with the learning rate decayed to its minimum at the end of each run. Plotting final loss against $N$ for one budget yields an IsoFLOP curve whose minimum is flat enough that a factor of two in parameter count often costs under 1% in loss, which is exactly the slack engineers exploit when memory or latency constraints bite.

Four U-shaped IsoFLOP curves of loss versus parameter count on a log x-axis for compute budgets of 1e19, 1e20, 1e21 and 1e22 FLOPs, with minima marked and joined by a dashed frontier line moving right and down

Figure 1: Illustrative IsoFLOP curves from $L = E + AN^{-0.34} + BD^{-0.34}$ with $C = 6ND$ . Each budget’s minimum sits at $D/N \approx 20$ , moving from 0.29B parameters / 5.8B tokens at $10^{19}$ FLOPs to 9.1B parameters / 183B tokens at $10^{22}$ FLOPs, and the basins are shallow near the optimum.

Mathematical Formulation:
$L(N,D) = E + A N^{-\alpha} + B D^{-\beta}$
$C \approx 6ND$
$N_{opt} \propto C^{a}$
$D_{opt} \propto C^{b}$
$a = \frac{\beta}{\alpha + \beta}$
$b = \frac{\alpha}{\alpha + \beta}$
$D_{opt} / N_{opt} \approx 20$

Where:

$L(N,D)$ is expected test loss in nats per token, and $E$ is the irreducible term: the entropy of the data plus whatever the architecture can never represent, so no scaling drives loss to zero.
$N$ is the number of trainable parameters (embeddings included, which is where the Kaplan fit differed) and $D$ is the number of training tokens.
$A, B$ are fitted scale constants and $\alpha, \beta$ are the power-law exponents; Chinchilla’s fit put both near $0.3$ , meaning the two resources have comparable marginal value.
$C$ is training compute in FLOPs; the factor 6 comes from roughly 2 FLOPs per parameter per token forward and 4 backward, and it ignores attention’s quadratic term, which is minor while $d_{model}$ dominates sequence length.
$a, b$ are the frontier exponents obtained by minimizing $L$ subject to the compute constraint; $a + b = 1$ always holds, so if parameters take a larger share of compute growth, tokens must take a smaller one.
The ratio $D_{opt}/N_{opt}$ is constant only when $\alpha = \beta$ ; the “20 tokens per parameter” rule is that special case, and it drifts slowly with budget under asymmetric fits.

Log-log plot of compute-optimal parameter count and token count versus training compute from 1e19 to 1e25 FLOPs, with Chinchilla curves scaling as C to the 0.5 for both quantities and Kaplan curves scaling as C to the 0.73 for parameters and C to the 0.27 for tokens, diverging by more than an order of magnitude at high compute

Figure 2: Illustrative comparison of the two prescriptions, normalized to agree at $10^{19}$ FLOPs. Because the exponents differ ( $0.5$ versus $0.73$ ), the gap compounds: at $10^{24}$ FLOPs one recipe asks for 91B parameters on 1.8T tokens and the other for 1.3T parameters on 130B tokens, about a $14\times$ disagreement in both directions.

Modern frontier practice has moved past the compute-optimal point on purpose. If a model will serve billions of tokens, total lifetime FLOPs are dominated by inference, and a smaller model trained on far more data is cheaper end to end even though its training run is nominally suboptimal. Meta’s Llama 3 8B was trained on roughly 15T tokens, near 1,875 tokens per parameter, almost two orders of magnitude past the Chinchilla ratio, and the authors report the loss was still improving log-linearly at that point. The scaling law is still doing the work here; the objective being minimized has simply changed from training loss per training FLOP to quality per unit of deployment cost.

Regime	Tokens Per Parameter	What It Optimizes	When It Is The Right Call
Kaplan-style (parameter-heavy)	About 1 to 2	Loss under a truncated schedule and an embedding-free parameter count	Essentially never today; understand it to explain why GPT-3 looked the way it did
Chinchilla-optimal	About 20	Best loss for a fixed training budget	Research runs, ablations, and any model whose main cost is the training job
Deliberately overtrained	150 to 2,000 or more	Quality per unit of training plus inference cost	High-traffic serving, on-device targets, fixed memory or latency budgets
Data-constrained	Set by the corpus, not by compute	Loss given a finite unique-token pool with repeated epochs	Specialist domains (code, medical, low-resource languages) where fresh tokens run out

August 15, 2026

DL0075 Attention Sink

What is an attention sink, and why do models attend to the first token so heavily?

Answer

An attention sink is a token that a head dumps its attention mass onto when it has nothing relevant to retrieve, and in decoder-only LLMs that token is almost always the first one in the sequence (often the BOS token). Measured on a trained model, a middle-layer head can place well over half of its probability on that single position even though its value vector carries almost no task-relevant content. The cause is structural rather than semantic: softmax normalizes every attention row to sum to exactly 1, so a head has no way to say “attend to nothing” and must put the leftover mass somewhere. Causal masking makes the first token the only key visible from every query position, and it is the one token whose representation is not yet mixed with context, so training converges on it as the default no-op target. The model then reinforces the arrangement by growing massive activations and unusually large key norms at that position, which is why the sink shows up as an outlier in both attention maps and activation histograms.

(1) Softmax Has No No-Op: the row constraint $\sum_j \alpha_{ij} = 1$ means an idle head still emits a full distribution, so the mass must land on some low-cost token.
(2) Why The First Token Wins: under causal masking it is the only key every query can see, and it precedes all content, so it is both universally available and information-free, the ideal dumping ground.
(3) The Sink Is Load-Bearing: it is not decorative; drop the first token from a sliding-window KV cache and perplexity jumps by one to two orders of magnitude, because every head loses its default target and redistributes mass onto real tokens.
(4) Fingerprints Beyond Attention: the same position carries hidden-state outliers that widen activation ranges, which is a known headache for per-tensor quantization; vision transformers show the analogous effect by repurposing background patches as sinks.

Log-scale bar chart of attention mass by key position for one head at query position 512: token 0 receives 0.42, tokens 1 to 3 receive 0.06, 0.03 and 0.02, three middle bins receive 0.07, 0.06 and 0.09, and the most recent 64 positions receive 0.25

Figure 1: Illustrative single head at query position 512: the first token absorbs 0.42 of the mass, the most recent 64 positions share 0.25, and the roughly 444 middle positions split only 0.22, about $5\times 10^{-4}$ each.

The cleanest way to read the sink is as a learned bias term smuggled into the softmax. Because the denominator sums over all visible keys, the only way for a head to shrink the weight it gives to real tokens is to inflate the score of some other key; the first token, whose value vector contributes near-zero to the output, is that escape valve. This also explains the standard fixes. Giving the softmax an explicit extra term in the denominator (the off-by-one softmax, or a learned per-head sink logit as used in some recent open-weight models such as OpenAI’s gpt-oss) lets attention rows sum to less than 1, so no real token needs to be sacrificed. Prepending dedicated register tokens achieves the same thing with ordinary softmax by supplying a scratch position, which is what cleaned up the noisy attention maps in vision transformers. Both changes must be present during pretraining: retrofitting them onto a model that already routes mass through token 0 changes the normalization the weights were trained under.

Mathematical Formulation:
$s_{ij} = \frac{q_i^{\top} k_j}{\sqrt{d_k}}$
$\alpha_{ij} = \frac{\exp(s_{ij})}{\sum_{m=1}^{i} \exp(s_{im})}$
$\sum_{j=1}^{i} \alpha_{ij} = 1$
$\tilde{\alpha}_{ij} = \frac{\exp(s_{ij})}{\exp(b_h) + \sum_{m=1}^{i} \exp(s_{im})}$

Where:

$s_{ij}$ is the scaled dot-product score between query $q_i$ and key $k_j$ , with head dimension $d_k$ .
$\alpha_{ij}$ is the attention weight; $i$ indexes query positions and $j, m$ index key positions, restricted to $m \leq i$ by causal masking, which is why $j = 1$ is the only key shared by every query.
The third line is the normalization constraint that creates the sink: no configuration of scores lets a head emit less than one unit of total mass.
$b_h$ is a learned per-head sink logit added to the denominator, so $\sum_j \tilde{\alpha}_{ij}$ can fall below 1; the off-by-one softmax is the special case $b_h = 0$ , since $\exp(0) = 1$ .

Log-scale perplexity versus tokens streamed for three attention policies: dense attention rises past the 4096-token pretraining length, sliding-window attention jumps from about 10 to nearly 400 once the first token is evicted at 1024 tokens, and a window that retains 4 sink tokens stays flat near 10.5

Figure 2: Illustrative streaming behavior: a plain sliding window collapses the moment the first token leaves the KV cache, while retaining a handful of sink tokens alongside the window keeps perplexity flat far beyond the pretraining length.

Approach	What It Changes	Training Cost	When to Reach for It
Dense Attention	Nothing; the sink stays at token 0 and is never evicted	None	Short contexts where the full KV cache fits
Keep First k Tokens (StreamingLLM)	Cache policy only: pin the first 4 tokens plus a sliding window	None, works on an already-trained model	Streaming or long-running serving with a fixed cache budget
Register or Sink Tokens	Prepends learnable scratch tokens that heads can dump onto	Pretraining or substantial fine-tuning	New models, and ViTs whose attention maps must stay interpretable
Learned Sink Logit / Softmax Off-by-One	Extra denominator term so a row can sum to less than 1	Must be present from pretraining	New architectures, especially where activation outliers hurt quantization

August 14, 2026

DL0073 EfficientNet Compound Scaling
How does EfficientNet scale networks, and what is compound scaling?
Answer
EfficientNet starts from a small baseline, EfficientNet-B0, produced by a multi-objective architecture search that rewards accuracy and FLOPs together, and then grows that fixed topology into the family B1 through B7 by scaling three dimensions simultaneously: depth (number of layers), width (number of channels), and input resolution. The empirical observation behind the method is that scaling any single dimension saturates: beyond a point, extra layers, extra channels, or extra pixels buy almost no accuracy while still costing compute. Compound scaling ties the three together through one user-chosen coefficient $\phi$ and three fixed exponents $\alpha, \beta, \gamma$ obtained by a small grid search on B0, subject to $\alpha \cdot \beta^{2} \cdot \gamma^{2} \approx 2$ so that each unit of $\phi$ roughly doubles the FLOPs budget. With $\alpha = 1.2$ , $\beta = 1.1$ , $\gamma = 1.15$ , the family climbs from 77.1% ImageNet top-1 at 0.39B FLOPs (B0) to 84.3% at 37B FLOPs (B7), matching the best accuracy of its era with about 8.4x fewer parameters than GPipe.
(1) The Baseline Is a Prerequisite: compound scaling only multiplies an existing topology, so a weak baseline yields a weak family; B0 is itself a search result built from MBConv blocks with squeeze-and-excitation, and the same scaling rule applied to MobileNet or ResNet gives smaller gains.
(2) Single-Dimension Scaling Saturates: very deep networks hit optimization and degradation limits, very wide shallow networks capture fine-grained patterns but few high-level ones, and resolution alone raises cost quadratically for shrinking returns.
(3) The Balance Rule: a larger input needs more layers to grow the receptive field and more channels to encode the finer patterns those extra pixels expose, which is why the three factors should move in a fixed ratio rather than one at a time.
(4) Two-Step Search: fix $\phi = 1$ and grid-search $\alpha, \beta, \gamma$ once on the cheap baseline, then freeze them and sweep $\phi$ to get B1 through B7, which avoids re-searching the architecture at every model size.

Figure 1: Illustrative accuracy-versus-compute curves from the same B0 baseline: depth-only, width-only, and resolution-only scaling flatten near 80% top-1, while compound scaling keeps converting FLOPs into accuracy.
The constraint has a direct cost interpretation. A standard convolution’s FLOPs scale linearly with the number of layers and quadratically with both channel count and spatial size, so total compute grows like $d \cdot w^{2} \cdot r^{2}$ . Forcing $\alpha \cdot \beta^{2} \cdot \gamma^{2} \approx 2$ therefore makes $\phi$ a clean compute dial: each additional unit costs about 2x the FLOPs, and the exponents decide how that doubled budget is split across the three dimensions. The exponents are searched once, on a model cheap enough that a small grid over $\alpha, \beta, \gamma$ is affordable.
Mathematical Formulation:
$d = \alpha^{\phi}$
$w = \beta^{\phi}$
$r = \gamma^{\phi}$
$\alpha \cdot \beta^{2} \cdot \gamma^{2} \approx 2$
$\mathrm{FLOPs}(\phi) \approx 2^{\phi} \cdot \mathrm{FLOPs}(0)$
Where:
$d$ , $w$ , and $r$ are the multipliers applied to the baseline’s layer count per stage, channel count per layer, and input side length.
$\phi$ is the user-chosen compound coefficient that sets the resource budget; $\phi = 0$ recovers the baseline B0.
$\alpha, \beta, \gamma$ are constants from a small grid search on B0 with $\alpha \geq 1$ , $\beta \geq 1$ , $\gamma \geq 1$ ; the published values are $1.2$ , $1.1$ , and $1.15$ .
Convolution cost scales as $d \cdot w^{2} \cdot r^{2}$ , so the product constraint is what turns $\phi$ into an approximate doubling of FLOPs per unit.

Figure 2: The same topology at two budgets: a bigger input square (resolution), taller blocks (width), and more blocks (depth) all grow together instead of one dimension racing ahead.
Model Depth Width Resolution FLOPs ImageNet Top-1
B0 1.0x 1.0x 224 0.39B 77.1%
B3 1.4x 1.2x 300 1.8B 81.6%
B5 2.2x 1.6x 456 9.9B 83.6%
B7 3.1x 2.0x 600 37B 84.3%
Two caveats matter in practice. The released coefficients are rounded rather than exact powers of a single $\phi$ , so treat the formula as the design principle and the published table as the shipped configuration. More importantly, the objective is FLOPs, not latency or memory: depthwise separable convolutions have low arithmetic intensity and underuse GPU and TPU matrix units, and activation memory grows with $r^{2}$ , so the largest variants train slowly and can exhaust device memory. EfficientNetV2 addressed exactly this by replacing early MBConv stages with Fused-MBConv, capping the maximum image size, and adding training-aware search plus progressive resizing.
Login to view more content
August 14, 2026

Model	Depth	Width	Resolution	FLOPs	ImageNet Top-1
B0	1.0x	1.0x	224	0.39B	77.1%
B3	1.4x	1.2x	300	1.8B	81.6%
B5	2.2x	1.6x	456	9.9B	83.6%
B7	3.1x	2.0x	600	37B	84.3%

DL0070 Multi-Task Loss Balancing

How would you design a loss for multi-task learning when the tasks have very different scales?

Answer

Start from the failure mode: with a naive sum $L = \sum_i L_i$ , each task contributes gradient in proportion to its loss scale, so a depth-regression MSE sitting near 200 drowns out a segmentation cross-entropy sitting near 1, and the shared trunk optimizes the loud task while the quiet ones stall. The fix comes in three tiers. First, put the losses on a common scale by dividing each by a fixed or running estimate of its magnitude, such as its initial value. Second, learn the weights: Kendall et al.’s uncertainty weighting attaches a trainable $\sigma_i$ per task and minimizes $\sum_i L_i / (2\sigma_i^2) + \log \sigma_i$ , derived from Gaussian and categorical likelihoods, so noisy tasks are down-weighted automatically while the $\log \sigma_i$ term stops $\sigma_i$ from blowing up. Third, when scales are balanced but gradient directions still conflict, move to gradient-level methods: GradNorm tunes the task weights so each task’s gradient norm on the shared trunk approaches a common scale adjusted by its relative inverse training rate, so tasks that are learning slowly get pushed harder, and PCGrad projects away mutually conflicting components.

(1) Scale Equals Loudness: task $i$ ‘s gradient share scales with the units of $L_i$ , so the unweighted sum is an implicit weighting set by arbitrary unit choices; expressing depth in millimeters instead of meters multiplies its MSE by $10^6$ and hands it the entire gradient budget.
(2) Uncertainty Weighting: treat $\sigma_i$ as task $i$ ‘s observation noise; the weight $1/(2\sigma_i^2)$ falls as noise grows while the $\log \sigma_i$ penalty (the likelihood’s normalizing constant) rises, so the optimum is a genuine trade-off learned by gradient descent alongside the network weights.
(3) Magnitude Is Not Direction: scale balancing fixes how loudly tasks speak, not whether they agree; when task gradients point in opposing directions (negative transfer), gradient-space methods such as GradNorm or PCGrad are the right lever.

Grouped bar chart on a log scale of illustrative gradient norms on the shared trunk for segmentation, depth, and normals tasks: unweighted bars are 1.1, 160, and 0.8, while uncertainty-weighted bars are 0.9, 1.0, and 0.7

Figure 1: Illustrative three-task trunk, gradient norms on a log scale: unweighted, the depth MSE contributes over 100x the others; once each loss carries a learned $1/(2\sigma_i^2)$ weight, the three contribute comparably.

The uncertainty objective is not ad hoc. Modeling regression noise as $p(y \mid f(x)) = \mathcal{N}(f(x), \sigma^2)$ gives the negative log-likelihood $\|y - f(x)\|^2 / (2\sigma^2) + \log \sigma$ per task, up to constants; classification slots in through a scaled softmax likelihood whose approximation yields the analogous weight $1/\sigma^2$ (without the factor 2) alongside the same $\log \sigma$ penalty. Two properties matter in practice: $\sigma_i$ is learned by the same optimizer as the network, so balancing needs no manual grid search; and the $\log \sigma_i$ term keeps the objective honest, because without it every $\sigma_i$ would grow without bound, all task weights would collapse to zero, and nothing would be learned. Most implementations optimize $s_i = \log \sigma_i^2$ and compute $e^{-s_i} L_i + s_i$ for numerical stability.

Two panels against task uncertainty sigma: top shows the task weight 1/(2 sigma^2) collapsing on a log scale as sigma grows from 0.25 to 4; bottom shows the penalty log sigma rising over the same range

Figure 2: The learned trade-off: as a task’s noise $\sigma$ grows, its weight $1/(2\sigma^2)$ collapses while the penalty $\log \sigma$ rises, so the optimizer cannot silence a noisy task for free.

Mathematical Formulation:
$L_{\mathrm{naive}} = \sum_{i=1}^{T} L_i$
$L_{\mathrm{total}} = \sum_{i=1}^{T} \left( \frac{1}{2\sigma_i^2}\, L_i + \log \sigma_i \right)$

Where:

$L_i$ is the loss of task $i$ (cross-entropy for segmentation, MSE for depth) and $T$ the number of tasks sharing the trunk.
$\sigma_i$ is a learned per-task scalar modeling homoscedastic (task-level, input-independent) observation noise, initialized at 1 and trained by the same optimizer as the weights.
$1/(2\sigma_i^2)$ is the derived task weight and $\log \sigma_i$ the likelihood’s normalizing term that penalizes inflating $\sigma_i$ ; in code, parameterize $s_i = \log \sigma_i^2$ and optimize $e^{-s_i} L_i + s_i$ instead.

Method	What It Balances	Extra Cost	When to Reach for It
Fixed Weights (Grid Search)	Loss scales, set by hand	Search cost grows fast with task count	Two tasks and plenty of compute
Loss Normalization	Loss magnitudes via running scale estimates	Negligible	Quick baseline before anything fancier
Uncertainty Weighting (Kendall)	Loss weights via learned sigma per task	One extra scalar per task	Default starting point for shared-trunk training
GradNorm	Gradient norms toward a common scale by training rate	Extra backward bookkeeping each step	Tasks learning at very different speeds
PCGrad	Conflicting gradient directions	Per-task gradients every step	Negative transfer between tasks

August 12, 2026

DL0067 BERT VS GPT Pretraining
How does BERT’s pre-training objective differ from GPT’s?
Answer
BERT is trained primarily with masked language modeling: selected input tokens are corrupted, and a bidirectional encoder predicts the original tokens from visible context on both sides. GPT uses causal language modeling: a decoder predicts each next token using only earlier tokens, enforced by a triangular attention mask. MLM supplies loss only at selected positions and creates a corruption mismatch between pre-training and ordinary inputs, whereas causal modeling supplies a target at nearly every position and matches left-to-right generation. Original BERT also used next sentence prediction, while standard GPT pre-training relies on the autoregressive token objective rather than NSP.
(1) Context Visibility: BERT’s visible tokens attend bidirectionally; GPT position $t$ cannot attend to positions later than $t$ .
(2) Prediction Targets: BERT reconstructs a selected masked subset, while GPT shifts the sequence and predicts the next token at each usable position.
(3) Downstream Bias: BERT naturally supports full-context understanding, whereas GPT’s objective directly trains open-ended autoregressive generation; either family can be adapted beyond that bias.

Figure 1: BERT reconstructs selected corrupted tokens from two-sided context, while GPT predicts the next token at every step from a left-only prefix.
Mathematical Formulation:
$\mathcal{L}_{\mathrm{BERT}}=-\sum_{i\in\mathcal{M}}\log p(x_i\mid x_{\setminus\mathcal{M}})$
$\mathcal{L}_{\mathrm{GPT}}=-\sum_{t=1}^{T}\log p(x_t\mid x_{1:t-1})$
Where:
$\mathcal{L}_{\mathrm{BERT}}$ and $\mathcal{L}_{\mathrm{GPT}}$ are the masked and causal language-modeling losses.
$\mathcal{M}$ is BERT’s selected target set and $i\in\mathcal{M}$ indexes one target token $x_i$ .
$x_{\setminus\mathcal{M}}$ denotes BERT’s visible corrupted context outside the selected targets, and $p(x_i\mid x_{\setminus\mathcal{M}})$ is the reconstruction probability.
$t\in\{1,\ldots,T\}$ indexes a GPT target position, $T$ is sequence length, and $x_{1:t-1}$ is the preceding token prefix.
$p(x_t\mid x_{1:t-1})$ is the next-token probability and $\log$ converts each probability into a log-likelihood term.

Figure 2: Attention visibility explains the objective difference: BERT uses a full visible-context matrix, while GPT uses a lower-triangular causal matrix.
Login to view more content
July 20, 2026
DL0056 FlashAttention
Explain FlashAttention. Why can it compute exact attention faster while using less memory?
Answer
FlashAttention is an exact, IO-aware implementation of scaled dot-product attention. Instead of materializing the full $N\times N$ score and probability matrices in high-bandwidth memory (HBM), it loads blocks of $Q$ , $K$ , and $V$ into fast on-chip SRAM, computes attention block by block, and maintains online softmax statistics. Tiling reduces expensive HBM reads and writes, while recomputation during the backward pass can be cheaper than storing large intermediates. The mathematical result matches standard attention up to normal floating-point differences; the speedup comes from changing the execution schedule, not from approximating attention.
(1) IO Awareness: The kernel is organized around the GPU memory hierarchy so most score computation and softmax updates happen in SRAM.
(2) Online Softmax: A running row maximum and normalization sum allow each $K,V$ tile to update the output without retaining previous score blocks.
(3) Exact Result: All query-key interactions are evaluated; causal masking, dropout, and backward gradients are fused into specialized kernels rather than approximated.

Figure 1: IO comparison showing why avoiding N×N intermediate writes makes FlashAttention faster and more memory efficient.
Mathematical Formulation:
$m_i^{(t)}=\max\!\left(m_i^{(t-1)},\max_j S_{ij}^{(t)}\right)$
$\ell_i^{(t)}=e^{m_i^{(t-1)}-m_i^{(t)}}\ell_i^{(t-1)}+\sum_j e^{S_{ij}^{(t)}-m_i^{(t)}}$
Where:
$t$ indexes key/value tiles, $i$ indexes a query row, and $j$ indexes keys inside tile $t$ .
$S_{ij}^{(t)}=Q_iK_j^T/\sqrt d$ is the scaled score between query row $Q_i$ and key row $K_j$ , with head width $d$ .
$m_i^{(t)}$ is the running row maximum after tile $t$ , initialized with $m_i^{(0)}=-\infty$ .
$\ell_i^{(t)}$ is the running softmax denominator, initialized with $\ell_i^{(0)}=0$ ; the exponential factors rescale earlier partial sums when the maximum changes.

Figure 2: Blockwise FlashAttention loop with running maximum, normalization, output rescaling, and final HBM write.
Login to view more content
July 20, 2026
DL0039 Transformer Weight Tying
Explain weight sharing in Transformers.
Answer
Weight sharing (weight tying) reuses the same parameters in multiple places instead of learning separate matrices. The most common form ties the input embedding matrix with the output projection that produces vocabulary logits: one matrix $E$ maps token IDs into $d$ -dimensional space at the bottom and maps hidden states back to logits at the top. A stronger form (ALBERT) shares weights across all Transformer layers, trading capacity for extreme parameter efficiency.
(1) Embedding–Output Tying: The output logits are computed with the transpose of the input embedding, $z = Eh$ , saving an entire $K \times d$ matrix.
(2) Consistency Benefit: Input and output spaces share one geometry: tokens that embed close together also get similar output preferences, which acts as a useful regularizer.
(3) Layer Sharing (ALBERT): One block’s weights are reused for every layer: depth comes from recurrence through the same block, not from new parameters.
Mathematical Formulation:
$\mathrm{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$
$z = E\, h$
Where:
$E \in \mathbb{R}^{K \times d}$ is the shared embedding matrix, used to look up inputs and, transposed, to score outputs.
$h \in \mathbb{R}^{d}$ is the final hidden state; $z_i = (Eh)_i$ is the logit of vocabulary token $i$ ; $K$ is vocabulary size.

Figure 1: One matrix, two jobs: $E$ embeds input tokens at the bottom and, transposed, projects hidden states to logits at the top.
How Much It Saves: The untied output projection is a full $K \times d$ matrix; for a 50k vocabulary at $d = 1024$ that is ~51M parameters, often the single largest matrix in a small model. Tying removes it entirely while typically costing little or no accuracy.

Figure 2: Tying deletes the output projection matrix, one of the largest single blocks of parameters in vocabulary-heavy models.
Vocabulary Mismatch Caveat: Tying assumes input and output share a vocabulary. When source and target vocabularies differ (multilingual translation), only the decoder input embedding and output projection can be tied; the encoder embedding stays separate.
Login to view more content
August 28, 2025
DL0033 Transformer Computation
In a Transformer architecture, which components are the primary contributors to computational cost, and why?
Answer
It depends on sequence length. For short sequences, the feed-forward network (FFN) usually dominates: its two wide GEMMs cost $O(n d^2)$ while attention’s quadratic term is still small. For long sequences, multi-head attention takes over: forming the $n \times n$ score matrix costs $O(n^2 d)$ and grows much faster than anything else in the block.
(1) Multi-Head Attention: Q/K/V projections cost $O(nd^2)$ , but the score matrix $QK^\top$ and its product with V cost $O(n^2 d)$ , the quadratic term that explodes on long sequences.
(2) Feed-Forward Network: Two dense layers with expansion factor 4 cost $O(nd^2)$ , dominant when $n$ is small but $d$ is large.
(3) Crossover Point: Attention overtakes the FFN roughly when $n \approx 2d$ ; near $n = 1024$ for the classic $d_{model} = 512$ design.

Figure 1: FLOP share vs sequence length ( $d = 512$ ): FFN dominates below $n \approx 1024$ ; past the crossover, attention’s quadratic cost becomes the bottleneck.
Mathematical Formulation (per block):
$\text{Cost}_{\text{attn}} = \underbrace{4nd^2}_{\text{QKVO projections}} + \underbrace{2n^2 d}_{QK^\top \text{ and } AV}$
$\text{Cost}_{\text{FFN}} = 2 \cdot n \cdot d \cdot 4d = 8nd^2$
Where:
$n$ is the sequence length and $d = d_{model}$ ; constants count multiply–adds per GEMM.
The $2n^2 d$ term comes from multiplying $(n \times d) \cdot (d \times n)$ to form scores, and again to mix values.
Softmax itself is cheap elementwise work but also scales with $n^2$ entries.
Sequence Length n MHA Share (%) FFN Share (%) Dominant Component
64 34.69 65.31 FFN
256 38.46 61.54 FFN
1024 50.00 50.00 Tie
4096 71.43 28.57 MHA
Table 1: FLOP breakdown at $d = 512$ : the tie at n = 1024 marks where quadratic attention catches up with the linear-in-n FFN.

Figure 2: Where the FLOPs live inside one block: only the $QK^\top$ and $AV$ pair grows quadratically with sequence length.
Practical Caveat: The FFN’s dominance at short $n$ assumes the standard 4x expansion; shrink the expansion to 1x and the Q/K/V projections become the largest term, while efficient attention variants (sliding-window, linear attention) specifically attack the $n^2$ term for long contexts.
Login to view more content
August 14, 2025

Sequence Length n	MHA Share (%)	FFN Share (%)	Dominant Component
64	34.69	65.31	FFN
256	38.46	61.54	FFN
1024	50.00	50.00	Tie
4096	71.43	28.57	MHA