Tag: Fit

DL0197 Continuous Incremental Online Learning

What is continuous learning after model deployment, and how do incremental learning and online learning differ in their handling of new data streams and catastrophic forgetting?

Answer

Continuous learning is the practice of keeping a deployed model current as the serving distribution drifts, by updating its parameters from the live stream instead of freezing the artifact that passed offline evaluation. The umbrella covers three regimes that differ in how much data one update sees and how often it ships: periodic batch retraining on the accumulated corpus, incremental learning that resumes from the current weights on a new chunk or task, and online learning that takes one low-learning-rate step per example or micro-batch and never revisits it. Incremental learning still controls the update: it can mix a replay sample of old data into every batch, run a few epochs over the chunk, and gate the result behind a shadow evaluation, so forgetting is a tunable quantity. Online learning gives up that control by construction, because each example is seen once, in arrival order, with no shuffling and no held-out replay, which makes the gradient sequence strongly non-i.i.d. and makes catastrophic forgetting the default rather than the exception. The core tension in both is the stability-plasticity dilemma: enough plasticity to absorb today’s traffic, enough stability to keep yesterday’s competence.

(1) Why Forgetting Happens: gradients computed only on new data are free to move weights that encoded the old distribution, since nothing in the loss references it. Breaking the i.i.d. sampling assumption of SGD is the mechanism, not model capacity.
(2) Incremental Learning Batches The Stream: updates operate on a chunk, task, or day of data, so replay ratios, epoch counts, and evaluation gates are all design knobs.
(3) Online Learning Streams One Pass: a single step per example, bounded memory, and no revisits, which buys seconds-level freshness and pays with instability and high variance.
(4) Three Families Of Mitigation: rehearsal (replay buffers, generative replay), regularization (EWC, distillation from the previous checkpoint), and parameter isolation (adapters, LoRA branches, masks).
(5) Forgetting Is Measurable: report per-task retention and average accuracy from the accuracy matrix, never a single aggregate on fresh traffic, which hides collapse on older slices.
(6) The Dual Failure Is Loss Of Plasticity: a model updated too conservatively for months stops learning at all, so stability alone is not the objective.

Diagram with a live traffic box on the left feeding three horizontal lanes: batch retraining on weeks of shuffled logged data with full offline evaluation and weekly deploys, incremental learning on a new chunk plus a replay sample with a few epochs from current weights and hourly to daily shadow-gated deploys, and online learning taking a single low-learning-rate step per example with learning-rate clipping, canary and rollback guardrails shipping in seconds

Figure 1: The gradient step is identical in all three lanes. What changes is how much data one update sees, how often it ships, and what protects the old distribution: shuffling over the full corpus, an explicit replay sample plus a shadow gate, or nothing but a small learning rate and a rollback button.

In production the choice is usually driven by label latency rather than by an appetite for novelty. Recommendation and ad ranking get feedback in seconds and genuinely benefit from near-online updates, so they run streaming updates on the embedding tables and the last layers while the backbone is refreshed on a slower incremental schedule. Fraud, credit, and medical models often wait days or weeks for a trustworthy label, so a per-example update would be trained on noise; incremental daily or weekly chunks with a replay mix are the safer default. The practical recipe that survives contact with traffic is rarely exotic: keep a stratified replay buffer covering old slices, mix roughly 5 to 20 percent of every batch from it, use a small re-warmed learning rate, and gate every candidate on a fixed regression suite of historical slices before promotion. Parameter isolation is attractive when tasks are known and separable, because a per-task adapter cannot be overwritten, but it grows parameters linearly in the number of tasks and needs task identity at inference unless the routing is learned.

Mathematical Formulation:
$\theta_t = \theta_{t-1} - \eta_t \nabla \ell(\theta_{t-1}; z_t)$
$B_t = \alpha B^{new}_t + (1-\alpha) B^{replay}_t$
$\mathcal{L}(\theta) = \mathcal{L}_{new}(\theta) + \lambda \Omega(\theta)$
$\Omega(\theta) = \tfrac{1}{2}\sum_i F_i (\theta_i - \theta^{*}_i)^2$
$A_K = \frac{1}{K}\sum_{j=1}^{K} a_{K,j}$
$f_j = \max_{k} a_{k,j} - a_{K,j}$

Where:

$\theta_t$ are the parameters after the $t$ -th update, $\eta_t$ the step size, and $z_t = (x_t, y_t)$ the arriving example; the pure online case uses each $z_t$ exactly once.
$B_t$ is the batch actually used at step $t$ , built from fresh data $B^{new}_t$ and a buffer sample $B^{replay}_t$ , with replay ratio $1-\alpha$ typically in the 0.05 to 0.20 range.
$\mathcal{L}_{new}$ is the loss on the incoming chunk and $\Omega$ a stability penalty anchored at the previous checkpoint $\theta^{*}$ , with $\lambda$ trading plasticity for retention.
$F_i$ is the diagonal Fisher information for parameter $i$ under the old task, so weights the old task depended on move less; setting $F_i = 1$ reduces the penalty to plain L2-to-previous.
$a_{k,j}$ is accuracy on task $j$ measured after training through task $k$ , with $j$ and $k$ indexing the $K$ chunks seen so far.
$A_K$ is average accuracy over everything seen and $f_j$ the forgetting on task $j$ , the drop from its best-ever value to its current one.

Figure 2: Sequential updates without rehearsal do not degrade gracefully, they collapse: task 1 loses 69 accuracy points by task 10. A 5 percent replay buffer recovers most of the gap to joint retraining at a fraction of the compute, while a stability penalty alone lands in between because it also suppresses learning on the new chunk.

Property	Batch retraining	Incremental learning	Online learning
Data per update	The full accumulated corpus, shuffled	One chunk, day, or task plus a replay sample	One example or micro-batch, in arrival order
Passes over data	Many epochs, i.i.d. sampling holds	A few epochs inside the chunk, old data only via buffer	Exactly one, no revisits
Freshness	Days to weeks behind traffic	Hours to a day behind	Seconds to minutes behind
Forgetting exposure	None by construction, old data is in every epoch	Bounded and tunable through replay ratio and penalty weight	High, the update sees only the current regime
Main safeguard	Full offline evaluation before promotion	Replay buffer, EWC or distillation, shadow eval on old slices	Small clipped learning rate, canary traffic, instant rollback
Dominant failure	Staleness under drift, and rising retraining cost	Buffer becomes unrepresentative, or the penalty freezes learning	Label noise and feedback loops steer the model within hours
Good fit	Slow drift, regulated or audited models	New domains, languages, or product surfaces arriving over time	Fast implicit feedback such as clicks, prices, or trending content

August 21, 2026

DL0181 Label Smoothing and Model Calibration

What is Label Smoothing, and how does it affect model calibration, overconfidence, and the trade-off between accuracy and log-likelihood?

Answer

Label smoothing replaces the one-hot target with a mixture of the one-hot vector and the uniform distribution, so the true class receives $1-\epsilon+\epsilon/K$ and every other class receives $\epsilon/K$ . Written this way it is exactly standard cross-entropy plus an $\epsilon$ -weighted KL term pulling the prediction toward uniform, which gives the loss a finite minimizer instead of one that is reached only as the true-class probability approaches 1. That single change removes the pressure that makes logit gaps grow without bound after the argmax is already correct, so the model stops becoming more confident with more training and the classic overconfidence of modern networks largely disappears. The consequences are asymmetric: Expected Calibration Error (ECE) usually drops sharply and top-1 accuracy is flat or slightly better, but the hard-label negative log-likelihood gets worse, because a converged smoothed model deliberately withholds probability mass from the correct class. The Transformer paper states this trade-off explicitly, using $\epsilon=0.1$ and noting that it hurts perplexity while improving accuracy and BLEU.

(1) Target Mixing: the label becomes $(1-\epsilon)y+\epsilon/K$ , which is equivalent to cross-entropy plus $\epsilon\,\mathrm{KL}(u\,\|\,p)$ up to an additive constant.
(2) Bounded Logit Gap: the optimum sits at a finite logit difference, roughly 9.1 nats for $\epsilon=0.1$ and $K=1000$ , so logit norms stop inflating.
(3) Confidence Ceiling: a well-fit smoothed model cannot report top-class confidence above $1-\epsilon+\epsilon/K$ , which is 0.900 at $\epsilon=0.1$ .
(4) Calibration Improves, Then Overshoots: ECE is U-shaped in $\epsilon$ , moving the model from overconfident through calibrated to systematically underconfident.
(5) Log-Likelihood Penalty: the confidence ceiling implies a hard-label NLL floor of $-\log(1-\epsilon+\epsilon/K)$ , about 0.105 nats per example at $\epsilon=0.1$ .
(6) Information Erasure: smoothing equalizes the wrong-class logits, which tightens penultimate-layer clusters and damages knowledge distillation and feature transfer.

Reliability diagram plotting bin accuracy against mean predicted confidence for three training settings: hard labels sit below the perfect-calibration diagonal indicating overconfidence, label smoothing with epsilon 0.1 tracks the diagonal closely and stops at confidence 0.900, and label smoothing with epsilon 0.3 sits above the diagonal indicating underconfidence and stops at confidence 0.700

Figure 1: Reliability curves for the same architecture under three targets. Hard labels fall below the diagonal (confidence exceeds accuracy), $\epsilon=0.1$ tracks the diagonal, and $\epsilon=0.3$ lands above it. Note the truncated horizontal extent: each smoothed curve stops at its confidence ceiling $1-\epsilon+\epsilon/K$ , 0.900 and 0.700 for $K=1000$ . Curves are illustrative of published trends rather than a single benchmark run.

The mechanism is easiest to read off the gradient with respect to the true-class logit, which is $p_y-1$ under hard labels and $p_y-(1-\epsilon+\epsilon/K)$ under smoothing. The hard-label gradient is strictly negative for every finite logit, so optimization keeps pushing the correct logit away from the others long after the prediction is right, and confidence keeps drifting upward while accuracy has already plateaued. The smoothed gradient changes sign once the model reaches the target probability, which pins confidence and caps the logit gap. The wrong-class gradients change too: each incorrect logit is pulled toward the same small target $\epsilon/K$ , so the model is discouraged from expressing that “husky” is much closer to “wolf” than to “airliner”. That equalization is exactly the property Müller and colleagues identified as the reason a label-smoothed teacher makes a worse distillation teacher than a hard-label teacher of identical accuracy.

Mathematical Formulation:
$y^{\mathrm{LS}}_k = (1-\epsilon)\,y_k + \epsilon/K$
$\mathcal{L}_{\mathrm{LS}} = -\sum_{k=1}^{K} y^{\mathrm{LS}}_k \log p_k$
$\mathcal{L}_{\mathrm{LS}} = (1-\epsilon)\mathcal{L}_{\mathrm{CE}} + \epsilon\,\mathrm{KL}(u\,\|\,p) + c$
$p^{\star}_{y} = 1 - \epsilon + \epsilon/K$
$z_y - z_j = \log\left(\frac{K(1-\epsilon)+\epsilon}{\epsilon}\right)$
$\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{N}\left|\mathrm{acc}(B_m) - \mathrm{conf}(B_m)\right|$

Where:

$y \in \{0,1\}^{K}$ is the one-hot label and $y^{\mathrm{LS}}$ the smoothed target over $K$ classes.
$\epsilon \in [0,1)$ is the smoothing strength, $\epsilon=0$ recovering ordinary cross-entropy; $0.1$ is the near-universal default.
$p_k = \mathrm{softmax}(z)_k$ is the predicted probability, $z$ the logits, $y$ the index of the correct class and $j$ any incorrect class.
$u_k = 1/K$ is the uniform distribution, and $c = \epsilon H(u)$ is a constant independent of the parameters, which is why the objective is cross-entropy plus a uniform-KL penalty.
$p^{\star}_{y}$ is the per-example minimizer, giving the finite logit gap in the fifth line; the gap grows only logarithmically as $\epsilon$ shrinks.
$B_m$ is the set of the $N$ validation examples whose top confidence falls in bin $m$ of $M$ (typically $M=15$ ), and ECE is the confidence-weighted gap plotted in Figure 1.

Log-Likelihood Floor At $\epsilon=0.1$ , $K=1000$ :
$p^{\star}_{y} = 0.9 + 0.0001 = 0.9001$
$-\log(0.9001) = 0.1052$
$\exp(0.1052) = 1.111$

The third line is the practical reading for sequence models: a converged smoothed model pays roughly 11% higher perplexity than its own hard-label counterpart would at the same accuracy, purely from the withheld mass, which is why translation systems that smooth at training time are usually evaluated with BLEU or COMET rather than perplexity. Note also that this floor is a property of the optimum, not a hard constraint on the network: a single global temperature fitted on held-out data can sharpen the smoothed logits back and recover most of the lost likelihood. That observation is the reason smoothing is not the right tool if calibrated probabilities are the goal, since post-hoc temperature scaling reaches comparable ECE at zero training cost, one scalar parameter, and no distortion of the wrong-class ranking.

Three side-by-side line charts against smoothing strength epsilon from 0 to 0.4: hard-label negative log-likelihood dips slightly then rises while an analytic floor curve rises monotonically, expected calibration error forms a U shape with a minimum near epsilon 0.1, and top-1 accuracy stays on a flat plateau before declining past epsilon 0.2

Figure 2: The three metrics disagree about the best $\epsilon$ . ECE is U-shaped with a minimum near $0.05$ to $0.1$ , hard-label NLL dips only briefly before the analytic floor $-\log(1-\epsilon+\epsilon/K)$ dominates, and accuracy is flat across a broad plateau. Choosing $\epsilon$ therefore means choosing which metric you are optimizing. Values are illustrative of published sweeps.

Property	Label smoothing	Temperature scaling	Confidence penalty / focal loss
When applied	Training time, changes the targets	Post-hoc, one scalar fitted on a held-out split	Training time, changes the loss on the prediction side
Effect on accuracy	Flat to slightly better; degrades past roughly 0.2	Exactly zero, the argmax is invariant to a positive temperature	Task dependent; focal loss helps mainly under heavy class imbalance
Effect on hard-label NLL	Worse at convergence, floored at $-\log(1-\epsilon+\epsilon/K)$	Directly minimized by the fitting objective, so it improves	Usually worse, for the same mass-withholding reason
Logit ranking preserved	No, wrong-class logits are equalized	Yes, it is a monotone rescaling of all logits	Partially, the entropy term flattens the tail
Main failure mode	Underconfidence at large $\epsilon$ ; weaker distillation teacher and weaker transfer features	One global scalar cannot fix per-class or per-slice miscalibration, and it needs a clean validation split	Extra hyperparameter with little calibration gain over a fitted temperature

August 20, 2026

DL0180 Convex vs Non-Convex Optimization Landscapes

Explain the difference between convex and non-convex optimization landscapes in high-dimensional deep learning, and why non-convexity in deep networks is empirically more tractable than classical theory predicts.

Answer

A convex objective is shaped like a single bowl: every local minimum is a global minimum, the stationarity condition $\nabla f(x) = 0$ is sufficient for optimality, and convergence rates hold from any initialization. Deep networks violate convexity by construction, because composing affine maps with nonlinearities produces a loss that is invariant under permuting hidden units within a layer, so the surface already carries $\prod_l h_l!$ equivalent copies of every solution and no convex reparameterization of the weights exists. Classical theory then warns that gradient descent could stall in a poor local minimum, but in high dimension the geometry is different: a local minimum requires all $d$ Hessian eigenvalues to be positive at once, which random-matrix and spin-glass arguments make exponentially improbable except very close to the loss floor, so the overwhelming majority of critical points are saddles that gradient noise escapes. Overparameterized networks add a second effect: along the training trajectory the loss satisfies a local Polyak-Lojasiewicz condition, which gives linear convergence to a near-zero-loss solution without any convexity assumption. The solutions SGD reaches are also not isolated, since distinct runs are joined by low-loss curved paths, so the practical obstructions are ill-conditioning, plateaus, and degenerate saddles rather than the isolated bad basins convexity theory teaches you to fear.

(1) What Convexity Buys: local equals global, a checkable optimality certificate, duality gaps, and initialization-independent rates such as $O(1/k)$ for gradient descent on smooth convex losses.
(2) Deep Nets Are Non-Convex By Construction: nested nonlinearities plus permutation and rescaling symmetries mean the loss is a highly multimodal function of the weights even when it is convex in the network output.
(3) Saddles Dominate Critical Points: the fraction of negative Hessian eigenvalues, the index, is almost never zero away from the loss floor, so most flat regions are passable rather than terminal.
(4) Band Structure Of Critical Values: the expected loss of a critical point grows roughly linearly with its index, so high-loss critical points come with escape directions and low-loss ones are what remains.
(5) Overparameterization Tames The Trajectory: in the wide-network regime the loss is nearly a convex function of the output and satisfies a PL inequality locally, giving linear convergence to global training loss near zero.
(6) The Low-Loss Set Is Connected: mode connectivity shows independent solutions are linked by low-loss curves, and permutation alignment removes most of the naive linear-interpolation barrier.
(7) The Real Enemies Are Conditioning And Degeneracy: plateaus with near-zero curvature in every direction, saturated units, and Hessian condition numbers in the thousands cost far more wall-clock time than bad minima.

Two contour maps side by side. Left: a single elliptical bowl with three gradient descent trajectories from different initializations all converging to the same central minimum. Right: a rugged multi-basin surface where three trajectories from different initializations settle into three different basins of comparable depth, with one saddle point marked between two basins.

Figure 1: The structural difference in two dimensions. On the convex bowl every trajectory reaches the same point, so the answer is a property of the objective. On the rugged surface the endpoint depends on initialization, yet the basins reached have comparable depth, which is the two-dimensional cartoon of what actually happens in $d \approx 10^{8}$ : many solutions, not many bad solutions.

The intuition that non-convexity means “gradient descent gets trapped” comes from low-dimensional pictures, where a basin is bounded by walls in both directions. In $d$ dimensions a critical point is a minimum only if the Hessian is positive definite in every one of $d$ directions, and a single negative eigenvalue is enough to make it a saddle with a downhill escape route. Under a Gaussian-random-field or spin-glass model of the loss, the probability that a randomly encountered critical point has index zero decays like $e^{-c d}$ , and the critical values organize into a band: the higher the loss, the larger the expected index. Measured Hessian spectra at converged deep-network solutions match the qualitative picture, with a bulk of near-zero eigenvalues (the flat, degenerate directions created by symmetry and overparameterization), a handful of large positive outliers roughly matching the number of classes, and a thin slightly negative tail. Theory for deep linear networks makes the same point exactly: every local minimum is a global minimum, and the non-convexity manifests as saddles rather than as spurious basins.

Left panel: scatter plot of loss value versus index fraction for simulated critical points, forming an upward band so that critical points with zero or near-zero index cluster at the lowest loss values, with a shaded strip marking the index-zero region. Right panel: log-scale histogram of Hessian eigenvalues at a converged solution showing a very tall bulk concentrated at zero, a thin negative tail, and a dozen isolated large positive outliers.

Figure 2: Left, the band structure: critical points with a high fraction of negative curvature directions sit at high loss, and only points near the loss floor are close to index zero, so “escapable” and “bad” are the same set. Right, a representative Hessian spectrum: thousands of near-zero eigenvalues, a few large outliers that set the effective step-size limit, and a small negative tail that keeps the point technically a saddle.

Mathematical Formulation:
$f(\lambda x + (1-\lambda) y) \leq \lambda f(x) + (1-\lambda) f(y)$
$\nabla^2 f(x) \succeq 0$
$\alpha(x) = k_{-}(x) / d$
$\mathbb{E}[f \mid \alpha] \approx f_0 + c\,\alpha$
$P(\alpha = 0) \sim e^{-c d}$

Where:

$f$ is the training loss as a function of the parameter vector, $x, y$ are two parameter settings, and $\lambda \in [0,1]$ interpolates between them; the first line is the definition of convexity and the second its twice-differentiable equivalent.
$\nabla^2 f(x) \succeq 0$ means the Hessian is positive semidefinite everywhere, which is exactly the condition a deep network’s loss fails to satisfy.
$k_{-}(x)$ counts negative Hessian eigenvalues at a critical point, $d$ is the parameter count, and $\alpha(x)$ is the index fraction; $\alpha = 0$ is a local minimum and any $\alpha > 0$ is a saddle.
$f_0$ is the loss floor and $c$ a positive constant, so the fourth line states the band structure: expected critical value increases with index.
The last line is the random-matrix scaling for an index-zero critical point of a Gaussian random field, which is why bad local minima are measure-zero in practice for large $d$ .

Why Non-Convexity Is Still Solvable:
$\|\nabla f(x)\|^{2} \geq 2\mu\,(f(x) - f^{*})$
$f(x_k) - f^{*} \leq (1 - \mu/L)^{k}\,(f(x_0) - f^{*})$
$\|\nabla f(x)\| \leq \epsilon$
$\lambda_{\min}(\nabla^2 f(x)) \geq -\sqrt{\epsilon}$
$T = O(\epsilon^{-2}\log^{4}(d/\epsilon))$

Where:

The first line is the Polyak-Lojasiewicz (PL) inequality with constant $\mu > 0$ and optimal value $f^{*}$ : wherever the loss is far from optimal, the gradient is large.
PL plus $L$ -smoothness yields the second line, a linear convergence rate with no convexity assumption, and wide networks provably satisfy a local PL condition in a neighborhood of their initialization.
The next two lines define an $\epsilon$ –second-order stationary point: small gradient and no strongly negative curvature, which rules out strict saddles rather than merely first-order stalls.
$T$ is the iteration count in which perturbed gradient descent reaches such a point, with only polylogarithmic dependence on the dimension $d$ ; SGD’s minibatch noise plays the same role for free.

The last piece of the empirical story is that the solutions are not isolated points in separate valleys. Naively interpolating linearly between two independently trained networks produces a large loss barrier, which looks like evidence for distinct basins, but that barrier is largely an artifact of the permutation symmetry: after matching the hidden units of one network to the other, the linear path becomes far flatter, and a low-loss quadratic Bezier path can be found between essentially any pair of solutions. Functionally the minima behave like a single connected low-loss manifold, modulo relabeling, which is why techniques that assume a connected solution set (weight averaging, ensembling along a path, cyclical learning rates) work at all. In production this reframes debugging: a run stuck at high loss is almost never in a bad minimum, it is in a plateau caused by dead units, a saturated nonlinearity, a bad initialization scale, or a step size fighting the largest Hessian eigenvalue.

Line chart of training loss along a path between two independently trained solutions as the interpolation coefficient goes from zero to one. The naive linear interpolation curve rises to a tall barrier in the middle, the permutation-aligned linear interpolation curve rises only slightly, and the curved Bezier path stays essentially flat at the endpoint loss, with a double-headed arrow marking the barrier height.

Figure 3: The apparent wall between two solutions is mostly symmetry, not geometry. Naive linear interpolation crosses a large barrier; permutation alignment flattens most of it, and a learned curved path stays at the endpoint loss the whole way, evidence that the low-loss set is connected rather than a collection of isolated basins.

Property	Convex objective (logistic regression, SVM, LASSO)	Non-convex deep network loss
Stationary points	One connected set of global minima; no saddles	Exponentially many critical points, mostly saddles; global minima replicated by permutation symmetry
Optimality certificate	Zero gradient (or a duality gap bound) proves global optimality	None available; you report training loss and validation metrics, not optimality
Effect of initialization	Affects speed only; the solution is unique up to degeneracy	Selects which solution you land in; scale of initialization can decide whether training starts at all
Role of dimension	Higher $d$ costs compute and risks overfitting	Higher $d$ helps: index-zero critical points become rare and a local PL condition appears
What stalls training	Condition number of the Hessian, non-smooth regularizers	Degenerate plateaus, dead or saturated units, exploding curvature at the step-size limit
Reproducibility	Bitwise-comparable solutions across seeds and solvers	Different weights every seed; only the function learned is comparable

August 20, 2026

DL0069 RMSProp Adaptive Learning Rates

How does RMSProp adapt the learning rate per parameter?

Answer

RMSProp keeps a per-parameter exponential moving average of squared gradients and divides each gradient component by the square root of that average before stepping, so the effective learning rate shrinks for parameters with historically large gradients and grows for those with small or sparse ones. The running estimate $v_t$ acts as a per-coordinate normalizer: two parameters share one global learning rate yet move by very different amounts. On ill-conditioned surfaces, where curvature differs wildly across directions, this damps oscillation along steep directions while sustaining progress along shallow ones, something a single global learning rate cannot do. Tieleman and Hinton introduced the method in their 2012 Coursera lecture series to cope with non-stationary objectives such as mini-batch and recurrent training; Adam is essentially RMSProp plus a first-moment momentum term and bias correction.

(1) Per-Coordinate Normalization: each parameter steps by $\alpha\, g_t / (\sqrt{v_t} + \epsilon)$ ; a parameter whose gradients run 10x larger builds a 10x larger $\sqrt{v_t}$ , so its effective learning rate shrinks 10x and the two coordinates end up moving by comparable amounts instead of one dwarfing the other.
(2) Memory With a Window: the decay $\rho$ (0.9 in the original lecture; PyTorch’s RMSprop defaults to 0.99) makes $v_t$ an average over recent history, so the normalization adapts as the landscape changes instead of stalling like AdaGrad’s monotonically growing accumulator.
(3) No Bias Correction: $v_t$ starts at zero and is never corrected, so early steps are oversized; Adam fixes this with $\hat{v}_t = v_t / (1 - \rho^t)$ , one of the two additions (the other is momentum) that turn RMSProp into Adam.

Contour plot of the ravine f(x,y) = 0.05x^2 + 2y^2 with two 42-step trajectories from (-10, 4): the red SGD path zigzags across the steep y direction with decaying overshoots while creeping along x to about -1.5, and the blue RMSProp path settles into the valley without oscillating and travels along it to the minimum

Figure 1: On the ravine $f(x,y) = 0.05x^2 + 2y^2$ , SGD spends its budget oscillating across the steep $y$ direction and after 42 steps is still at $x \approx -1.5$ ; RMSProp normalizes both coordinates, never overshoots, and reaches $x \approx -0.3$ in the same 42 steps.

The same normalization explains RMSProp’s strength on sparse features. For an embedding row that receives a gradient only occasionally, $v_t$ decays toward zero between updates, so when a gradient finally arrives its effective learning rate $\alpha / \sqrt{v_t}$ is several times larger than a densely-updated parameter’s, and the rare signal is not drowned out by a global rate tuned for frequent gradients. The flip side appears at the start of every run: with $\rho = 0.99$ the first estimate is $v_1 = 0.01\, g_1^2$ , so the first step has magnitude $10\,\alpha$ whatever the gradient scale, a warmup-like quirk that Adam’s bias correction removes. In practice RMSProp remains a solid choice for RNNs and other non-stationary objectives, while vision recipes still often prefer well-tuned SGD with momentum for final generalization.

Two stacked panels over 60 training steps: top shows sqrt of v_t rising from 0.2 toward its asymptote of 2 for a parameter with large steady gradient while staying near 0.3 for a sparse-gradient parameter; bottom shows the effective learning rate alpha over sqrt(v_t) falling from 5 to 0.74 for the first parameter while the second stays near 2.9 in a sawtooth pattern

Figure 2: Two parameters over 60 steps with $\rho = 0.99$ : the one with a large steady gradient sees its effective learning rate $\alpha / \sqrt{v_t}$ fall 6.7x by step 60 (heading for $\alpha/2$ as $\sqrt{v_t} \to |g|$ ), while the sparse-gradient parameter still enjoys a roughly 4x larger rate.

Mathematical Formulation:
$v_t = \rho\, v_{t-1} + (1 - \rho)\, g_t^2$
$\theta_{t+1} = \theta_t - \frac{\alpha\, g_t}{\sqrt{v_t} + \epsilon}$

Where:

$g_t = \nabla f_t(\theta_t)$ is the minibatch gradient at step $t$ ; the squaring in the accumulator is element-wise.
$v_t$ is the per-parameter running average of squared gradients, with the same shape as $\theta$ , initialized to zero.
$\rho$ is the decay (0.9 in Hinton’s lecture, 0.99 by default in PyTorch), $\alpha$ the global learning rate, and $\epsilon \approx 10^{-8}$ a numerical floor guarding the division.
All operations are element-wise, so each parameter gets its own effective rate $\alpha / (\sqrt{v_t} + \epsilon)$ under one shared $\alpha$ .

Aspect	SGD + Momentum	RMSProp	Adam
State per Parameter	Velocity m (first moment)	Squared-gradient average v (second moment)	Both m and v
Normalization	None; one global rate for all coordinates	Gradient divided by sqrt(v) per coordinate	Bias-corrected m divided by sqrt(v-hat)
Early Steps	Stable from step one	Oversized; v underestimated with no correction	Bias correction keeps steps near alpha scale
Typical Strength	Final test accuracy on tuned vision recipes	RNNs and non-stationary objectives	Default for transformers and general use

August 12, 2026

ML0083 Weight Decay

What is weight decay in neural network training, and how does it affect model parameters?

Answer

Weight decay multiplies every weight by a factor slightly below one at each step, so parameters continuously shrink toward zero unless the data gradient pushes them back up. The equilibrium effect: only directions where the loss gradient persistently outweighs the decay survive with large magnitude, while noise-fitting directions get eroded, which is exactly the capacity control we want. For plain SGD, weight decay is mathematically identical to adding an L2 penalty to the loss. For adaptive optimizers like Adam it is not, because the L2 gradient gets rescaled per-parameter along with everything else; AdamW fixes this by decoupling the decay from the adaptive update, and is now the default recipe for training transformers.

(1) Mechanics: each step shrinks weights multiplicatively before (or alongside) the gradient step, so unneeded complexity decays exponentially while useful weights are constantly re-earned from the data gradient.
(2) SGD Equivalence and Its Caveat: for SGD, weight decay with rate lambda equals L2 regularization with coefficient lambda/alpha, so the two are coupled through the learning rate; retuning alpha silently retunes your regularization.
(3) AdamW Fix: Loshchilov and Hutter showed Adam’s adaptive scaling under-regularizes high-gradient weights when L2 is folded into the loss; AdamW applies decay outside the adaptive step, and PyTorch’s AdamW defaults to a weight decay of 0.01.

Elliptical loss contours centered away from the origin, a dashed L2 constraint circle around the origin, and the regularized optimum where the circle touches a contour

Figure 1: The geometric view: the unregularized optimum sits far from the origin; weight decay pulls the solution to where the smallest L2 ball first touches the loss contours, trading a little fit for a lot of magnitude.

The Adam subtlety is worth understanding precisely, because it is a favorite interview trap. In Adam, every component of the gradient, including the penalty’s contribution, is divided by the square root of its running second moment. Parameters with historically large data gradients therefore experience a heavily damped regularization force, which is backwards: the parameters the optimizer is already moving hardest are the ones the penalty reaches least. Note the denominator tracks the second moment of the gradient, not the weight’s own magnitude, so the effect is uneven regularization across parameters rather than a simple “big weights decay less” rule. AdamW removes the penalty from the adaptive gradient entirely and applies it as a direct multiplicative shrink, restoring uniform, controllable decay, which is why nearly every modern pretraining run (GPT-style models included) specifies AdamW.

Weight norm over training steps: no decay grows steadily, Adam plus L2 barely shrinks it, AdamW settles at a lower plateau

Figure 2: The decoupling effect: with no decay the weight norm grows unchecked; Adam with L2 in the loss barely contains it because the penalty is adaptively rescaled; AdamW’s decoupled decay settles at a controlled plateau.

Mathematical Formulation:
$\theta_{t+1} = (1 - \lambda)\,\theta_t - \alpha\, \nabla f_t(\theta_t)$
$f^{reg}_t(\theta) = f_t(\theta) + \frac{\lambda'}{2}\,\|\theta\|^2, \qquad \lambda' = \lambda / \alpha$

Where:

$\theta_t$ collects all parameters at step $t$ , $\alpha$ is the learning rate, and $\lambda$ is the per-step decay rate.
$f_t$ is the minibatch loss; the $(1 - \lambda)$ factor is the multiplicative shrink applied every step.
$\lambda' = \lambda/\alpha$ is the L2 coefficient that makes the two formulations equivalent for SGD, exposing the coupling: change the learning rate and the effective regularization changes too (this equivalence fails for adaptive optimizers, hence AdamW).

Aspect	L2 in the Loss (Adam)	Decoupled Decay (AdamW)
Where Applied	Penalty gradient added to the loss gradient	Direct multiplicative shrink, outside the adaptive step
Adaptive Scaling	Penalty is divided by sqrt(v), so large-gradient weights decay least	Decay is uniform across parameters
Hyperparameter Coupling	Effective regularization entangled with learning rate and gradient scale	Decay rate tuned independently of learning rate
Default Use	Legacy; PyTorch Adam weight_decay argument is actually L2	Standard for transformer pretraining; PyTorch default 0.01

August 11, 2026

ML0040 Bias and Variance
Can you explain the bias-variance tradeoff?
Answer
The bias-variance tradeoff decomposes a model’s expected prediction error into three parts: squared bias, variance, and irreducible noise. Bias is the error from overly simplified assumptions: a high-bias model misses the real pattern and underfits. Variance is the error from sensitivity to the particular training sample: a high-variance model wiggles to fit noise and overfits. The tradeoff arises because increasing model complexity typically decreases bias but increases variance, while simplifying does the reverse: total error as a function of complexity is U-shaped, and the goal is the sweet spot that minimizes the sum. Practically, high bias shows as large training error; high variance shows as a large gap between training and validation error, and each has its own remedies (more capacity/features for bias; more data, regularization, or simpler models for variance).
(1) Bias: Error from wrong assumptions: underfitting, poor fit on both train and test data.
(2) Variance: Error from sample sensitivity: overfitting, big train/test gap.
(3) Tradeoff: Complexity trades one for the other; total error $= \text{Bias}^2 + \text{Variance} + \sigma^2$ is U-shaped: minimize the sum.

Figure 1: The classic tradeoff curve: bias² falls and variance rises as complexity grows; total error is their U-shaped sum, and the best model sits at the minimum, not at maximum complexity.
Mathematical Formulation:
$\mathbb{E}\big[(y - \hat{f}(x))^2\big] = \underbrace{\big(\mathbb{E}[\hat{f}(x)] - f(x)\big)^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}\big[(\hat{f}(x) - \mathbb{E}[\hat{f}(x)])^2\big]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{noise}}$
Where:
$f(x)$ is the true relationship and $\hat{f}(x)$ the model’s prediction; expectations are over training sets.
$\text{Bias}^2$ measures how far the average model is from the truth; $\text{Variance}$ how much predictions scatter around that average.
$\sigma^2$ is the irreducible error: noise in the data itself that no model can eliminate.

Figure 2: The tradeoff on real-shaped data: the high-bias model is too rigid to follow the curve (both errors high); the high-variance model chases every noisy point (train error low, test error high); the balanced model tracks the true function and minimizes test error.
Login to view more content
May 21, 2025
ML0004 Underfitting
What is underfitting, how do you recognize it, and how can you fix it?
Answer
Underfitting occurs when a model is too simple to capture the underlying patterns in the data, so it performs poorly on both the training data and new, unseen data: it has not even learned the training set. An underfit model exhibits high bias and low variance: its predictions are consistently wrong in the same way, regardless of the particular training sample. Common causes are an overly simple model, inadequate training (stopped too early), over-regularization, and poor feature selection. The fixes mirror the causes: increase model complexity, train longer, reduce regularization, and engineer more informative features.
(1) Definition: The model lacks the capacity to fit the signal, so error stays high on training and test data alike.
(2) Recognition: High training error is the key signature; contrast with overfitting, where training error is low and only validation error suffers.
(3) Fixes: Add capacity (more layers, higher-degree features), train longer, weaken regularization, and improve the feature set.

Figure 1: The same data fitted three ways. The underfit line misses the pattern entirely; the good fit follows the trend; the overfit curve chases every noisy point.
Mathematical Formulation:
$\mathrm{Err}(x_0) = \mathrm{Bias}^2\big(\hat{f}(x_0)\big) + \mathrm{Var}\big(\hat{f}(x_0)\big) + \sigma^2$
$\text{underfitting} \;\Rightarrow\; \mathrm{Bias}^2 \text{ dominates the total error}$
Where:
$\mathrm{Err}(x_0)$ is the expected prediction error of the model $\hat{f}$ at a new point $x_0$ .
$\mathrm{Bias}^2$ measures how far the average prediction sits from the truth: the term that dominates when a model underfits.
$\mathrm{Var}$ measures how much the prediction swings across different training sets: the term that dominates in overfitting.
$\sigma^2$ is the irreducible noise in the data, which no model can remove.
Login to view more content
February 26, 2025
ML0003 Overfitting
What is overfitting and how to avoid overfitting?
Answer
Overfitting happens when a model learns the training data too well (including its noise and outliers) and as a result performs poorly on new, unseen data. The model becomes too specialized to the training set and fails to generalize. The telltale sign is a growing generalization gap: training loss keeps falling while validation loss turns back up. To avoid overfitting: simplify the model, get more data or use data augmentation, apply regularization (L1/L2), validate frequently with early stopping, and for neural networks use dropout.
(1) Definition: The model memorizes noise as if it were signal, so training error keeps dropping while test error rises.
(2) Detection: Watch the train/validation loss gap and use cross-validation: wildly varying performance across folds indicates overfitting.
(3) Remedies: More or augmented data, L1/L2 regularization, dropout, early stopping, or a smaller model; all reduce effective capacity or expose the model to more variation.

Figure 1: The validation loss minimum marks the ideal stopping point; training past it widens the generalization gap; that widening is overfitting.
Mathematical Formulation:
$\mathcal{L}_{\mathrm{reg}}(\theta) = \mathcal{L}(\theta) + \lambda \lVert \theta \rVert_2^2$
$\mathrm{gap} = \mathcal{L}_{\mathrm{val}}(\theta) - \mathcal{L}_{\mathrm{train}}(\theta)$
Where:
$\mathcal{L}(\theta)$ is the original training loss over parameters $\theta$ .
$\lambda$ is the regularization strength; larger values shrink the weights $\theta$ toward zero, trading training fit for generalization.
$\lVert \theta \rVert_2^2$ is the squared L2 norm of the weights (weight decay); an L1 penalty $\lVert \theta \rVert_1$ instead drives weights to exactly zero.
$\mathcal{L}_{\mathrm{val}}$ and $\mathcal{L}_{\mathrm{train}}$ are validation and training loss; a small, stable gap indicates good generalization.

Figure 2: Why the remedies work: they move the model left along the complexity axis, out of the high-variance region and back toward the total-error minimum.
Login to view more content
February 25, 2025