Author: admin

DL0070 Multi-Task Loss Balancing

How would you design a loss for multi-task learning when the tasks have very different scales?

Answer

Start from the failure mode: with a naive sum $L = \sum_i L_i$ , each task contributes gradient in proportion to its loss scale, so a depth-regression MSE sitting near 200 drowns out a segmentation cross-entropy sitting near 1, and the shared trunk optimizes the loud task while the quiet ones stall. The fix comes in three tiers. First, put the losses on a common scale by dividing each by a fixed or running estimate of its magnitude, such as its initial value. Second, learn the weights: Kendall et al.’s uncertainty weighting attaches a trainable $\sigma_i$ per task and minimizes $\sum_i L_i / (2\sigma_i^2) + \log \sigma_i$ , derived from Gaussian and categorical likelihoods, so noisy tasks are down-weighted automatically while the $\log \sigma_i$ term stops $\sigma_i$ from blowing up. Third, when scales are balanced but gradient directions still conflict, move to gradient-level methods: GradNorm tunes the task weights so each task’s gradient norm on the shared trunk approaches a common scale adjusted by its relative inverse training rate, so tasks that are learning slowly get pushed harder, and PCGrad projects away mutually conflicting components.

(1) Scale Equals Loudness: task $i$ ‘s gradient share scales with the units of $L_i$ , so the unweighted sum is an implicit weighting set by arbitrary unit choices; expressing depth in millimeters instead of meters multiplies its MSE by $10^6$ and hands it the entire gradient budget.
(2) Uncertainty Weighting: treat $\sigma_i$ as task $i$ ‘s observation noise; the weight $1/(2\sigma_i^2)$ falls as noise grows while the $\log \sigma_i$ penalty (the likelihood’s normalizing constant) rises, so the optimum is a genuine trade-off learned by gradient descent alongside the network weights.
(3) Magnitude Is Not Direction: scale balancing fixes how loudly tasks speak, not whether they agree; when task gradients point in opposing directions (negative transfer), gradient-space methods such as GradNorm or PCGrad are the right lever.

Grouped bar chart on a log scale of illustrative gradient norms on the shared trunk for segmentation, depth, and normals tasks: unweighted bars are 1.1, 160, and 0.8, while uncertainty-weighted bars are 0.9, 1.0, and 0.7

Figure 1: Illustrative three-task trunk, gradient norms on a log scale: unweighted, the depth MSE contributes over 100x the others; once each loss carries a learned $1/(2\sigma_i^2)$ weight, the three contribute comparably.

The uncertainty objective is not ad hoc. Modeling regression noise as $p(y \mid f(x)) = \mathcal{N}(f(x), \sigma^2)$ gives the negative log-likelihood $\|y - f(x)\|^2 / (2\sigma^2) + \log \sigma$ per task, up to constants; classification slots in through a scaled softmax likelihood whose approximation yields the analogous weight $1/\sigma^2$ (without the factor 2) alongside the same $\log \sigma$ penalty. Two properties matter in practice: $\sigma_i$ is learned by the same optimizer as the network, so balancing needs no manual grid search; and the $\log \sigma_i$ term keeps the objective honest, because without it every $\sigma_i$ would grow without bound, all task weights would collapse to zero, and nothing would be learned. Most implementations optimize $s_i = \log \sigma_i^2$ and compute $e^{-s_i} L_i + s_i$ for numerical stability.

Two panels against task uncertainty sigma: top shows the task weight 1/(2 sigma^2) collapsing on a log scale as sigma grows from 0.25 to 4; bottom shows the penalty log sigma rising over the same range

Figure 2: The learned trade-off: as a task’s noise $\sigma$ grows, its weight $1/(2\sigma^2)$ collapses while the penalty $\log \sigma$ rises, so the optimizer cannot silence a noisy task for free.

Mathematical Formulation:
$L_{\mathrm{naive}} = \sum_{i=1}^{T} L_i$
$L_{\mathrm{total}} = \sum_{i=1}^{T} \left( \frac{1}{2\sigma_i^2}\, L_i + \log \sigma_i \right)$

Where:

$L_i$ is the loss of task $i$ (cross-entropy for segmentation, MSE for depth) and $T$ the number of tasks sharing the trunk.
$\sigma_i$ is a learned per-task scalar modeling homoscedastic (task-level, input-independent) observation noise, initialized at 1 and trained by the same optimizer as the weights.
$1/(2\sigma_i^2)$ is the derived task weight and $\log \sigma_i$ the likelihood’s normalizing term that penalizes inflating $\sigma_i$ ; in code, parameterize $s_i = \log \sigma_i^2$ and optimize $e^{-s_i} L_i + s_i$ instead.

Method	What It Balances	Extra Cost	When to Reach for It
Fixed Weights (Grid Search)	Loss scales, set by hand	Search cost grows fast with task count	Two tasks and plenty of compute
Loss Normalization	Loss magnitudes via running scale estimates	Negligible	Quick baseline before anything fancier
Uncertainty Weighting (Kendall)	Loss weights via learned sigma per task	One extra scalar per task	Default starting point for shared-trunk training
GradNorm	Gradient norms toward a common scale by training rate	Extra backward bookkeeping each step	Tasks learning at very different speeds
PCGrad	Conflicting gradient directions	Per-task gradients every step	Negative transfer between tasks

August 12, 2026

DL0069 RMSProp Adaptive Learning Rates

How does RMSProp adapt the learning rate per parameter?

Answer

RMSProp keeps a per-parameter exponential moving average of squared gradients and divides each gradient component by the square root of that average before stepping, so the effective learning rate shrinks for parameters with historically large gradients and grows for those with small or sparse ones. The running estimate $v_t$ acts as a per-coordinate normalizer: two parameters share one global learning rate yet move by very different amounts. On ill-conditioned surfaces, where curvature differs wildly across directions, this damps oscillation along steep directions while sustaining progress along shallow ones, something a single global learning rate cannot do. Tieleman and Hinton introduced the method in their 2012 Coursera lecture series to cope with non-stationary objectives such as mini-batch and recurrent training; Adam is essentially RMSProp plus a first-moment momentum term and bias correction.

(1) Per-Coordinate Normalization: each parameter steps by $\alpha\, g_t / (\sqrt{v_t} + \epsilon)$ ; a parameter whose gradients run 10x larger builds a 10x larger $\sqrt{v_t}$ , so its effective learning rate shrinks 10x and the two coordinates end up moving by comparable amounts instead of one dwarfing the other.
(2) Memory With a Window: the decay $\rho$ (0.9 in the original lecture; PyTorch’s RMSprop defaults to 0.99) makes $v_t$ an average over recent history, so the normalization adapts as the landscape changes instead of stalling like AdaGrad’s monotonically growing accumulator.
(3) No Bias Correction: $v_t$ starts at zero and is never corrected, so early steps are oversized; Adam fixes this with $\hat{v}_t = v_t / (1 - \rho^t)$ , one of the two additions (the other is momentum) that turn RMSProp into Adam.

Contour plot of the ravine f(x,y) = 0.05x^2 + 2y^2 with two 42-step trajectories from (-10, 4): the red SGD path zigzags across the steep y direction with decaying overshoots while creeping along x to about -1.5, and the blue RMSProp path settles into the valley without oscillating and travels along it to the minimum

Figure 1: On the ravine $f(x,y) = 0.05x^2 + 2y^2$ , SGD spends its budget oscillating across the steep $y$ direction and after 42 steps is still at $x \approx -1.5$ ; RMSProp normalizes both coordinates, never overshoots, and reaches $x \approx -0.3$ in the same 42 steps.

The same normalization explains RMSProp’s strength on sparse features. For an embedding row that receives a gradient only occasionally, $v_t$ decays toward zero between updates, so when a gradient finally arrives its effective learning rate $\alpha / \sqrt{v_t}$ is several times larger than a densely-updated parameter’s, and the rare signal is not drowned out by a global rate tuned for frequent gradients. The flip side appears at the start of every run: with $\rho = 0.99$ the first estimate is $v_1 = 0.01\, g_1^2$ , so the first step has magnitude $10\,\alpha$ whatever the gradient scale, a warmup-like quirk that Adam’s bias correction removes. In practice RMSProp remains a solid choice for RNNs and other non-stationary objectives, while vision recipes still often prefer well-tuned SGD with momentum for final generalization.

Two stacked panels over 60 training steps: top shows sqrt of v_t rising from 0.2 toward its asymptote of 2 for a parameter with large steady gradient while staying near 0.3 for a sparse-gradient parameter; bottom shows the effective learning rate alpha over sqrt(v_t) falling from 5 to 0.74 for the first parameter while the second stays near 2.9 in a sawtooth pattern

Figure 2: Two parameters over 60 steps with $\rho = 0.99$ : the one with a large steady gradient sees its effective learning rate $\alpha / \sqrt{v_t}$ fall 6.7x by step 60 (heading for $\alpha/2$ as $\sqrt{v_t} \to |g|$ ), while the sparse-gradient parameter still enjoys a roughly 4x larger rate.

Mathematical Formulation:
$v_t = \rho\, v_{t-1} + (1 - \rho)\, g_t^2$
$\theta_{t+1} = \theta_t - \frac{\alpha\, g_t}{\sqrt{v_t} + \epsilon}$

Where:

$g_t = \nabla f_t(\theta_t)$ is the minibatch gradient at step $t$ ; the squaring in the accumulator is element-wise.
$v_t$ is the per-parameter running average of squared gradients, with the same shape as $\theta$ , initialized to zero.
$\rho$ is the decay (0.9 in Hinton’s lecture, 0.99 by default in PyTorch), $\alpha$ the global learning rate, and $\epsilon \approx 10^{-8}$ a numerical floor guarding the division.
All operations are element-wise, so each parameter gets its own effective rate $\alpha / (\sqrt{v_t} + \epsilon)$ under one shared $\alpha$ .

Aspect	SGD + Momentum	RMSProp	Adam
State per Parameter	Velocity m (first moment)	Squared-gradient average v (second moment)	Both m and v
Normalization	None; one global rate for all coordinates	Gradient divided by sqrt(v) per coordinate	Bias-corrected m divided by sqrt(v-hat)
Early Steps	Stable from step one	Oversized; v underestimated with no correction	Bias correction keeps steps near alpha scale
Typical Strength	Final test accuracy on tuned vision recipes	RNNs and non-stationary objectives	Default for transformers and general use

August 12, 2026

DL0068 Computational Graphs

What is a computational graph, and why is it useful in deep learning frameworks?

Answer

A computational graph is a directed acyclic graph whose nodes are operations or stored values and whose edges are the tensors flowing between them. Deep learning frameworks materialize this graph while running the forward pass: PyTorch records it dynamically on every iteration (define-by-run), while early TensorFlow built the whole graph statically before execution. Its central use is reverse-mode automatic differentiation: every node carries a known local derivative, so the chain rule becomes a mechanical backward sweep that produces every parameter’s gradient in a single pass, whose cost is roughly 2-3 times the forward pass and does not grow with the number of parameters being differentiated. The same data structure also drives memory planning such as activation checkpointing, whole-graph compiler optimizations like kernel fusion in XLA or torch.compile, device placement, and deployment export to formats such as ONNX.

(1) Structure and Evaluation: nodes are ops or values and edges are tensors; the forward pass evaluates nodes in topological order and caches every intermediate, because the backward sweep needs those values to compute local derivatives.
(2) One Sweep, All Gradients: each node multiplies its incoming adjoint $\bar{v} = \partial L / \partial v$ by its local Jacobian and passes the result to its inputs, summing contributions at fan-out; a single reverse traversal yields gradients for all parameters, which is why backprop scales to billion-parameter models.
(3) Dynamic vs Static Graphs: PyTorch’s define-by-run graph follows arbitrary Python control flow and debugs like ordinary code; static graphs (TensorFlow 1, exported ONNX) trade that flexibility for ahead-of-time whole-graph optimization and portable deployment.

Forward computational graph of L = (wx + b - y)^2: input boxes x=2, w=3 feed a multiply node producing u=6, then an add node with b=1 producing z=7, a subtract node with y=5 producing e=2, and a square node producing L=4

Figure 1: Forward pass of $L = (wx + b - y)^2$ with $x=2,\ w=3,\ b=1,\ y=5$ : each op consumes its input tensors, emits one value, and caches it for the backward sweep.

The backward sweep traverses the same graph in reverse. Starting from the seed $\bar{L} = 1$ , each node applies the chain rule locally: the square node returns $2e = 4$ , the subtraction forwards $+4$ to $z$ and $-4$ to $y$ , the addition copies its adjoint to both $u$ and $b$ , and the multiplication swaps operands, delivering $\bar{w} = 4 \cdot x = 8$ and $\bar{x} = 4 \cdot w = 12$ . No node needs global knowledge of the loss; correctness of the whole sweep follows from composing local derivatives edge by edge. The figure shows $\bar{x}$ and $\bar{y}$ to make the mechanics uniform, but a framework skips those branches when the input and target carry requires_grad=False, so only $\bar{w}$ and $\bar{b}$ are actually materialized. When a value feeds several nodes (fan-out), its adjoint is the sum of the contributions along each outgoing edge, which is exactly the multivariate chain rule.

Backward pass over the same graph with arrows reversed: adjoint labels on every edge showing e-bar=4, z-bar=4, y-bar=-4, u-bar=4, b-bar=4, w-bar=4 times 2=8, x-bar=4 times 3=12, seeded by L-bar=1

Figure 2: The same graph in reverse: each node multiplies the incoming adjoint by its local derivative, so one sweep accumulates $\bar{w} = 8$ and $\bar{b} = 4$ at the leaves.

The graph pays for itself well beyond gradients. Since it records exactly which intermediates feed the loss, frameworks can plan memory: reverse mode must retain cached activations, so training a depth- $N$ network costs $O(N)$ activation memory, and gradient checkpointing cuts this to $O(\sqrt{N})$ by storing only boundary activations and recomputing each segment during the backward pass, at the price of roughly one extra forward evaluation. A static snapshot of the same graph lets compilers fuse ops, pick kernel layouts, and place tensors across devices, and exporters freeze it into a portable artifact. The trade-off is flexibility: a define-by-run graph is rebuilt every iteration, so data-dependent if-statements and loops simply work, while a static graph must capture control flow symbolically, which is why TensorFlow 2 defaulted to eager execution and recovers graph benefits selectively through tf.function tracing.

Mathematical Formulation:
$L = (w\,x + b - y)^2$
$\bar{v}_i = \sum_{j \,\in\, \mathrm{children}(i)} \bar{v}_j \, \frac{\partial v_j}{\partial v_i}$

Where:

$L$ is the scalar loss; $w, x, b, y$ are the weight, input, bias, and target scalars of the running example.
$v_i$ is the output value of node $i$ , and $\bar{v}_i = \partial L / \partial v_i$ is its adjoint, the gradient of the loss with respect to that node’s output.
$\mathrm{children}(i)$ are the nodes that consume $v_i$ ; each child contributes its incoming adjoint times the local partial $\partial v_j / \partial v_i$ , and the results sum, which is the multivariate chain rule.

Aspect	Dynamic Graph (PyTorch Eager)	Static or Compiled (TF1, ONNX, torch.compile)
Graph Built	Every forward pass, fresh per iteration	Once, ahead of execution or by tracing
Control Flow	Native Python if/for; graph follows the data	Symbolic capture; data-dependent branches are hard
Debugging	Standard Python debugger on eager values	Graph-level tooling; values not materialized eagerly
Optimization	Op-by-op dispatch, limited cross-op rewrites	Whole-graph fusion, layout, device placement
Deployment	Needs an explicit export step	Graph is already a portable artifact

August 12, 2026

ML0100 Duplicate Detection

How do you detect and handle duplicates in a dataset, from exact matching to NVIDIA NeMo Curator’s GPU-accelerated fuzzy deduplication?

Answer

Duplicate detection has two levels: exact duplicates (identical rows or records) and near-duplicates (small edits, reformatting, or paraphrases). Exact duplicates are found by hashing: compute a hash (MD5, SHA) of each row or document, group by hash, and flag collisions in O(n) time. Near-duplicates are harder and require similarity-based methods: MinHash converts each document into a set of shingles and estimates the Jaccard similarity via shared min-hash signatures, then Locality-Sensitive Hashing (LSH) buckets signatures so that similar documents collide in the same bucket, reducing the O(n^2) all-pairs comparison to near-linear. NVIDIA NeMo Curator provides GPU-accelerated fuzzy deduplication using MinHash and LSH, and the research framework SEDD pushes this further by replacing data shuffling with streaming, reaching 158x over CPU-based SlimPajama tooling and 7.8x over NeMo Curator while deduplicating 1.2 trillion tokens in 3 hours on a 32-GPU cluster. LSHBloom replaced the expensive LSH index with Bloom filters, achieving 12x speedup and 18x less disk space at the same deduplication quality.

(1) Exact Duplicates: hash each row or document, group by hash, flag collisions; O(n) time, zero false positives; handles identical rows, copy-paste records, and repeated data entries.
(2) Near-Duplicates (MinHash + LSH): shingle each document, compute MinHash signatures (k random permutations), bucket via LSH so similar documents collide, then verify candidate pairs by exact Jaccard similarity; reduces O(n^2) all-pairs to near-linear.
(3) Production Scale: NVIDIA NeMo Curator supplies the GPU MinHash LSH baseline, and SEDD deduplicates 1.2 trillion tokens in 3 hours on a 32-GPU cluster (158x over CPU tooling, 7.8x over NeMo Curator) while keeping Jaccard similarity above 0.95; LSHBloom replaces the LSH index with Bloom filters for 12x speedup and 18x less disk space at internet scale.

Two-tier pipeline: top tier shows exact dedup via hashing (hash each row, group by hash, flag collisions); bottom tier shows near-duplicate dedup via shingling, MinHash signatures, LSH bucketing, and Jaccard verification

Figure 1: The two-tier deduplication pipeline: exact duplicates are found by hashing and grouping (O(n), zero false positives), while near-duplicates use shingling, MinHash signatures, and LSH bucketing to reduce the O(n^2) all-pairs comparison to near-linear.

Duplicates matter because they inflate evaluation metrics and bias the model. If the same example appears in both training and test sets, the model memorizes it during training and gets it right at test time, producing an accuracy estimate that will not hold on truly unseen data. Even within the training set alone, duplicates cause the model to over-weight those examples, effectively up-sampling them. The handling strategy depends on the duplicate type: exact duplicates within one split should be removed; cross-split duplicates (train-test contamination) require deduplication before the split; near-duplicates in LLM training corpora are removed at scale to prevent memorization and recitation, which is why NVIDIA NeMo Curator and HuggingFace’s datasets deduplication tools exist. For structured tabular data, pandas’ drop_duplicates() handles exact duplicates, and fuzzy matching libraries (recordlinkage, dedupe) handle near-duplicates via string similarity. For text corpora at scale, MinHash LSH is the standard, and GPU acceleration made it practical for trillion-token pretraining datasets.

Mathematical Formulation:
$J(A, B) = \frac{|A \cap B|}{|A \cup B|}$
$P[\min(\pi(A)) = \min(\pi(B))] = J(A, B)$
$s(x) = (h_1(x), h_2(x), \ldots, h_k(x))$

Where:

$J(A, B)$ is the Jaccard similarity between two sets $A$ and $B$ (e.g., the shingle sets of two documents); near-duplicates have Jaccard similarity above a threshold (typically 0.8-0.9).
$\pi$ is a random permutation of the universe; the MinHash property states that the probability of two sets sharing the same minimum hash value equals their Jaccard similarity, so averaging over $k$ permutations gives an unbiased estimate.
$s(x)$ is the MinHash signature of document $x$ : a vector of $k$ minimum hash values. LSH buckets signatures so that similar signatures (and thus similar documents) collide with high probability, reducing candidate pairs from O(n^2) to near-linear.

Method	Duplicate Type	Complexity	Scale
Hashing (exact)	Exact duplicates	O(n)	Any (pandas drop_duplicates)
MinHash + LSH	Near-duplicates (text)	O(nk) + candidate verification	Millions of documents
GPU MinHash LSH (NeMo Curator, SEDD)	Near-duplicates (GPU)	GPU-parallelized MinHash LSH	1.2T tokens in 3 hours (32 GPUs, SEDD)
LSHBloom	Near-duplicates (Bloom)	O(nk), 12x faster than LSH	Internet-scale (billions of docs)

August 12, 2026

ML0099 Noisy Labels

How would you detect and correct inconsistent or noisy labels in training data?

Answer

Noisy labels are training labels that are wrong or inconsistent: a dog image labeled “cat,” a sentiment review labeled “positive” that is clearly negative, or the same entity labeled differently by different annotators. Detection starts with Confident Learning, a model-agnostic framework that uses cross-validated predicted probabilities to estimate the joint distribution between noisy labels and true labels, then flags examples where the model’s confident prediction disagrees with the given label. The open-source Cleanlab library implements this and has surfaced over 100,000 label issues in ImageNet. Correction ranges from simple relabeling (flip the label if the model is very confident) to weak supervision (using foundation models as labeling functions and denoising their outputs via a label model), to noise-robust training (CANOLA, 2026, achieves 19-52% improvement over SOTA label correction via noise-aware learning and iterative soft label refinement). Confident Learning is now taught as a core data-centric AI technique, and the original framework was published in JAIR 2021.

(1) Detection via Confident Learning: cross-validate the model to get out-of-sample predicted probabilities, estimate the joint distribution of noisy vs true labels, and flag examples where the predicted label confidently disagrees with the given label; Cleanlab surfaced 100,000+ label issues in ImageNet this way.
(2) Correction via Weak Supervision: foundation models (Llama 3.1, GPT-4, CLIP) serve as labeling functions that produce noisy labels, then a label model denoises them by learning each function’s precision and correlations, achieving 19.5% error reduction over zero-shot on the WRENCH benchmark.
(3) Correction via Noise-Aware Training: instead of hard relabeling, use soft labels (probability distributions over classes) refined iteratively during training; CANOLA (2026) achieves 19-52% relative improvement over SOTA, and the Relabeler framework (2026) achieves 58% improvement in label correction precision by jointly leveraging local and global data relationships.

Pipeline: cross-validated model produces out-of-sample probabilities, Confident Learning estimates the noisy-vs-true joint distribution and flags disagreements, then correction routes to relabel, weak supervision, or noise-aware soft-label training

Figure 1: The noisy-label pipeline: cross-validated predictions feed Confident Learning to detect label errors via the noisy-vs-true joint distribution, then correction routes to manual relabeling, weak supervision, or noise-aware soft-label training.

The detection step relies on a key insight: if a well-trained model confidently predicts “dog” for an image labeled “cat,” the label is more likely wrong than the model. Confident Learning formalizes this by estimating the joint distribution $P(\tilde{y}, y^*)$ of observed (noisy) labels and true labels using the cross-validated probability matrix, then identifying the most likely mislabeled examples via pruning, counting, and ranking. This requires no hyperparameters and works with any classifier. For correction, the simplest approach is to remove or relabel the flagged examples if you have access to a human annotator. If human annotation is expensive, weak supervision lets you define labeling functions as natural-language prompts to foundation models, and the label model combines their outputs by learning each function’s accuracy and correlations. For training-time correction, noise-aware methods like CANOLA replace hard labels with soft labels (class probability distributions) that are refined iteratively, avoiding the hard commitment of relabeling while reducing the impact of noise. A 2026 paper on spectral signatures showed that the tail index of eigenvalue distributions at network bottleneck layers predicts test accuracy under label noise with R-squared of 0.984, providing a diagnostic that identifies 9% noise in CIFAR-10N with 3% error.

A three-by-three matrix for classes cat, dog and bird: rows are the given noisy labels and columns the model's confident predictions, each row summing to 1, with green diagonal cells for agreement and red off-diagonal cells marking likely mislabeled examples

Figure 2: Row-normalized rates of confident predictions given each noisy label: off-diagonal mass is the estimated per-class mislabeling rate, and Confident Learning turns these confident counts into the noisy-versus-true joint distribution used to prune, count, and rank label errors.

Mathematical Formulation:
$\hat{P}(\tilde{y}=i,\, y^*=j) = \frac{1}{n}\sum_{k=1}^{n} \mathbb{1}[\tilde{y}_k = i] \cdot \mathbb{1}[\hat{p}_j(x_k) \geq t_j]$
$\hat{y}^*_k = \arg\max_j \hat{p}_j(x_k)$

Where:

$\hat{P}(\tilde{y}=i,\, y^*=j)$ is the estimated joint distribution of the noisy label $\tilde{y}$ and the true label $y^*$ ; $\hat{p}_j(x_k)$ is the cross-validated predicted probability of class $j$ for example $k$ .
$t_j$ is the average per-class confidence threshold (the average self-confidence for class $j$ ); examples where $\hat{p}_j(x_k) \geq t_j$ but $\tilde{y}_k \neq j$ are flagged as likely mislabeled.
$\hat{y}^*_k$ is the Confident Learning estimate of the true label: the argmax of the cross-validated probabilities, used for relabeling or soft-label assignment in noise-aware training.

Method	Approach	Requires Labels?
Confident Learning (Cleanlab)	Cross-validated probabilities estimate noisy-vs-true joint distribution	Noisy labels only
Weak Supervision	Foundation models as labeling functions; label model denoises	No labels needed; optional ground truth
Noise-Aware Training (CANOLA)	Iterative soft-label refinement during training	Noisy labels only
Spectral Signatures (2026)	Eigenvalue tail index at bottleneck layers predicts noise level	Diagnostic; no labels needed

August 12, 2026

ML0098 Outlier Detection

How do you detect outliers in a dataset, from simple statistical rules to Isolation Forest and beyond?

Answer

Outlier detection ranges from simple statistical rules to model-based methods, and the right choice depends on the data dimensionality, distribution, and whether you have labels. For univariate data, the two standard rules are the z-score method (flag points more than 2 or 3 standard deviations from the mean) and the IQR rule (flag points below Q1 – 1.5*IQR or above Q3 + 1.5*IQR); the IQR rule is more robust because the mean and std are themselves inflated by outliers. For multivariate and high-dimensional data, statistical rules break down, and model-based methods take over: Isolation Forest isolates anomalies by random splits (outliers need fewer splits to isolate), Local Outlier Factor (LOF) compares local density to neighbors, and autoencoders flag points with high reconstruction error. Production data quality tools combine Isolation Forest for anomaly detection with robust statistics (median plus 5 times the robust standard deviation) for per-column outlier flagging, and a 2024 PMLR paper introduced HPOD, the first continuous hyperparameter search method for unsupervised outlier detection, achieving 58% and 66% improvement over default LOF and Isolation Forest hyperparameters.

(1) Statistical (Univariate): z-score flags points beyond 2-3 sigma from the mean; IQR flags points outside Q1 – 1.5*IQR to Q3 + 1.5*IQR; IQR is robust to the outliers themselves, z-score is not.
(2) Model-Based (Multivariate): Isolation Forest isolates anomalies with fewer random splits (O(n log n), scales well); LOF compares local density to k-nearest neighbors (O(n^2) naive, good for local anomalies); autoencoders flag high reconstruction error (good for complex distributions but needs training data).
(3) Production (Data Quality Tools + HPOD): production data quality reports use Isolation Forest with anomaly scores plus robust per-column statistics (median plus or minus 5*RSTD); HPOD (PMLR 2024) capitalizes on prior benchmark performance to tune LOF and Isolation Forest hyperparameters without labels, improving detection by 58-66%.

Four panels: z-score shows a bell curve with the tails beyond plus and minus 3 sigma shaded and labelled flagged; IQR shows a horizontal boxplot with outliers drawn as X markers past the whiskers; Isolation Forest shows a two-feature scatter where far-from-centre points are marked as isolated; LOF shows two dense clusters with a single low-local-density point between them marked

Figure 1: Four outlier detection methods: z-score flags points beyond 3 sigma (sensitive to outliers itself), IQR uses the interquartile range (robust), Isolation Forest isolates anomalies with few random splits, and LOF compares local density to neighbors.

The practical workflow is to start simple and escalate. First, visualize: boxplots per feature and scatter plots of feature pairs reveal obvious outliers. Second, apply the IQR rule per feature for a quick, robust baseline. Third, for multivariate outliers that no single feature reveals, run Isolation Forest (scales to high dimensions, O(n log n), no distributional assumption) and inspect the anomaly score distribution. Fourth, if anomalies are local (dense regions with sparse sub-regions), use LOF. Fifth, for complex nonlinear structure (images, text embeddings), train an autoencoder on the majority class and flag high reconstruction error. Always investigate flagged points before removing them: an outlier may be a genuine rare event (fraud, anomaly) rather than bad data, and domain expertise is the final arbiter. Production data quality tools automate this by generating a Data Quality and Insights Report that combines Isolation Forest anomaly scores with per-column robust statistics and time-series decomposition for temporal anomalies.

Mathematical Formulation:
$z_i = \frac{x_i - \mu}{\sigma},\quad |z_i| > 3 \Rightarrow \text{outlier}$
$\text{IQR} = Q_3 - Q_1,\quad x_i \leq Q_1 - 1.5\,\text{IQR} \;\text{or}\; x_i \geq Q_3 + 1.5\,\text{IQR} \Rightarrow \text{outlier}$
$s(x, n) = 2^{-\mathbb{E}[h(x)]\,/\,c(n)}$

Where:

$z_i$ is the z-score of point $x_i$ ; $\mu$ and $\sigma$ are the mean and standard deviation of the feature. The z-score method is not robust because outliers inflate $\mu$ and $\sigma$ , masking themselves.
$Q_1$ and $Q_3$ are the 25th and 75th percentiles; the IQR rule is robust because percentiles are unaffected by extreme values. A related robust rule used in production: flag values outside median plus or minus 5 times the robust standard deviation (RSTD).
$s(x, n)$ is the Isolation Forest anomaly score. $\mathbb{E}[h(x)]$ is the average path length needed to isolate $x$ across the trees and $c(n)$ is the expected path length for $n$ points, so outliers isolate in fewer splits, which drives the exponent up and pushes $s$ toward 1, while normal points sit near 0.5 or below.

Method	Type	Complexity	Best For
z-score	Univariate, parametric	O(n)	Approximately normal data
IQR rule	Univariate, robust	O(n log n)	Skewed data, robust baseline
Isolation Forest	Multivariate, model-based	O(n log n)	High-dimensional, scalable
LOF	Multivariate, density-based	O(n^2) naive	Local anomalies, varying density
Autoencoder	Multivariate, reconstruction	Training + O(nd) inference	Complex nonlinear (images, embeddings)

August 12, 2026

ML0097 Data Leakage

What is data leakage, and what are the most common ways it occurs in ML pipelines?

Answer

Data leakage is when information from outside the training set enters the model training process, giving the model knowledge it would not have at prediction time. The result is inflated training and validation metrics that collapse when the model meets real test data, because the leaked signal is absent in production. The most common forms are target leakage (a feature that encodes the label, like “post-purchase amount” in a churn model), train-test contamination (fitting a preprocessing step on the full dataset before splitting, so test statistics bleed into training), and temporal leakage (using future data to predict the past, like computing rolling statistics that include the target time step). At production scale, data governance frameworks use Information Flow Control to enforce purpose limitations on data, blocking problematic data transfers before they occur, and require data lineage tracking and policy compliance checks at training job configuration time to prevent training-data contamination.

(1) Target Leakage: a feature is derived from the label or from information unavailable at prediction time (e.g., “days since last purchase” computed after the churn window closes); the model learns a shortcut that disappears in production.
(2) Train-Test Contamination: fitting scalers, imputers, feature selectors, or PCA on the full dataset before the train-test split, so test-set statistics leak into training; the fix is to fit preprocessing only on the training fold and apply it to test, ideally via a scikit-learn Pipeline.
(3) Temporal Leakage: in time-series, using future observations to compute features for past predictions (e.g., a rolling mean that includes the target timestamp, or shuffling time-series data before splitting); the fix is a time-based split and forward-only feature computation, plus production-scale data governance frameworks for purpose limitation and lineage tracking.

Three panels: target leakage shows a feature arrow from the label into the feature set; train-test contamination shows a scaler fitted on all data before splitting; temporal leakage shows a rolling window that includes future data points relative to the prediction time

Figure 1: The three most common leakage types: target leakage (a feature derived from the label), train-test contamination (preprocessing fitted before the split), and temporal leakage (future data in past feature computation).

Detection starts with a sanity check: if validation or test performance is suspiciously high compared to production, or if a single feature has outsized importance, suspect leakage. A practical diagnostic is permutation importance on a held-out set: a leaked feature will show extreme importance because shuffling it destroys the shortcut. For time-series, always use a time-based split (train on the past, test on the future) and never shuffle before splitting. For preprocessing, use a scikit-learn Pipeline or ColumnTransformer that fits only on the training fold inside cross-validation, so scalers, imputers, and encoders never see test data. For feature engineering, audit each feature for availability at prediction time: if a feature depends on data that arrives after the prediction, it is leaked. At scale, production data governance frameworks propagate privacy annotations across millions of daily data flows and block problematic transfers before they occur, while requiring data lineage tracking and policy compliance checks at training job configuration time.

Mathematical Formulation:
$\text{leakage} \iff I(Y_{\text{test}};\, X_{\text{train}}) > 0$
$\hat{f} = \arg\min_{f} \sum_{i \in \text{train}} \ell(y_i, f(x_i))$

Where:

$I(Y_{\text{test}};\, X_{\text{train}})$ is the mutual information between the test labels and the training features; leakage means this is positive, i.e., the training features contain information about the test labels that should not be there.
$\hat{f}$ is the model trained on the training set; the goal is that $\hat{f}$ depends only on $X_{\text{train}}, Y_{\text{train}}$ and nothing from the test set, including test-set statistics used in preprocessing.
Target leakage is the case where $X_{\text{train}}$ itself contains a feature derived from $Y_{\text{train}}$ (or $Y_{\text{test}}$ ), so the model learns $Y$ from $X$ via a shortcut. Temporal leakage is the case where a feature $X_t$ is computed from data observed at some later time $t' > t$ , which is unavailable when the prediction must actually be made.

Leakage Type	Mechanism	Prevention
Target Leakage	Feature derived from label or post-prediction data	Audit feature availability at prediction time; remove derived features
Train-Test Contamination	Preprocessing fitted on full dataset before split	Fit preprocessing only on training fold; use sklearn Pipeline
Temporal Leakage	Future data used in past feature computation	Time-based split; forward-only rolling features; no shuffling
Duplicate Leakage	Same record in both train and test splits	Deduplicate before splitting; use GroupKFold for grouped data

August 12, 2026

ML0096 Maximum Likelihood Estimation

What is maximum likelihood estimation (MLE), and how does it connect to the cross-entropy loss used in modern neural network training?

Answer

Maximum likelihood estimation is a method for estimating the parameters of a statistical model by finding the parameter values that make the observed data most probable. Given a dataset and a parametric model, MLE chooses the parameters that maximize the likelihood function: the joint probability of the observed data under the model. Equivalently, we maximize the log-likelihood, which converts the product of probabilities into a sum of log-probabilities, making optimization tractable and numerically stable. MLE is the foundation of most modern ML training: an ICLR 2025 blog post explicitly derives the standard classification cross-entropy loss from the MLE principle, showing that minimizing cross-entropy is equivalent to maximizing the conditional log-likelihood of the training data under the model. For a simple coin flip example, if you observe 3 heads in 4 flips, the MLE of the heads probability is simply 3/4, the sample proportion.

(1) Likelihood vs Probability: probability is the chance of data given fixed parameters; likelihood is the same function viewed as a function of parameters with the data fixed, and MLE finds the parameters that maximize it.
(2) Log-Likelihood Trick: taking the log converts the product $\prod P(x_i \mid \theta)$ into the sum $\sum \log P(x_i \mid \theta)$ , which is easier to differentiate, numerically stable, and decomposes additively over data points for stochastic gradient descent.
(3) Connection to Production ML: an ICLR 2025 blog derives cross-entropy loss as the negative log-likelihood under a categorical model, so training a classifier with cross-entropy is MLE; logistic regression is MLE under a Bernoulli model, and linear regression with squared error is MLE under a Gaussian noise model.

Top panel: the likelihood curve L(theta) = theta^3 (1-theta) over theta from 0 to 1, peaking at theta = 0.75 where a dashed line and diamond marker label the MLE; bottom panel: the log-likelihood 3 ln(theta) + ln(1-theta), peaking at the same 0.75

Figure 1: Coin flip MLE: with 3 heads in 4 flips, the likelihood $L(\theta) = \theta^3(1-\theta)$ and log-likelihood $3\ln\theta + \ln(1-\theta)$ both peak at $\hat{\theta} = 3/4 = 0.75$ , the sample proportion of heads.

The coin flip example makes the principle concrete. Suppose you flip a coin 4 times and observe 3 heads and 1 tail. The coin has an unknown probability $\theta$ of heads. The likelihood of observing this data is $L(\theta) = \theta^3 (1 - \theta)^1$ , and the log-likelihood is $\ell(\theta) = 3 \ln \theta + \ln(1 - \theta)$ . Taking the derivative and setting it to zero gives $3/\theta - 1/(1-\theta) = 0$ , which solves to $\hat{\theta} = 3/4$ . This is the sample proportion, and it is the MLE because the likelihood is maximized there. The same principle scales to neural networks: the cross-entropy loss $L = -\frac{1}{N}\sum_i \log q_\theta(y_i \mid x_i)$ is the negative average log-likelihood, and gradient descent on this loss is MLE for the network’s parameters. A 2024 arXiv paper established finite-sample guarantees for MLE in logistic regression, showing the sample complexity depends on both dimension and signal strength, with distinct regimes at different signal-to-noise ratios.

Mathematical Formulation:
$\hat{\theta}_{\mathrm{MLE}} = \arg\max_{\theta}\; \prod_{i=1}^{N} P(x_i \mid \theta)$
$\ell(\theta) = \sum_{i=1}^{N} \log P(x_i \mid \theta)$
$\hat{\theta}_{\mathrm{MLE}} = \arg\max_{\theta}\; \ell(\theta)$

Where:

$\hat{\theta}_{\mathrm{MLE}}$ is the parameter value that maximizes the likelihood; $P(x_i \mid \theta)$ is the probability of observation $x_i$ under the model with parameters $\theta$ .
$\ell(\theta)$ is the log-likelihood, which converts the product into a sum for tractability; maximizing $\ell$ is equivalent to maximizing the likelihood because log is monotonically increasing.
For the coin flip: $\ell(\theta) = N_H \ln \theta + N_T \ln(1 - \theta)$ , and setting $d\ell/d\theta = 0$ gives $\hat{\theta} = N_H / (N_H + N_T)$ , the sample proportion. For classification: $\ell = \sum_i \log q_\theta(y_i \mid x_i)$ , and minimizing $-\ell/N$ is the cross-entropy loss.

Model	Likelihood	MLE Solution	Equivalent Loss
Bernoulli (coin)	theta^NH (1-theta)^NT	NH / (NH + NT)	Binary cross-entropy
Gaussian (regression)	prod N(yi \| f(xi), sigma^2)	Least squares solution	Mean squared error
Categorical (classifier)	prod q_theta(yi \| xi)	Gradient descent on -log q	Cross-entropy loss

August 12, 2026

ML0095 Confidence Interval

What is a confidence interval, and what is the correct interpretation of a 95% confidence interval, as used in sequential A/B testing and experimentation platforms?

Answer

A confidence interval is a range computed from sample data that is designed to contain an unknown population parameter with a specified probability under repeated sampling. The correct frequentist interpretation of a 95% confidence interval is: if you repeated the experiment many times, computing a new interval from each sample, about 95% of those intervals would contain the true parameter. It is a statement about the procedure, not about any single interval. Once data is observed, the true parameter is either inside your specific interval or not (probability 0 or 1); you cannot say “there is a 95% probability the parameter is in this interval,” because that is a Bayesian credible interval interpretation requiring a prior. Sequential A/B testing platforms use “anytime-valid” confidence sequences that maintain this coverage guarantee across continuous monitoring, and asymmetric experiment designs treat confidence interval width as a design parameter controlled through sample allocation between test and control groups.

(1) Procedure vs Single Interval: the 95% refers to the long-run coverage rate of the interval-construction procedure across hypothetical repeated samples, not the probability that the true parameter is in any one observed interval.
(2) Frequentist vs Bayesian: saying “95% probability the parameter is in this interval” is a Bayesian credible interval statement; a 2024 paper showed this distinction has real business impact, as frequentist estimators overstate A/B test effect sizes (winner’s curse) while Bayesian estimators with correct priors do not.
(3) Production Use: anytime-valid confidence sequences enable canary testing so the 95% coverage holds throughout continuous monitoring, not just at a pre-specified endpoint; asymmetric experiment designs optimize CI width by nonlinearly increasing control group size when shrinking test group size (40% test reduction requires 3x control increase).

Twenty horizontal confidence intervals computed from repeated samples, with a vertical line at the true parameter value; about 19 of 20 intervals cross the true value while one misses it, illustrating 95% coverage

Figure 1: The frequentist interpretation: across 20 repeated experiments, about 19 of the 95% confidence intervals contain the true parameter (vertical line); the 95% is a property of the procedure, not of any single interval.

The width of a confidence interval depends on three things: the confidence level (higher confidence means wider intervals), the sample size (larger n means narrower intervals), and the variability of the data (higher variance means wider intervals). For a mean with known variance, the 95% CI is the sample mean plus or minus 1.96 standard errors. When the variance is unknown and estimated from the sample, the t-distribution replaces the normal, giving slightly wider intervals for small samples. A critical production issue is peeking: repeatedly checking whether a confidence interval excludes zero and stopping when it does inflates the false positive rate, because each check is a separate test. Anytime-valid confidence sequences solve this by using a test statistic whose coverage guarantee holds uniformly over time, so you can monitor continuously without breaking the 95% promise. Conformal prediction (2024-2025) extends the idea to distribution-free prediction intervals for ML models, providing finite-sample coverage without normality assumptions.

Mathematical Formulation:
$\bar{x} \pm z_{\alpha/2}\, \frac{\sigma}{\sqrt{n}}$
$P\!\left(\bar{X} - z_{\alpha/2}\frac{\sigma}{\sqrt{n}} \leq \mu \leq \bar{X} + z_{\alpha/2}\frac{\sigma}{\sqrt{n}}\right) = 1 - \alpha$

Where:

$\bar{x}$ is the sample mean, $\sigma$ is the population standard deviation (replaced by sample standard deviation $s$ with the t-distribution when unknown), and $n$ is the sample size.
$z_{\alpha/2}$ is the critical value of the standard normal at level $\alpha/2$ ; for 95% confidence, $\alpha = 0.05$ and $z_{0.025} = 1.96$ .
The probability statement is over the random interval (which depends on $\bar{X}$ ), not over the fixed parameter $\mu$ ; this is the frequentist distinction. The interval either covers $\mu$ or not for any single sample, but the procedure covers it $1 - \alpha$ of the time across repeated samples.

Concept	Frequentist CI	Bayesian Credible Interval
Interpretation	95% of repeated-sample intervals contain the true value	95% probability the parameter is in this interval
Requires Prior?	No	Yes (prior over the parameter)
Parameter Is	Fixed, unknown constant	Random variable with a distribution
Production Note	Frequentist: winner’s curse overstates effect sizes	Bayesian: correct priors avoid winner’s curse

August 12, 2026

ML0094 GMM vs K-Means

How does Gaussian Mixture Model clustering differ from k-means, and why do production systems like speaker diarization prefer soft assignments?

Answer

A Gaussian Mixture Model is a probabilistic clustering model that represents the data as a weighted sum of K Gaussian distributions, each with its own mean, covariance, and mixing weight. Where k-means assigns each point to exactly one cluster (hard assignment) by nearest centroid, a GMM assigns each point a responsibility (posterior probability) for each cluster (soft assignment), so a point on the boundary between two clusters can be 60% in one and 40% in another. K-means is the limiting case of a GMM with shared spherical covariances as that shared variance goes to zero, which collapses the E-step posterior to an argmax. The GMM is fit by Expectation-Maximization: the E-step computes responsibilities, the M-step updates means, covariances, and mixing weights, iterating until convergence. Production systems use GMMs where uncertainty matters: joint diarization and separation systems pair a complex Angular Central Gaussian Mixture Model for source separation with a von Mises-Fisher mixture for diarization, so overlapping speech can be attributed to multiple speakers at once, and Flash-GMM (2026) made GMM training viable at 100x larger scale via fused Triton kernels.

(1) Hard vs Soft Assignment: k-means assigns each point to one centroid (argmin distance); GMM assigns a probability vector (responsibilities) across all clusters, so border points express uncertainty instead of being forced into one group.
(2) Spherical vs Elliptical Clusters: k-means assumes spherical, equally sized clusters (isotropic variance); GMM allows each cluster to have its own full covariance matrix, capturing elliptical and differently sized clusters.
(3) Production Soft Clustering: speaker diarization uses cACGMM and vMFMM mixture models so frame-level embeddings can belong to multiple speakers during overlap, and Flash-GMM’s fused Triton kernel achieved 20x speedup and 100x larger dataset scale, making soft GMM clustering a viable drop-in for k-means in approximate nearest-neighbor search with 2-12 point recall@10 gains.

Two panels: left shows k-means hard assignment with Voronoi boundaries and each point colored as one cluster; right shows GMM soft assignment with elliptical cluster contours and a border point labeled with 60/40 responsibility split

Figure 1: k-means forces each point into one cluster via nearest-centroid (hard assignment with spherical Voronoi cells), while a GMM assigns posterior probabilities (soft assignment) and can model elliptical clusters with per-component covariance matrices.

The EM algorithm for GMMs iterates two steps. In the E-step, each point’s responsibility for cluster k is computed as the posterior $\gamma_{ik} = \pi_k \mathcal{N}(x_i \mid \mu_k, \Sigma_k) / \sum_j \pi_j \mathcal{N}(x_i \mid \mu_j, \Sigma_j)$ . In the M-step, the parameters are updated: the mean is the responsibility-weighted average, the covariance is the responsibility-weighted scatter, and the mixing weight is the average responsibility. Each iteration is guaranteed never to decrease the log-likelihood, but it converges only to a local optimum, so multiple random restarts are standard. The key trade-off versus k-means is cost: k-means is O(nKd) per iteration with a simple distance computation, while GMM is O(nKd^2) per iteration because of the multivariate Gaussian PDF with full covariance, plus the matrix inversion in the PDF. Flash-GMM (2026) addressed this by eliminating the full responsibility matrix from GPU memory, reducing memory from O(nK) to O(KD) and enabling 100x larger datasets on a single GPU.

EM iteration flow: E-step computes responsibilities as posterior probabilities from current parameters, M-step updates means, covariances, and mixing weights from weighted statistics, with an arrow looping back to E-step until convergence

Figure 2: The EM loop for GMMs: the E-step computes soft responsibilities from current parameters, the M-step updates means, covariances, and mixing weights from responsibility-weighted statistics, and the cycle repeats until log-likelihood converges.

Mathematical Formulation:
$p(x) = \sum_{k=1}^{K} \pi_k\, \mathcal{N}(x \mid \mu_k, \Sigma_k)$
$\gamma_{ik} = \frac{\pi_k\, \mathcal{N}(x_i \mid \mu_k, \Sigma_k)}{\sum_{j=1}^{K} \pi_j\, \mathcal{N}(x_i \mid \mu_j, \Sigma_j)}$
$\mu_k = \frac{\sum_i \gamma_{ik}\, x_i}{\sum_i \gamma_{ik}},\quad \Sigma_k = \frac{\sum_i \gamma_{ik}\,(x_i - \mu_k)(x_i - \mu_k)^\top}{\sum_i \gamma_{ik}}$

Where:

$\pi_k$ is the mixing weight of cluster $k$ (prior probability, sums to 1), $\mu_k$ is its mean, and $\Sigma_k$ is its covariance matrix.
$\gamma_{ik}$ is the responsibility of cluster $k$ for point $i$ : the posterior probability that point $i$ was generated by component $k$ . This is the soft assignment that k-means replaces with a hard argmax.
The M-step updates are weighted by responsibilities: $\mu_k$ is the responsibility-weighted mean, $\Sigma_k$ is the responsibility-weighted covariance, and $\pi_k = \frac{1}{n}\sum_i \gamma_{ik}$ is the average responsibility. k-means is the limit when all $\Sigma_k = \sigma^2 I$ and $\sigma \to 0$ , collapsing responsibilities to 0 or 1.

Property	k-means	GMM
Assignment	Hard (argmin distance)	Soft (posterior responsibilities)
Cluster Shape	Spherical, equal variance	Elliptical, per-component covariance
Objective	Minimize within-cluster SSE	Maximize log-likelihood
Cost per Iteration	O(nKd)	O(nKd^2) with full covariance
Production Use	Vector quantization, image compression	Speaker diarization (cACGMM), ANN search (Flash-GMM)

August 12, 2026