Category: Medium

ML0086 Drift Detection in Production

How would you detect data drift in a deployed machine learning model?

Answer

You monitor in layers, because no single statistic sees everything. Layer one watches the inputs: per-feature distribution statistics (PSI, KS distance) of live traffic against the training baseline. Layer two watches the outputs: the prediction distribution and confidence profile, which catch label shift and model-side anomalies. Layer three estimates or measures performance: delayed ground truth when it arrives, and label-free estimators like NannyML’s CBPE in the gap. Each layer has thresholds with alerting, and alerts route to a decision: investigate the pipeline, retrain, or roll back. The whole pattern is standardized in AWS SageMaker Model Monitor: a baseline job computes statistics and constraints from training data, scheduled jobs compare live captures, and violations fire CloudWatch alarms.

(1) Baseline vs Window: freeze reference statistics from the training distribution, then compute drift metrics over sliding or scheduled production windows; the metric is a distance between two empirical distributions, not a model property.
(2) What Each Layer Sees: input stats see covariate shift, output stats see label shift and confidence collapse, and only ground truth (or calibrated estimation) sees concept drift; NannyML’s CBPE handles covariate shift but explicitly not concept drift.
(3) Production Pattern (AWS): Model Monitor’s built-in container (Deequ on Spark) emits statistics.json and constraints.json from a baseline, then scheduled monitoring jobs check distribution distance (linf_simple / two-sample KS via linf_robust, LInfinity or ChiSquared for categoricals) and write violation reports that drive alarms. AWS has announced that Model Monitor closes to new customers on 30 July 2026, with existing customers unaffected, but this baseline/constraints/violations design remains the reference pattern.

PSI over weekly production windows: values hover low for months, cross the moderate threshold, then breach the significant threshold where an alarm marker fires

Figure 1: A drift monitor in action: per-window PSI against the training baseline crosses the moderate band (0.1) and then the significant band (0.25), which is where alerting and retraining triggers fire.

Design details separate a working monitor from alert fatigue. Thresholds must account for sample size (small windows inflate every distance metric), per-feature tests need multiplicity control or aggregation, and the window length trades detection latency against statistical power. When labels are delayed, record them when they land and backfill realized metrics so you can audit how well the label-free estimators tracked reality. Finally, every alert needs a playbook: distinguish a broken upstream pipeline (schema violations, sudden null spikes) from genuine distribution drift, because the fix for the former is a data engineer, not a retraining job.

Monitoring stack flow: live traffic feeds input statistics checks and output confidence checks, delayed labels feed realized metrics, all signals join a decision block that routes to investigate, retrain, or roll back

Figure 2: The layered monitoring stack: input-distribution checks, output/confidence checks, and delayed-label evaluation feed one decision layer that separates pipeline breakage from genuine drift and triggers the right response.

Mathematical Formulation:
$\mathrm{PSI} = \sum_{b=1}^{B} (a_b - e_b)\,\ln\frac{a_b}{e_b}$
$D_{KS} = \max_x \, |F_{ref}(x) - F_{cur}(x)|$

Where:

$a_b$ and $e_b$ are the fractions of the current (actual) and reference (expected) samples in bin $b$ ; rules of thumb: PSI above 0.1 indicates moderate drift, above 0.25 significant drift.
$F_{ref}, F_{cur}$ are the empirical CDFs of the baseline and current window; $D_{KS}$ is their maximum vertical gap (the statistic behind SageMaker’s linf distance checks).
Both are computed per feature per window and compared against thresholds tuned for window size and alert budget.

Layer	Signal	Catches	Blind To
Input Stats	PSI / KS per feature vs baseline	Covariate shift, schema breakage	Concept drift
Output Stats	Prediction mix, confidence histogram	Label shift, confidence collapse	Silent concept drift
Estimated Performance	CBPE from calibrated probabilities	Accuracy loss under covariate shift	Concept drift (by assumption)
Delayed Ground Truth	Realized metrics when labels land	Everything, eventually	Nothing, but late

August 11, 2026

ML0083 Weight Decay

What is weight decay in neural network training, and how does it affect model parameters?

Answer

Weight decay multiplies every weight by a factor slightly below one at each step, so parameters continuously shrink toward zero unless the data gradient pushes them back up. The equilibrium effect: only directions where the loss gradient persistently outweighs the decay survive with large magnitude, while noise-fitting directions get eroded, which is exactly the capacity control we want. For plain SGD, weight decay is mathematically identical to adding an L2 penalty to the loss. For adaptive optimizers like Adam it is not, because the L2 gradient gets rescaled per-parameter along with everything else; AdamW fixes this by decoupling the decay from the adaptive update, and is now the default recipe for training transformers.

(1) Mechanics: each step shrinks weights multiplicatively before (or alongside) the gradient step, so unneeded complexity decays exponentially while useful weights are constantly re-earned from the data gradient.
(2) SGD Equivalence and Its Caveat: for SGD, weight decay with rate lambda equals L2 regularization with coefficient lambda/alpha, so the two are coupled through the learning rate; retuning alpha silently retunes your regularization.
(3) AdamW Fix: Loshchilov and Hutter showed Adam’s adaptive scaling under-regularizes high-gradient weights when L2 is folded into the loss; AdamW applies decay outside the adaptive step, and PyTorch’s AdamW defaults to a weight decay of 0.01.

Elliptical loss contours centered away from the origin, a dashed L2 constraint circle around the origin, and the regularized optimum where the circle touches a contour

Figure 1: The geometric view: the unregularized optimum sits far from the origin; weight decay pulls the solution to where the smallest L2 ball first touches the loss contours, trading a little fit for a lot of magnitude.

The Adam subtlety is worth understanding precisely, because it is a favorite interview trap. In Adam, every component of the gradient, including the penalty’s contribution, is divided by the square root of its running second moment. Parameters with historically large data gradients therefore experience a heavily damped regularization force, which is backwards: the parameters the optimizer is already moving hardest are the ones the penalty reaches least. Note the denominator tracks the second moment of the gradient, not the weight’s own magnitude, so the effect is uneven regularization across parameters rather than a simple “big weights decay less” rule. AdamW removes the penalty from the adaptive gradient entirely and applies it as a direct multiplicative shrink, restoring uniform, controllable decay, which is why nearly every modern pretraining run (GPT-style models included) specifies AdamW.

Weight norm over training steps: no decay grows steadily, Adam plus L2 barely shrinks it, AdamW settles at a lower plateau

Figure 2: The decoupling effect: with no decay the weight norm grows unchecked; Adam with L2 in the loss barely contains it because the penalty is adaptively rescaled; AdamW’s decoupled decay settles at a controlled plateau.

Mathematical Formulation:
$\theta_{t+1} = (1 - \lambda)\,\theta_t - \alpha\, \nabla f_t(\theta_t)$
$f^{reg}_t(\theta) = f_t(\theta) + \frac{\lambda'}{2}\,\|\theta\|^2, \qquad \lambda' = \lambda / \alpha$

Where:

$\theta_t$ collects all parameters at step $t$ , $\alpha$ is the learning rate, and $\lambda$ is the per-step decay rate.
$f_t$ is the minibatch loss; the $(1 - \lambda)$ factor is the multiplicative shrink applied every step.
$\lambda' = \lambda/\alpha$ is the L2 coefficient that makes the two formulations equivalent for SGD, exposing the coupling: change the learning rate and the effective regularization changes too (this equivalence fails for adaptive optimizers, hence AdamW).

Aspect	L2 in the Loss (Adam)	Decoupled Decay (AdamW)
Where Applied	Penalty gradient added to the loss gradient	Direct multiplicative shrink, outside the adaptive step
Adaptive Scaling	Penalty is divided by sqrt(v), so large-gradient weights decay least	Decay is uniform across parameters
Hyperparameter Coupling	Effective regularization entangled with learning rate and gradient scale	Decay rate tuned independently of learning rate
Default Use	Legacy; PyTorch Adam weight_decay argument is actually L2	Standard for transformer pretraining; PyTorch default 0.01

August 11, 2026

ML0081 Gradient Boosting vs Random Forest

How do gradient-boosted decision trees work, and how do they differ from random forests?

Answer

Gradient boosting builds an additive ensemble sequentially: each new tree is fit to the negative gradient of the loss with respect to the current ensemble’s predictions (for squared error, the residuals), then added in with a small learning rate. The ensemble is a directed attack on bias; every stage shrinks whatever error remains. A random forest does the opposite: it grows many deep trees independently on bootstrap samples with random feature subsets and averages their votes, which attacks variance. So boosting uses shallow trees, a learning rate, and sequence, while bagging uses deep trees, randomness, and parallel averaging.

(1) Sequential Residual Fitting: tree m targets the current gradient (pseudo-residuals); with squared loss this is literally y minus the running prediction, and shrinkage keeps any single tree from dominating.
(2) Bias vs Variance: boosting reduces bias stage by stage but can chase noise if run long or deep; random forests reduce variance by decorrelating many overfit trees, but their bias is roughly that of one tree.
(3) Modern Refinement: Yandex showed classic gradient boosting leaks target information into its own gradients (prediction shift), and CatBoost’s ordered boosting fixes it by computing each example’s gradient from a model trained only on earlier examples in a random permutation.

Three panels: initial prediction is flat with large residuals; after adding tree one the curve follows the data and residuals shrink; after tree two the fit is close and residuals are small

Figure 1: Stage-wise fitting: each new tree targets the residuals (orange stems) left by the current ensemble, so the prediction curve (blue) bends toward the data a little at every stage.

Structurally, the two ensembles could not be more different. A random forest is embarrassingly parallel: tree one and tree one hundred do not know each other, so you can train them on separate machines and the average cancels their individual overfitting. Gradient boosting is an inherently sequential chain: tree one hundred is defined by the mistakes of the previous ninety-nine, which buys accuracy on structured tabular data but makes training serial and makes the model sensitive to label noise, because late-stage trees are explicitly trained to fit whatever is left, including the noise.

Side-by-side schematics: random forest trains deep trees in parallel on bootstrap samples and averages votes; gradient boosting trains shallow trees in a chain where each consumes the previous ensemble's residuals

Figure 2: Parallel averaging vs sequential correction: the forest’s trees are independent and deep, the boosted trees are shallow and chained through residuals, with a learning rate throttling each contribution.

Mathematical Formulation:
$F_m(x) = F_{m-1}(x) + \nu\, h_m(x)$
$r_{i,m} = -\left[\frac{\partial\, \ell(y_i, F(x_i))}{\partial\, F(x_i)}\right]_{F = F_{m-1}}$

Where:

$F_m$ is the ensemble after stage $m$ , $h_m$ the tree added at that stage, and $\nu$ the learning rate (shrinkage, typically 0.01-0.1).
$r_{i,m}$ is the pseudo-residual for example $i$ : the negative gradient of the loss $\ell$ with respect to the current prediction, i.e. the direction that most reduces the loss.
For squared loss, $r_{i,m} = y_i - F_{m-1}(x_i)$ , the ordinary residual; other losses (logistic, absolute) change what the trees are fit to.

Aspect	Random Forest	Gradient Boosting
Training	Independent trees, bootstrap samples, fully parallel	Sequential; each tree fits current pseudo-residuals
Tree Shape	Deep, low-bias, high-variance	Shallow (depth 3-8), high-bias learners
What It Reduces	Variance (averaging decorrelated trees)	Bias (stage-wise error correction)
Noise Robustness	Robust; noise averages out	Fragile; late trees chase label noise
Known Issue	Averaging cannot reduce base-tree bias	Target leakage in gradients; fixed by CatBoost ordered boosting

August 11, 2026

ML0078 Self-Supervised Learning

What is self-supervised learning, and how does it differ from unsupervised learning?

Answer

Self-supervised learning manufactures its own labels from the structure of the data: hide part of the input and train the model to predict it. Masked words in a sentence, the next token, a rotated image’s angle, or two augmented views of the same photo all give a supervised training signal without any human annotation. The result is a pretrained representation that transfers to downstream tasks with small labeled sets. Unsupervised learning (clustering, density estimation) discovers structure with no prediction target at all, and its output is the structure itself. Self-supervised learning is supervised machinery run on invented labels; that is why it scales with the same architectures and optimizers as supervised learning.

(1) Pretext Tasks: the label is a withheld piece of the input itself, for example masked-token prediction in BERT, next-token prediction in GPT-style models, or augmentation agreement in SimCLR and SwAV.
(2) The Key Difference: unsupervised learning has no target and answers “how is the data organized”, while self-supervised learning has a target derived from the input and answers “what representation predicts the hidden parts”; both need zero human labels.
(3) Scale Evidence: Meta’s SEER pretrained a billion-parameter network with SwAV on one billion uncurated Instagram images and reached 84.2% ImageNet top-1 after fine-tuning; with only 10% of ImageNet’s labels it still hit 77.9% top-1, and with just 1% it reached 60.5%.

Two pretext tasks: a sentence with a masked token that the model must predict, and an image producing two augmented views whose embeddings are pulled together

Figure 1: Pretext tasks invent labels from the input itself: left, mask a token and predict it from context (BERT style); right, pull the embeddings of two augmented views of the same image together (contrastive style).

The standard production pattern is two-stage. Stage one pretrains on the cheap unlabeled corpus with a pretext objective, which is where most of the compute goes. Stage two fine-tunes the frozen or lightly thawed representation on the small labeled set for the real task, which is where the labels go. The pretext objective is not the product; it is a scaffold whose only job is to force the network to learn transferable structure such as grammar, object parts, and semantic similarity.

Two-stage pipeline: stage one pretrains an encoder on a large unlabeled corpus with a pretext objective, stage two fine-tunes it with a small head on a small labeled set for the downstream task

Figure 2: The two-stage pattern: expensive self-supervised pretraining happens once on unlabeled data; cheap fine-tuning happens per task on labeled data, which is why a 1% labeled slice can still yield strong accuracy.

Mathematical Formulation:
$\mathcal{L}_{mlm} = -\sum_{i \in \mathcal{M}} \log p(x_i \mid x_{\setminus \mathcal{M}})$
$\mathcal{L}_{cl} = -\log \frac{\exp(\mathrm{sim}(u, v) / \tau)}{\sum_{k=1}^{K} \exp(\mathrm{sim}(u, v_k) / \tau)}$

Where:

$\mathcal{M}$ is the set of masked positions, $x_i$ the hidden token, and $x_{\setminus \mathcal{M}}$ the visible context (masked language modeling).
$u, v$ are embeddings of two augmented views of the same input, $v_k$ covers all candidate views, $\mathrm{sim}$ is a similarity such as cosine, and $\tau$ is a temperature (contrastive InfoNCE).

Aspect	Supervised	Unsupervised	Self-Supervised
Target Source	Human labels	No target at all	Withheld part of the input
Typical Output	Task predictions	Clusters, densities, embeddings of structure	Transferable representation for fine-tuning
Loss Form	Supervised loss on labels	Reconstruction, likelihood, linkage	Supervised loss on invented labels
Canonical Examples	ImageNet classifiers	k-means, Gaussian mixtures, PCA	BERT, GPT, SimCLR, SEER (SwAV)

August 11, 2026

ML0077 Semi-Supervised Learning

What is semi-supervised learning, and when is it useful in practice?

Answer

Semi-supervised learning trains on a small labeled set together with a much larger unlabeled set. The unlabeled points reveal the shape of the input distribution, so the learned decision boundary can follow the data’s structure instead of being pinned down by a handful of labels. It works when the cluster assumption roughly holds: points in the same high-density region tend to share a label, so the boundary should pass through low-density regions. It pays off when labels are expensive (medical images, speech transcription, content moderation) but unlabeled data is nearly free. The classic failure mode is confirmation bias: early wrong pseudo-labels get fed back as training targets and the model amplifies its own mistakes.

(1) Core Mechanisms: self-training (pseudo-labeling) trains a teacher on the labeled set, labels the unlabeled pool, and retrains a student on the union; consistency regularization instead penalizes prediction changes across augmented views of the same unlabeled input.
(2) The Assumption Behind It: the gain is real only when unlabeled data comes from the same distribution as the labeled data and classes form separable clusters; off-distribution unlabeled points get confidently wrong pseudo-labels and hurt training.
(3) Production Evidence: Google’s Noisy Student trained an EfficientNet teacher on labeled ImageNet, pseudo-labeled 300 million unlabeled images, and retrained a larger noised student, reaching 88.4% top-1 accuracy; its simplified variant, semi-supervised distillation, was then applied to language understanding inside Google Search.

Two panels of a two-moons dataset: with only six labeled points the boundary cuts through a cluster, while adding unlabeled points pushes the boundary into the low-density gap

Figure 1: Why unlabeled data helps: with six labels the linear boundary slices through a cluster; the unlabeled cloud (gray) reveals the two moons, and a semi-supervised boundary can settle in the low-density gap between them.

Self-training is the workhorse loop: train on labels, predict on the unlabeled pool, keep predictions whose confidence clears a threshold, and retrain on the enlarged set. Noisy Student adds two twists that matter at scale: the student is equal-or-larger than the teacher, and the student is trained with noise (data augmentation, dropout, stochastic depth) while the teacher stays clean when generating labels, so the student must learn a robust function rather than merely copying the teacher.

Self-training loop: train teacher on labeled data, pseudo-label the unlabeled pool, filter by confidence threshold, retrain student on the union, then the student becomes the next teacher

Figure 2: The self-training loop used by Noisy Student: pseudo-labels are filtered by confidence, the noised student retrains on labeled plus pseudo-labeled data, and the loop iterates with the student promoted to teacher.

Mathematical Formulation:
$\mathcal{L} = \mathcal{L}_{sup} + \lambda\,\mathcal{L}_{unsup}$
$\mathcal{L}_{unsup} = \frac{1}{M}\sum_{j=1}^{M} \mathbf{1}\left[\max_c p_j(c) > \tau\right]\, \ell(f(x_j), \hat{y}_j)$

Where:

$\mathcal{L}_{sup}$ is the ordinary supervised loss over the $N$ labeled examples, and $\lambda$ weights the unlabeled term.
$M$ counts the unlabeled examples, $x_j$ is an unlabeled input, and $f$ is the current model.
$p_j(c)$ is the predicted probability of class $c$ , $\hat{y}_j$ is its argmax pseudo-label, and $\tau$ is the confidence threshold (the indicator drops low-confidence examples).

Feature	When It Helps	When It Backfires
Label Economics	Labels need experts or slow review (radiology, legal, moderation)	Labels are cheap; just label more data instead
Unlabeled Pool	Same distribution as the labeled data, same task classes	Pool contains new classes or a shifted distribution
Structure	Classes form clusters with low-density gaps between them	Classes overlap heavily; boundary must cross dense regions
Teacher Quality	The labeled-only model is already decent, so most pseudo-labels are right	A weak teacher seeds errors that the loop then amplifies

August 11, 2026

ML0075 t-SNE vs PCA

What is t-SNE, and when should you use it instead of PCA?

Answer

t-SNE (t-distributed stochastic neighbor embedding) is a non-linear visualization method that turns pairwise distances in high-dimensional space into neighborhood probabilities, then arranges low-dimensional points so their own neighborhood probabilities, computed with a heavy-tailed Student-t kernel, match them by minimizing the KL divergence between the two distributions. Use it instead of PCA when the goal is a 2D map that reveals cluster structure: PCA preserves global variance along linear axes, while t-SNE sacrifices global geometry to keep close neighbors close. Do not use it as a preprocessing step for a downstream model: it is transductive (no transform for new points), stochastic across runs, and its cluster sizes and separations are not quantitative.

(1) What It Optimizes: match high-dimensional affinities $p_{ij}$ (Gaussian, calibrated per point) with low-dimensional affinities $q_{ij}$ (Student-t) via KL divergence; the heavy t-tails let dissimilar points sit far apart without penalty, solving the crowding problem.
(2) Perplexity Is the Knob: it acts as the effective neighbor count per point. Too small fragments clusters, too large merges them; 5 to 50 is the usual range, and results vary with the random seed.
(3) Production Pattern: PCA first, neighbor embedding second. 10x Genomics’ Cell Ranger pipeline runs PCA on gene expression, then computes t-SNE or UMAP projections for its Loupe Browser; UMAP scales better than t-SNE on large cell counts, which is why it has become the default projection at that scale.

Same three-cluster data embedded with PCA on the left showing partially overlapping clusters and with t-SNE on the right showing cleanly separated clusters

Figure 1: The same clustered data two ways: PCA (left) preserves global spread but overlaps clusters that are not linearly separable; t-SNE (right) separates them cleanly. The t-SNE panel’s axes and inter-cluster gaps carry no units and no meaning.

Mathematical Formulation:
$p_{j \mid i} \propto \exp\Big(-\frac{\|x_i - x_j\|^2}{2 \sigma_i^2}\Big)$
$q_{ij} \propto \big(1 + \|z_i - z_j\|^2\big)^{-1}$
$\mathcal{L} = \sum_{i,j} p_{ij} \log \frac{p_{ij}}{q_{ij}}$

Where:

$p_{j \mid i}$ is the probability that $x_i$ picks $x_j$ as a neighbor; the per-point $\sigma_i$ is set so the perplexity matches the chosen value, and the conditionals are symmetrized into $p_{ij}$ .
$q_{ij}$ is the same affinity computed on the map points $z_i$ with a Student-t kernel (1 degree of freedom), whose heavy tails prevent crowding.
$\mathcal{L}$ is the KL divergence between the two affinity matrices, minimized by gradient descent directly on the map coordinates.

Feature	t-SNE	PCA
Type	Non-linear, neighbor-graph based	Linear orthogonal projection
Preserves	Local neighborhoods	Global variance and large-scale structure (projection can only shrink distances)
Output	2-3D coordinates only, no transform	Reusable projection, any $k$
Determinism	Stochastic; reruns differ	Deterministic up to sign flips
Right Use	Final-step cluster visualization	Preprocessing, compression, de-correlation

Three t-SNE embeddings of the same data at perplexity 5 showing fragmented clusters, perplexity 30 showing clean clusters, and perplexity 100 showing merged structure

Figure 2: Perplexity sweeps the effective neighborhood size: at 5 the clusters fragment into fake sub-clusters, at 30 the structure is clean, at 100 distinct clusters merge. Always check a second perplexity before trusting a t-SNE picture.

August 10, 2026

ML0072 Bayesian Optimization

How does Bayesian optimization (e.g., Gaussian processes) work for hyperparameter tuning, and when is it worth it?

Answer

Bayesian optimization tunes an expensive black-box objective by maintaining a probabilistic surrogate model of it, usually a Gaussian process that predicts both the mean performance and the uncertainty at every untried configuration. An acquisition function (expected improvement is the common default) scores each candidate by trading off exploitation (likely to be good) against exploration (highly uncertain), and the configuration maximizing it is evaluated next; the result updates the surrogate and the loop repeats. It needs far fewer trials than grid or random search when each evaluation costs minutes to hours, but the Gaussian process costs $O(n^3)$ to fit in the number of trials and struggles in high-dimensional or heavily categorical spaces.

(1) Surrogate: the GP posterior gives a predictive mean $\mu(x)$ and uncertainty $\sigma(x)$ from a handful of noisy trials, and it is cheap to query, unlike the real objective.
(2) Acquisition: expected improvement picks the point with the largest expected gain over the incumbent, automatically balancing exploration and exploitation without hand-tuned schedules.
(3) When It Is Worth It: when trials dwarf surrogate cost (minutes or more per trial, tens of dimensions at most). Meta’s Ax platform (built on BoTorch) runs exactly this loop to tune recommender systems and AR/VR hardware designs, while Google’s Vizier has tuned over 70 million objectives and swaps in more scalable algorithms once the trial count outgrows GP fitting costs.

Top panel: Gaussian process posterior mean with a shaded uncertainty band fitted to a few observed points; bottom panel: expected improvement curve peaking at the next point to evaluate

Figure 1: One iteration of the loop: the GP posterior (mean with uncertainty band, top) is fitted to the evaluated points, and the expected-improvement acquisition (bottom) peaks where a high predicted mean and high uncertainty combine. That peak becomes the next expensive evaluation.

Mathematical Formulation:
$\mathrm{EI}(x) = \mathbb{E}\big[\max(f(x) - f^{*}, 0)\big]$
$x_{t+1} = \arg\max_{x}\ \mathrm{EI}(x)$

Where:

$f(x)$ is the expensive black-box objective (for example validation accuracy) and $f^{*}$ the best value observed so far.
The expectation is taken under the GP posterior, so $\mathrm{EI}(x)$ grows both with predicted quality $\mu(x)$ and with uncertainty $\sigma(x)$ .
$x_{t+1}$ is the next configuration to evaluate; the acquisition maximization is cheap because it queries only the surrogate.

Feature	Bayesian Optimization	Grid / Random Search
How Points Are Chosen	Surrogate model plus acquisition function	Fixed lattice / uniform sampling
Trials to a Good Region	Fewest, for smooth low-dimensional objectives	Grows exponentially (grid) or slowly (random)
Per-Step Overhead	GP fit $O(n^3)$ plus acquisition search	None
Parallelism	Needs batch acquisition (qEI)	Trivially parallel
Worth It When	Trials cost minutes or more, up to tens of dimensions	Cheap trials, high dimensions, quick baseline

Best value found so far versus number of evaluations: Bayesian optimization curve rises fastest, random search in the middle, grid search slowest

Figure 2: Best value found so far against evaluations spent. Bayesian optimization’s surrogate-guided choices reach a good region in far fewer trials than random or grid search, which is exactly why it pays off only when each trial is expensive.

August 10, 2026

ML0070 A/B Test Design

How would you design an A/B test, including sample size and duration?

Answer

An A/B test is a randomized controlled trial: randomly assign users to control (A) and treatment (B), measure a pre-registered primary metric, and compare the two arms with a hypothesis test. The design happens before any traffic flows: pick one primary metric and a minimum detectable effect (MDE), fix $\alpha$ and power (0.05 and 0.8 are common defaults), and compute the required sample size per arm from the metric’s variance. Duration follows from daily traffic, rounded up to whole business cycles (at least one to two weeks) so day-of-week effects average out, and the decision is made once at the pre-committed end rather than by peeking at daily p-values.

(1) Hypothesis and MDE First: state the expected effect and the smallest effect worth shipping for. The MDE, $\alpha$ , power, and metric variance together pin down the sample size.
(2) Duration by Cycles, Not by Peeking: run whole weeks to cover weekly seasonality and never stop early on a crossing p-value. Booking.com simulated null experiments and found that daily peeking lets over half of them touch 95% significance at some point, versus the promised 5%.
(3) Sensitivity Is the Bottleneck: variance reduction beats longer runs. CUPED (developed at Microsoft) uses pre-experiment covariates to cut metric variance by about 50%, halving the required users or duration, and Booking.com warns that tracking users the change cannot reach dilutes the measured effect and inflates the required sample size by the dilution factor squared.

Flow diagram: hypothesis with MDE leads to random assignment into arms A and B, then a run of full weekly cycles, then one test at the pre-committed end, then a ship or iterate decision

Figure 1: The design pipeline: fix the hypothesis, MDE, $\alpha$ , and power up front; randomize; run whole business cycles; test exactly once at the pre-committed sample size; then ship, iterate, or abandon.

Mathematical Formulation:
$n = \frac{2 \, (z_{1-\alpha/2} + z_{1-\beta})^2 \, \sigma^2}{\delta^2}$
$\sigma^2 = p \, (1 - p) \ \text{for a binary metric}$

Where:

$n$ is the required sample size per arm; $\delta$ is the MDE; $\sigma^2$ is the per-user metric variance.
$z_{1-\alpha/2}$ and $z_{1-\beta}$ are normal quantiles for the significance level and power (1.96 and 0.84 at the usual settings).
$p$ is the baseline rate for a binary metric such as conversion, whose variance is set by $p(1 - p)$ .

Sample size per arm on a log scale falling with the square of the minimum detectable effect, with a marked point at 1 percentage point MDE on a 10 percent baseline needing about 14400 users per arm

Figure 2: Sample size per arm falls with the square of the MDE: halving the detectable effect quadruples the traffic needed. On a 10% conversion baseline, a 1-point MDE needs about 14,400 users per arm; a 0.25-point MDE needs about 230,000.

Decision Rule	When You Look	False Positive Control	Cost
Fixed horizon (pre-committed n)	Once, at the end	Exact $\alpha$	Requires patience
Naive peeking	Daily, stop on significance	Inflated: Booking’s nulls hit 95% over half the time	Ships noise
Sequential test (mSPRT, group-sequential)	Any time	Controlled by construction	Wider confidence sequences

Two histograms of the estimated treatment minus control difference: a wide raw distribution and a narrow CUPED-adjusted distribution centered on the same effect

Figure 3: CUPED regresses out the part of the metric explained by pre-experiment behavior, leaving the same estimated effect with roughly half the variance. That is equivalent to doubling the sample size without adding a single user.

August 10, 2026

ML0064 Random Forest II
Please explain the benefits and drawbacks of random forest.
Answer
Random forest is a powerful ensemble method that reduces overfitting and improves predictive accuracy by combining many decision trees. The trade-off: it sacrifices interpretability and computational efficiency, and it may require careful tuning on large, imbalanced, or sparse datasets.
(1) Benefit: Reduces Overfitting: Aggregating many trees lowers variance.
(2) Benefit: Robust: Less sensitive to noise and outliers; handles high-dimensional data well.
(3) Benefit: Feature Importance: Built-in estimates identify influential variables; bagging improves generalization.
(4) Drawback: Cost: Hard to interpret compared to a single tree; slower to train and predict; large forests consume significant memory.
(5) Drawback: Data Traps: Class imbalance can bias predictions, and very sparse data can make it underperform other algorithms.

Figure 1: The imbalance trap: on 96/4 skewed data the random forest’s regions bend toward the majority class and its minority-class F1 (0.73) loses even to a class-weighted logistic regression (0.76): a strong default is not automatically strong everywhere.
Mathematical Formulation:
$\mathrm{Var}\big(\bar{T}(x)\big) = \rho \, \sigma^2 + \frac{1 - \rho}{B} \, \sigma^2$
Where:
$\bar{T}(x)$ is the forest’s averaged prediction over $B$ trees, each with individual variance $\sigma^2$ .
$\rho$ is the pairwise correlation between trees: this is what bagging and random feature subsets try to shrink.
As $B \to \infty$ the second term vanishes but the $\rho\sigma^2$ floor remains: correlation, not tree count, is the limiting factor.
Login to view more content
July 8, 2025
ML0060 K Selection in K-Means
How to select K in K-Means?
Answer
To select the optimal number of clusters $K$ in K-means, combine visual tools (the elbow method), quantitative metrics (the silhouette score), and statistical methods (the gap statistic), balancing model fit against generalization without overfitting. Domain knowledge and interpretability should have the final word.
(1) Elbow Method: Plot within-cluster sum of squares (WCSS) against $K$ and pick the “elbow” where the rate of improvement sharply slows.
(2) Silhouette Score: Compute the average silhouette coefficient for each candidate $K$ and pick the highest: it rewards tight, well-separated clusters.
(3) Gap Statistic: Compare the observed WCSS against that of a random reference distribution; pick the $K$ maximizing the gap.

Figure 1: The elbow method: WCSS always decreases as K grows, but the marginal gain collapses after the true cluster count (the marked elbow at K = 3). Beyond the elbow you pay model complexity for noise.
Mathematical Formulation:
$\text{WCSS}(K) = \sum_{k=1}^{K} \sum_{x_i \in C_k} \|x_i - \mu_k\|^2$
$s(i) = \frac{b(i) - a(i)}{\max\big(a(i),\, b(i)\big)}$
Where:
$C_k$ is cluster $k$ and $\mu_k$ its centroid; WCSS is the total within-cluster squared distance that the elbow method plots against $K$ .
$a(i)$ is point $i$ ‘s mean distance to its own cluster (intra-cluster); $b(i)$ its mean distance to the nearest other cluster.
$s(i) \in [-1, 1]$ : near 1 means well clustered, near 0 means on a boundary, negative means likely misassigned; average it over all points for each $K$ .
Login to view more content
June 14, 2025