Tag: Basics

ML0090 Naive Bayes Classifier

Explain the Naive Bayes classifier and the “naive” conditional independence assumption behind it, as used in production spam detection and support ticket routing.

Answer

Naive Bayes is a probabilistic classifier that applies Bayes’ theorem with a “naive” assumption: given the class label, all features are conditionally independent. This lets the classifier multiply per-feature likelihoods instead of estimating the full joint distribution, which would require exponential data. Despite the assumption being almost always violated in practice, Naive Bayes trains in O(nd) time, predicts in O(d) time, and performs surprisingly well on text classification, spam filtering, and support ticket routing. A production case study processes 85,000 product reviews per day with Multinomial Naive Bayes, achieving AUC above 0.99 for spam detection while retraining in under 5 minutes on a single-core VM, and a TF-IDF plus Multinomial NB pipeline routes banking support tickets in under 1 millisecond per ticket on the Banking77 dataset.

(1) Bayes’ Theorem Applied: the classifier computes the posterior $P(y \mid x)$ for each class and picks the argmax; the naive assumption factors the likelihood $P(x_1, \ldots, x_d \mid y)$ into a product of one-dimensional terms $\prod P(x_j \mid y)$ .
(2) Why the Assumption Works Anyway: even when the independence assumption is wrong, the argmax decision is often correct because correlated features push the posterior in the same direction; the calibration is off but the ranking is not, which is why scikit-learn recommends CalibratedClassifierCV with isotonic regression for reliable probabilities.
(3) Production Strengths: trains in milliseconds, predicts in microseconds, needs no GPU, produces interpretable per-feature contributions, and a 2024 Generalized Naive Bayes paper showed that relaxing the independence assumption via optimal structure learning improves accuracy while keeping the computational profile.

Two panels: left shows the full joint distribution P(x1,x2,x3|y) as a large table requiring exponential data; right shows the naive factorization as three small per-feature tables P(x1|y), P(x2|y), P(x3|y) multiplied together

Figure 1: The naive independence assumption factorizes the intractable joint likelihood P(x1, x2, x3 | y) into a product of one-dimensional per-feature likelihoods, replacing an exponential-size table with d small tables.

The independence assumption is the single most important thing to understand about Naive Bayes, because it is both its source of speed and its source of error. When two features are highly correlated (e.g., “free” and “gift” both appearing in spam), the classifier double-counts their evidence, pushing the posterior toward 0 or 1 more aggressively than the true joint distribution would. This is why GaussianNB in scikit-learn tends to produce overconfident probabilities and why calibration is recommended. The assumption also means Naive Bayes cannot capture feature interactions; a 2024 paper on Generalized Naive Bayes (GNB) addresses this by learning an optimal dependency structure among features while preserving the efficient factorization, proving the approximation is at least as good as classical Naive Bayes. For text specifically, the classic remedies are TF-IDF weighting, document-length normalization, and complement Naive Bayes, introduced by Rennie et al. to correct the multinomial model’s mismatch with real text. In practice, the speed-accuracy trade-off favors Naive Bayes for high-volume, low-latency text tasks where a transformer would be overkill.

Mathematical Formulation:
$\hat{y} = \arg\max_{y}\; P(y) \prod_{j=1}^{d} P(x_j \mid y)$
$P(x_j \mid y) = \frac{N_{yj} + \alpha}{N_y + \alpha\, d}$

Where:

$\hat{y}$ is the predicted class; $P(y)$ is the class prior, and the product of per-feature likelihoods replaces the intractable joint $P(x_1, \ldots, x_d \mid y)$ under the naive conditional independence assumption.
The second equation is the smoothed estimator for discrete (count) features: $N_{yj}$ is the count of feature $j$ in class $y$ , $N_y$ is the total count for class $y$ , $d$ is the vocabulary size, and $\alpha$ is the Laplace smoothing parameter (typically 1) that prevents zero probabilities for unseen feature-class combinations. Continuous features use a density instead, such as a per-class Gaussian.
The argmax is computed in log space to avoid underflow: $\log P(y) + \sum_j \log P(x_j \mid y)$ , which is a sum of precomputed log-likelihoods and runs in O(d) per prediction.

Property	Naive Bayes	Logistic Regression
Training Cost	O(nd) single pass	O(nd) per iteration, multiple iterations
Key Assumption	Conditional independence of features given class	No independence assumption; linear decision boundary
Calibration	Overconfident; needs isotonic calibration	Well-calibrated by default
Interpretability	Per-feature log-likelihood contributions	Per-feature weights (coefficients)
Best For	High-volume text, spam, ticket routing	General tabular classification with calibrated probabilities

August 12, 2026

ML0085 Covariate Label Concept Drift

What is the difference between covariate shift, label shift, and concept drift?

Answer

All three are ways the joint distribution P(X, Y) at serving time can differ from training time, and they differ in which factor moved. Covariate shift: the input distribution P(X) changes while the labeling rule P(Y|X) stays fixed (your users got older, but age still means the same thing for risk). Label shift: the class prior P(Y) changes while the class-conditional input P(X|Y) stays fixed (fraud rate doubles in a crisis, but fraud still looks the same). Concept drift: the labeling rule P(Y|X) itself changes (the definition of spam evolves), so the model’s boundary is simply wrong now. Detection and repair differ per type, which is why the taxonomy matters operationally.

(1) Covariate Shift: P(X) moves, P(Y|X) fixed; visible in input statistics alone, and importance reweighting (weighting training points by density ratio) is the classical correction.
(2) Label Shift: P(Y) moves, P(X|Y) fixed; invisible in per-feature input stats if classes look alike, but visible in the model’s output distribution, and corrected by reweighting with estimated new priors.
(3) Concept Drift: P(Y|X) moves; no input-only monitor can see it in general, so you need ground truth (delayed labels) or estimation methods, and the only real fix is retraining on the new concept.

Three panels: covariate shift moves the input cloud while the boundary stays; label shift changes class proportions; concept drift rotates the boundary itself

Figure 1: The three drift types: covariate shift relocates the inputs (boundary still valid), label shift reweights class frequencies, and concept drift moves the true boundary, which invalidates the model even if inputs look unchanged.

The operational asymmetry is the interview-worthy insight. Input-side monitors (feature histograms, PSI, KS tests) reliably catch covariate shift and obvious label shift, because those live in P(X) and in the output distribution. But a pure concept drift can leave every input statistic untouched while accuracy collapses, which is why production monitoring stacks pair input-drift detectors with performance estimation or delayed ground-truth evaluation. NannyML’s CBPE documentation states exactly this split: confidence-based performance estimation stays accurate under covariate shift but cannot detect concept drift without labels.

Mathematical Formulation:
$P(X, Y) = P(Y \mid X)\,P(X) = P(X \mid Y)\,P(Y)$

Where:

$X$ is the input and $Y$ the target; the joint can be factorized in two directions.
Covariate shift: $P_{tr}(X) \neq P_{te}(X)$ while $P(Y \mid X)$ is unchanged.
Label shift: $P_{tr}(Y) \neq P_{te}(Y)$ while $P(X \mid Y)$ is unchanged; concept drift: $P(Y \mid X)$ itself changes (subscripts tr/te denote training vs serving).

Type	What Changes	Example	Detection Signal
Covariate Shift	P(X); boundary P(Y\|X) intact	User base ages; medical device sees older patients	Input stats (PSI/KS per feature)
Label Shift	P(Y); class appearance P(X\|Y) intact	Fraud rate spikes during a holiday season	Output/prediction distribution shift
Concept Drift	P(Y\|X); the rule itself moves	Spammers change tactics; same features, new meaning	Delayed labels; invisible to input-only monitors

August 11, 2026

ML0084 When Ensembles Fail

When can an ensemble of models perform worse than its best individual member?

Answer

Ensembles win through diversity, not through headcount. The ambiguity decomposition makes this exact: the ensemble’s error equals the average member error minus the average member disagreement, so combining helps only to the extent that members err on different examples. When errors are highly correlated, averaging or voting adds nothing and can dilute a genuinely better member. Stacked ensembles add a second failure surface: a meta-learner trained on too little or leaked validation data overfits the members’ quirks. And in production, the extra accuracy may not survive contact with engineering constraints, which is why Netflix never deployed the million-dollar Grand Prize ensemble.

(1) Diversity Is the Mechanism: majority voting only helps when member mistakes are at least partially independent; perfectly correlated members vote identically, and the ensemble is just the average member.
(2) Combination Failures: a strong model blended with weak-but-confident ones, or a stacking meta-learner fit on a tiny validation slice, can land below the best member; out-of-fold predictions and held-out weight tuning are the standard guards.
(3) Production Lesson (Netflix): the 2009 Grand Prize ensemble blended hundreds of models for a 10% RMSE gain, but Netflix reported the incremental accuracy “did not seem to justify the engineering effort” and shipped only two of the simpler Progress-Prize algorithms instead.

Two bar panels: with independent errors the majority vote beats every individual classifier, with correlated errors the vote falls below the best individual

Figure 1: The diversity condition: with independent errors (left) majority vote exceeds every member; with correlated errors (right) the vote inherits the shared blind spot and can fall below the best single model.

Two subtler traps complete the picture. First, aggregation weights: an average is optimal only when members are comparably accurate and comparably calibrated; one badly calibrated but overconfident member can dominate a soft-vote average and drag it below the best member. Second, the objective mismatch: members tuned individually for accuracy may combine poorly if their errors concentrate on the same hard slice; deliberately trading a little individual accuracy for decorrelation (different feature views, architectures, or training objectives) is how production ensembles are actually designed.

Mathematical Formulation:
$E_{ens} = \bar{E} - \bar{A}$
$\bar{A} = \frac{1}{M}\sum_{m=1}^{M} \mathrm{E}_x\left[ (f_m(x) - \bar{f}(x))^2 \right]$

Where:

$E_{ens}$ is the squared error of the average prediction $\bar{f}$ , and $\bar{E}$ is the mean of the members’ individual squared errors.
$\bar{A}$ is the ambiguity: the average squared disagreement of each member $f_m$ from the ensemble mean, i.e. the diversity dividend.
The ensemble beats the average member exactly by $\bar{A}$ ; if all members make identical errors then $\bar{A} = 0$ and ensembling gains nothing (Krogh-Vedelsby decomposition).

Failure Mode	Mechanism	Guard
Correlated Errors	Same architecture, features, and data produce shared blind spots	Diversify feature views, model families, training objectives
Overfit Stacking	Meta-learner trained on in-sample or tiny validation predictions	Out-of-fold predictions; simple meta-model (logistic, weighted mean)
Miscalibrated Member	Overconfident weak model dominates the soft vote	Calibrate members before averaging; weight by validation skill
Engineering Cost	Hundreds of members multiply latency, memory, and failure surface	Netflix lesson: ship the simple models that capture most of the gain

August 11, 2026

ML0078 Self-Supervised Learning

What is self-supervised learning, and how does it differ from unsupervised learning?

Answer

Self-supervised learning manufactures its own labels from the structure of the data: hide part of the input and train the model to predict it. Masked words in a sentence, the next token, a rotated image’s angle, or two augmented views of the same photo all give a supervised training signal without any human annotation. The result is a pretrained representation that transfers to downstream tasks with small labeled sets. Unsupervised learning (clustering, density estimation) discovers structure with no prediction target at all, and its output is the structure itself. Self-supervised learning is supervised machinery run on invented labels; that is why it scales with the same architectures and optimizers as supervised learning.

(1) Pretext Tasks: the label is a withheld piece of the input itself, for example masked-token prediction in BERT, next-token prediction in GPT-style models, or augmentation agreement in SimCLR and SwAV.
(2) The Key Difference: unsupervised learning has no target and answers “how is the data organized”, while self-supervised learning has a target derived from the input and answers “what representation predicts the hidden parts”; both need zero human labels.
(3) Scale Evidence: Meta’s SEER pretrained a billion-parameter network with SwAV on one billion uncurated Instagram images and reached 84.2% ImageNet top-1 after fine-tuning; with only 10% of ImageNet’s labels it still hit 77.9% top-1, and with just 1% it reached 60.5%.

Two pretext tasks: a sentence with a masked token that the model must predict, and an image producing two augmented views whose embeddings are pulled together

Figure 1: Pretext tasks invent labels from the input itself: left, mask a token and predict it from context (BERT style); right, pull the embeddings of two augmented views of the same image together (contrastive style).

The standard production pattern is two-stage. Stage one pretrains on the cheap unlabeled corpus with a pretext objective, which is where most of the compute goes. Stage two fine-tunes the frozen or lightly thawed representation on the small labeled set for the real task, which is where the labels go. The pretext objective is not the product; it is a scaffold whose only job is to force the network to learn transferable structure such as grammar, object parts, and semantic similarity.

Two-stage pipeline: stage one pretrains an encoder on a large unlabeled corpus with a pretext objective, stage two fine-tunes it with a small head on a small labeled set for the downstream task

Figure 2: The two-stage pattern: expensive self-supervised pretraining happens once on unlabeled data; cheap fine-tuning happens per task on labeled data, which is why a 1% labeled slice can still yield strong accuracy.

Mathematical Formulation:
$\mathcal{L}_{mlm} = -\sum_{i \in \mathcal{M}} \log p(x_i \mid x_{\setminus \mathcal{M}})$
$\mathcal{L}_{cl} = -\log \frac{\exp(\mathrm{sim}(u, v) / \tau)}{\sum_{k=1}^{K} \exp(\mathrm{sim}(u, v_k) / \tau)}$

Where:

$\mathcal{M}$ is the set of masked positions, $x_i$ the hidden token, and $x_{\setminus \mathcal{M}}$ the visible context (masked language modeling).
$u, v$ are embeddings of two augmented views of the same input, $v_k$ covers all candidate views, $\mathrm{sim}$ is a similarity such as cosine, and $\tau$ is a temperature (contrastive InfoNCE).

Aspect	Supervised	Unsupervised	Self-Supervised
Target Source	Human labels	No target at all	Withheld part of the input
Typical Output	Task predictions	Clusters, densities, embeddings of structure	Transferable representation for fine-tuning
Loss Form	Supervised loss on labels	Reconstruction, likelihood, linkage	Supervised loss on invented labels
Canonical Examples	ImageNet classifiers	k-means, Gaussian mixtures, PCA	BERT, GPT, SimCLR, SEER (SwAV)

August 11, 2026

ML0077 Semi-Supervised Learning

What is semi-supervised learning, and when is it useful in practice?

Answer

Semi-supervised learning trains on a small labeled set together with a much larger unlabeled set. The unlabeled points reveal the shape of the input distribution, so the learned decision boundary can follow the data’s structure instead of being pinned down by a handful of labels. It works when the cluster assumption roughly holds: points in the same high-density region tend to share a label, so the boundary should pass through low-density regions. It pays off when labels are expensive (medical images, speech transcription, content moderation) but unlabeled data is nearly free. The classic failure mode is confirmation bias: early wrong pseudo-labels get fed back as training targets and the model amplifies its own mistakes.

(1) Core Mechanisms: self-training (pseudo-labeling) trains a teacher on the labeled set, labels the unlabeled pool, and retrains a student on the union; consistency regularization instead penalizes prediction changes across augmented views of the same unlabeled input.
(2) The Assumption Behind It: the gain is real only when unlabeled data comes from the same distribution as the labeled data and classes form separable clusters; off-distribution unlabeled points get confidently wrong pseudo-labels and hurt training.
(3) Production Evidence: Google’s Noisy Student trained an EfficientNet teacher on labeled ImageNet, pseudo-labeled 300 million unlabeled images, and retrained a larger noised student, reaching 88.4% top-1 accuracy; its simplified variant, semi-supervised distillation, was then applied to language understanding inside Google Search.

Two panels of a two-moons dataset: with only six labeled points the boundary cuts through a cluster, while adding unlabeled points pushes the boundary into the low-density gap

Figure 1: Why unlabeled data helps: with six labels the linear boundary slices through a cluster; the unlabeled cloud (gray) reveals the two moons, and a semi-supervised boundary can settle in the low-density gap between them.

Self-training is the workhorse loop: train on labels, predict on the unlabeled pool, keep predictions whose confidence clears a threshold, and retrain on the enlarged set. Noisy Student adds two twists that matter at scale: the student is equal-or-larger than the teacher, and the student is trained with noise (data augmentation, dropout, stochastic depth) while the teacher stays clean when generating labels, so the student must learn a robust function rather than merely copying the teacher.

Self-training loop: train teacher on labeled data, pseudo-label the unlabeled pool, filter by confidence threshold, retrain student on the union, then the student becomes the next teacher

Figure 2: The self-training loop used by Noisy Student: pseudo-labels are filtered by confidence, the noised student retrains on labeled plus pseudo-labeled data, and the loop iterates with the student promoted to teacher.

Mathematical Formulation:
$\mathcal{L} = \mathcal{L}_{sup} + \lambda\,\mathcal{L}_{unsup}$
$\mathcal{L}_{unsup} = \frac{1}{M}\sum_{j=1}^{M} \mathbf{1}\left[\max_c p_j(c) > \tau\right]\, \ell(f(x_j), \hat{y}_j)$

Where:

$\mathcal{L}_{sup}$ is the ordinary supervised loss over the $N$ labeled examples, and $\lambda$ weights the unlabeled term.
$M$ counts the unlabeled examples, $x_j$ is an unlabeled input, and $f$ is the current model.
$p_j(c)$ is the predicted probability of class $c$ , $\hat{y}_j$ is its argmax pseudo-label, and $\tau$ is the confidence threshold (the indicator drops low-confidence examples).

Feature	When It Helps	When It Backfires
Label Economics	Labels need experts or slow review (radiology, legal, moderation)	Labels are cheap; just label more data instead
Unlabeled Pool	Same distribution as the labeled data, same task classes	Pool contains new classes or a shifted distribution
Structure	Classes form clusters with low-density gaps between them	Classes overlap heavily; boundary must cross dense regions
Teacher Quality	The labeled-only model is already decent, so most pseudo-labels are right	A weak teacher seeds errors that the loop then amplifies

August 11, 2026

ML0076 Reinforcement Learning Loop

What is reinforcement learning? Describe the agent-environment interaction loop.

Answer

Reinforcement learning is learning what to do from trial and error: an agent acts in an environment, receives a scalar reward, and adjusts its behavior to maximize expected cumulative reward. The loop: at step $t$ the agent observes state $s_t$ , picks an action $a_t$ from its policy $\pi(a \mid s)$ , the environment transitions to a new state and returns reward $r_{t+1}$ , and the agent updates its policy or value estimates from the experience; this repeats until the episode ends. Unlike supervised learning there are no labels: the only supervision is the reward signal, which may arrive many steps late, and the agent’s own actions decide what data it sees next, which is why exploration must be built in.

(1) The MDP Tuple: states, actions, transition probabilities, a reward function, and a discount factor $\gamma$ ; the Markov property says the state summarizes everything relevant about the history.
(2) Return and Value: the agent maximizes the expected discounted return, and value functions score states or state-action pairs so credit for delayed rewards can be assigned to earlier decisions.
(3) The Same Loop Now Trains LLMs: RLHF (InstructGPT) fits a reward model from human preference rankings and optimizes the language model against it with PPO; DeepSeek-R1’s GRPO removes the value critic and baselines each prompt against its own sampled answer group, lifting AIME pass@1 from 15.6% to 71.0% with pure RL.

Agent box and environment box with an action arrow from agent to environment and a state plus reward arrow back, annotated with the policy and the experience tuple

Figure 1: The interaction loop: the agent’s policy turns the observed state into an action, the environment answers with the next state and a scalar reward, and the experience tuple $(s_t, a_t, r_{t+1}, s_{t+1})$ feeds the learning update.

Mathematical Formulation:
$G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k+1}$
$\pi^{*} = \arg\max_{\pi}\ \mathbb{E}_{\pi}\big[G_0\big]$

Where:

$G_t$ is the discounted return from step $t$ ; $r_{t+k+1}$ is the reward received $k$ steps later.
$\gamma \in [0, 1)$ is the discount factor, trading immediate reward against future reward and keeping infinite-horizon returns finite.
$\pi$ is the policy and the expectation runs over trajectories induced by $\pi$ and the environment’s transition dynamics; $\pi^{*}$ is the optimal policy.

Feature	Reinforcement Learning	Supervised Learning
Supervision Signal	Scalar reward, possibly delayed many steps	A label per example
Data	Self-generated, non-stationary	Fixed, assumed i.i.d.
Objective	Maximize expected discounted return	Minimize a per-example loss
Feedback Timing	Credit assignment over time	Immediate and exact
Canonical Loop	act, observe reward and next state, update	epochs over a fixed dataset

Three boxes: the language model policy samples answers, a reward model scores them, and a PPO or GRPO update improves the policy, with a loop arrow back to the policy

Figure 2: The loop specialized to LLM post-training: the policy is the language model, actions are token sequences, and a learned reward model supplies the scalar reward. PPO (InstructGPT) trains a value critic alongside; GRPO (DeepSeek-R1) replaces it with a group baseline over sampled answers.

August 10, 2026

ML0073 Principal Component Analysis

How does Principal Component Analysis (PCA) work, step by step?

Answer

PCA finds the orthogonal directions of maximum variance in centered data and projects the data onto the top $k$ of them. Step by step: center each feature by subtracting its mean (and scale to unit variance when units differ); compute the covariance matrix; eigendecompose it, or equivalently run SVD on the data matrix, so eigenvectors become the principal directions and eigenvalues the variance each direction carries; keep the top $k$ eigenvectors and project every sample onto them. The result is a rotated coordinate system whose axes are uncorrelated and ordered by explanatory power, which is why the first few components often carry most of the variance.

(1) Variance as Information: each principal component is the direction of maximum residual variance, orthogonal to all previous ones, and its eigenvalue $\lambda_j$ is exactly the variance the data has along it.
(2) Explained-Variance Ratio Picks k: plot the cumulative share $\sum_{j \leq k} \lambda_j / \sum_j \lambda_j$ and keep enough components to cover, say, 95% of total variance.
(3) Linear and Scale-Sensitive: PCA rotates but never unrolls, so curved manifolds need non-linear methods, and features with different units must be standardized first. Learned embeddings sidestep truncation differently: OpenAI’s text-embedding-3 models were trained with Matryoshka representation learning, so shortening a 3072-dimension vector to 256 still beats the older full-size ada-002, a property naive PCA truncation of a learned embedding cannot promise.

Two-dimensional elliptical point cloud with the first principal component arrow along the major axis and the second along the minor axis

Figure 1: The two principal components of a 2D cloud: PC1 lies along the direction of maximum spread and carries the largest eigenvalue; PC2 is orthogonal to it and mops up the residual variance.

Mathematical Formulation:
$\Sigma = \frac{1}{N} X^{\top} X \ \ (\text{centered } X)$
$\Sigma v_j = \lambda_j v_j$
$z_i = V_k^{\top} x_i$

Where:

$X$ is the centered data matrix with $N$ samples and $d$ features; $\Sigma$ is its $d \times d$ covariance matrix.
$v_j, \lambda_j$ are the eigenvector-eigenvalue pairs of $\Sigma$ , sorted by descending $\lambda_j$ .
$V_k$ stacks the top $k$ eigenvectors and $z_i$ is the $k$ -dimensional score vector of sample $x_i$ .

Scree plot: bars of per-component variance falling off, with a cumulative explained variance line crossing the 95 percent mark

Figure 2: The scree plot: bars show the variance each component carries and the curve shows the cumulative share. Keeping components up to the 95% crossing is the standard rule for choosing $k$ .

Projection is lossy: the discarded components carry the remaining variance, so projecting back into the original space reconstructs the data only up to the kept subspace. In 2D this means every point collapses onto the PC1 line when only one component is kept, which is the geometric picture behind “95% of variance retained”.

Point cloud with each point connected by a thin line to its perpendicular reconstruction on the principal component axis

Figure 3: Reconstruction from one component: every point is replaced by its perpendicular projection onto the PC1 axis. The gray connectors are the reconstruction errors, whose mean squared length is exactly the discarded variance $\lambda_2$ (their sum is $N \lambda_2$ ).

Feature	Eigendecomposition of Covariance	SVD of the Data Matrix
Input	The $d \times d$ matrix $\Sigma$	The centered $N \times d$ matrix $X$ directly
Numerical Stability	Forming $X^{\top} X$ squares the condition number	Stable: never materializes $X^{\top} X$
Scalability	Fine for small $d$	Randomized SVD scales to huge $N$ and $d$
Practice	Textbook derivation	What libraries (scikit-learn) actually call

August 10, 2026

ML0071 Covariance vs Correlation

What is the difference between covariance and correlation?

Answer

Both measure how two variables move together. Covariance is the raw co-movement, expressed in the product of the two variables’ units: positive when the variables tend to sit above their means together, negative when one runs high while the other runs low. Correlation (Pearson’s $\rho$ ) is covariance divided by both standard deviations, which strips the units and bounds the result to $[-1, 1]$ , so strengths become comparable across datasets. Neither proves causation, and both capture only linear association: a perfect parabola scores zero correlation despite being fully determined.

(1) Units: covariance keeps the product of the units (dollars times kilograms); correlation is unitless and bounded, so rescaling a variable changes the covariance but leaves the correlation untouched.
(2) Sign vs Strength: covariance tells you the direction of co-movement; only correlation tells you a comparable strength, since $0.9$ vs $0.2$ means the same thing for any pair of variables.
(3) Linear Only: both are second-moment statistics of linear association. Zero correlation does not imply independence: $Y = X^2$ on symmetric data has $\rho = 0$ while $X$ fully determines $Y$ .

Four scatter panels: positive correlation, negative correlation, zero correlation with independent points, and a parabola with zero correlation but full dependence

Figure 1: Correlation reads linear co-movement: positive (top left), negative (top right), zero with independence (bottom left), and the trap case, zero correlation with complete dependence on a parabola (bottom right). The first three have matching covariance signs; the fourth shows why a second-moment statistic can miss structure.

Mathematical Formulation:
$\mathrm{Cov}(X, Y) = \mathbb{E}\big[(X - \mu_X)(Y - \mu_Y)\big]$
$\rho_{XY} = \frac{\mathrm{Cov}(X, Y)}{\sigma_X \, \sigma_Y}$

Where:

$X, Y$ are random variables with means $\mu_X, \mu_Y$ and standard deviations $\sigma_X, \sigma_Y$ .
$\mathbb{E}[\cdot]$ is expectation over the joint distribution; in practice both quantities are estimated by sample averages.
$\rho_{XY}$ lies in $[-1, 1]$ by the Cauchy-Schwarz inequality, with $\pm 1$ meaning an exact linear relationship.

Feature	Covariance	Correlation
Definition	Mean product of centered values	Covariance divided by both $\sigma$
Units	Product of the variables’ units	Unitless
Range	Unbounded	$[-1, 1]$
Scale Invariance	No: rescaling a variable rescales it	Yes: unchanged by a positive affine rescaling (a negative scale flips the sign)
Typical Use	Portfolio variance, covariance matrices, PCA	Comparing association strength, feature screening

August 10, 2026

ML0067 Probability vs Likelihood

What is the difference between probability and likelihood?

Answer

Probability and likelihood use the same joint density $p(x \mid \theta)$ but read it in opposite directions. Probability fixes the parameters $\theta$ and asks how plausible different data $x$ are; likelihood fixes the observed data and asks which parameter values would have made that data plausible. A probability distribution must sum or integrate to 1 over all possible data; a likelihood carries no such constraint over the parameters. Maximum likelihood estimation is exactly this direction flip: hold the sample fixed and choose $\theta$ to make the observed data as probable as possible.

(1) Same Formula, Two Directions: with $\theta$ fixed, $p(x \mid \theta)$ is a probability over data; with $x$ fixed at the observed sample, the same expression becomes the likelihood $L(\theta \mid x)$ , a function of $\theta$ .
(2) Normalization: probabilities integrate to 1 over all possible data, while likelihoods do not integrate to 1 over $\theta$ , so a likelihood is never “the probability of the parameter”.
(3) Where Each Is Used: probability drives prediction and simulation of unseen data; likelihood drives parameter estimation. Amazon’s DeepAR forecaster is trained by maximizing the likelihood of observed demand series, then at inference emits probability distributions over future demand.

Two panels: left, a probability density over data values with parameters fixed; right, a likelihood curve over the parameter theta with data fixed, peaking at the maximum likelihood estimate

Figure 1: The same density viewed two ways. With $\theta$ fixed, the curve over data values is a probability distribution and integrates to 1 (left). With the data fixed at the observed sample, the curve over $\theta$ is the likelihood (right), whose peak is the maximum likelihood estimate.

Mathematical Formulation:
$L(\theta \mid x) = p(x \mid \theta)$
$\hat{\theta}_{\mathrm{MLE}} = \arg\max_{\theta}\ \sum_{i=1}^{N} \log p(x_i \mid \theta)$

Where:

$x$ is the observed data and $\theta$ the model parameters.
$p(x \mid \theta)$ is the density of the data under parameters $\theta$ ; $L(\theta \mid x)$ is the same expression read as a function of $\theta$ .
$x_i$ with $i\in\{1,\ldots,N\}$ indexes independent samples, so the joint likelihood factorizes into a product and the logarithm turns it into a sum.

Feature	Probability	Likelihood
What Varies	The data $x$	The parameters $\theta$
What Is Fixed	The parameters $\theta$	The observed data $x$
Normalization	Binomial example: P(X = 3) = C(10,3)0.3^30.7^7 = 0.267	No constraint over $\theta$
Answers	“What data should I expect?”	“Which parameters fit the data I saw?”
Typical Use	Prediction, simulation, p-values	Estimation (MLE), model comparison

August 10, 2026

DL0050 Knowledge Distillation
Describe the process and benefits of knowledge distillation.
Answer
Knowledge distillation (KD) trains a small student model to imitate a large, accurate teacher model. The key trick is learning from the teacher’s temperature-softened output distribution (“dark knowledge”: e.g., a cat image looks a bit like a dog, nothing like a truck) rather than only from one-hot hard labels. The student ends up much smaller and faster while retaining most of the teacher’s accuracy, which is why KD is the standard route to deployable models.
(1) Soft Targets: Teacher logits are passed through softmax with a temperature $T > 1$ , exposing inter-class similarity structure that hard labels hide.
(2) Combined Loss: The student minimizes a mix of distillation loss (KL to the teacher’s soft targets) and ordinary cross-entropy on the true labels.
(3) Benefits: Compression and latency for edge/real-time deployment, plus a regularization effect: students often generalize better than the same architecture trained on hard labels alone.
Mathematical Formulation:
$q_i(T) = \mathrm{softmax}(z_i / T) = \frac{e^{z_i / T}}{\sum_{j=1}^{K} e^{z_j / T}}$
$\mathcal{L} = \alpha\, \mathcal{L}_{\text{CE}}(y, q^{\text{student}}) + (1 - \alpha)\, T^2\, \mathrm{KL}\!\left(q^{\text{teacher}}(T) \,\|\, q^{\text{student}}(T)\right)$
Where:
$z_i$ is the logit for class $i$ , $K$ the number of classes, and $T > 0$ the temperature; higher $T$ yields a smoother distribution.
$\alpha$ balances hard-label CE against distillation; the $T^2$ factor rescales the KL term because softening shrinks its gradients by $1/T^2$ .

Figure 1: Temperature smoothing: at $T = 1$ only the winner class is visible; at higher $T$ the class-similarity ratios (“dark knowledge”) emerge.
Training Setup: The teacher runs in inference mode (frozen); only the student’s weights update. Both models see the same inputs, and the two losses are computed on the student’s outputs only.

Figure 2: The KD setup: the student learns from both the teacher’s soft targets (distillation loss) and the ground truth (student loss).
Practical Caveats: A weak or biased teacher transfers its errors; $T$ and $\alpha$ need tuning; and an extremely small student may lack the capacity to absorb the teacher. Intermediate-feature distillation and task-specific data help close the gap.
Login to view more content
October 28, 2025