Interview for Machine Learning

Category: Medium

ML0015 ROC Curve
What is the ROC Curve, and how is it plotted?
Answer
The ROC (Receiver Operating Characteristic) curve is a graphical tool for evaluating a binary classifier by plotting the true positive rate against the false positive rate at every classification threshold. Because models output scores rather than hard labels, a threshold decides what counts as positive; sweeping that threshold from strict to lenient traces out the curve. To plot it: train the binary classifier, generate probability scores for the positive class, compute TPR and FPR at many threshold values, then plot TPR versus FPR. Reading the curve: points near the top-left corner mean high TPR with low FPR: excellent performance; the diagonal is random guessing; curves below the diagonal are worse than random (and can be flipped to beat it); a perfect classifier sits at the single point (0, 1).
(1) Axes: Y is TPR (recall, sensitivity); X is FPR (1 − specificity); both sweep with the threshold.
(2) How It Is Built: Score the data, then recompute the confusion counts at every candidate threshold and connect the points.
(3) Reading It: Closer to the top-left corner is better; the diagonal is random; (0, 1) is perfect.

Figure 1: Reference ROC curves: the further a model bows toward the top-left corner, the better it ranks positives above negatives.
Mathematical Formulation:
$\text{TPR}(\tau) = \frac{TP(\tau)}{TP(\tau) + FN(\tau)}$
$\text{FPR}(\tau) = \frac{FP(\tau)}{FP(\tau) + TN(\tau)}$
Where:
$\tau$ is the classification threshold; varying it from 1 down to 0 traces the curve.
$TP(\tau)$ , $FP(\tau)$ , $FN(\tau)$ , and $TN(\tau)$ are the confusion-matrix counts at threshold $\tau$ .
$\text{TPR}$ is the true positive rate (same as recall) and $\text{FPR}$ is the false positive rate, the curve’s two axes.
Login to view more content
March 27, 2025
ML0007 Dropout
What is dropout in neural network training?
Answer
Dropout is a regularization technique used during neural network training to prevent overfitting. At each training step, a fraction of neurons (and their connections) are randomly “dropped out”, meaning their activations are set to zero. This forces the network to learn more robust features, because it cannot rely on any single neuron; instead it learns distributed representations, effectively training an ensemble of many smaller sub-networks that share weights. At inference time every neuron is active, so the model uses all learned features without randomness. To bridge the train/inference gap, inverted dropout scales the active neurons by $1/(1-p)$ during training so no adjustment is needed at inference; the alternative standard dropout scales the weights by $(1-p)$ at inference instead.
(1) Mechanism: Randomly zero a fraction $p$ of activations per training step; the dropped set changes every step.
(2) Why It Works: Prevents co-adaptation of neurons and approximates training an exponentially large ensemble of sub-networks.
(3) Training vs Inference: Inference uses the full network; activations are rescaled (during training with inverted dropout, or at inference with standard dropout) so expected magnitudes match.

Figure 1: Left: the full network. Right: one training step with dropout: grayed, crossed-out neurons are zeroed, forcing the remaining sub-network to carry the prediction.
Mathematical Formulation:
$\tilde{h}_i = \frac{m_i}{1-p} \cdot h_i$
$m_i \sim \mathrm{Bernoulli}(1-p)$
Where:
$\tilde{h}_i$ is the scaled activation of neuron $i$ actually passed to the next layer during training.
$h_i$ is the original activation of neuron $i$ , and $i$ indexes the neurons in a layer.
$m_i$ is the binary dropout mask: 1 keeps the neuron, 0 drops it.
$p$ is the drop probability; $1-p$ is the keep probability, and dividing by it is the inverted-dropout scaling that keeps $\mathbb{E}[\tilde{h}_i] = h_i$ .

Figure 2: Choosing the drop probability $p$ : too little dropout leaves overfitting, too much underfits; the test-error minimum is the sweet spot (typically $p$ around 0.2–0.5).
Login to view more content
March 16, 2025
ML0006 Cross-Validation
What are the common cross-validation techniques?
Answer
Cross-validation is a statistical method used to evaluate the performance and generalizability of a model by rotating which part of the data serves as the validation set, so every sample is used for both training and validation. The most common technique is k-Fold Cross-Validation: the data is divided into $k$ equal folds, the model is trained $k$ times, each time on $k-1$ folds with the remaining fold held out, and the final score is the average over all $k$ runs. Leave-One-Out Cross-Validation (LOOCV) is the special case where $k$ equals the number of data points. Stratified k-Fold preserves the class distribution inside every fold, which matters for imbalanced datasets. Time Series Cross-Validation builds folds that respect temporal order, preventing future data from leaking into training.
(1) Core Idea: Rotate the validation fold and average the scores, giving a more reliable estimate than a single train/validation split.
(2) Choosing The Variant: $k=5$ or $10$ is the default, LOOCV suits tiny datasets, stratified folds suit class imbalance, and TSCV is mandatory for sequential data.
(3) Why It Matters: Every sample gets validated on, so the estimate has lower variance and uses all the data, at the cost of training $k$ times.

Figure 1: Four cross-validation schemes. Orange blocks are validation folds: they rotate (k-fold), shrink to one sample (LOOCV), keep class ratios (stratified), or move forward in time (TSCV).
Mathematical Formulation:
$\mathrm{CV}_k = \frac{1}{k}\sum_{i=1}^{k}\mathcal{E}_i$
$\mathcal{E}_i = \mathcal{L}\big(\hat{f}_{-i}, D_i\big)$
Where:
$\mathrm{CV}_k$ is the cross-validation score, the average error across all $k$ folds.
$\mathcal{E}_i$ is the validation error on fold $i$ , and $i\in\{1,\ldots,k\}$ indexes the folds.
$\hat{f}_{-i}$ is the model trained on all folds except fold $i$ , and $D_i$ is the held-out fold.
$\mathcal{L}$ is the evaluation loss or metric (e.g., error rate, log-loss).
Login to view more content
February 27, 2025
ML0005 Discriminative and Generative
What are the differences between discriminative and generative models?
Answer
Discriminative models learn to draw a boundary between classes: they model the conditional probability $P(y \mid x)$ directly, mapping features $x$ to labels $y$ without modeling how the data was generated. Examples include logistic regression, support vector machines, and neural network classifiers. Generative models instead estimate the joint probability $P(x, y)$ (via $P(x \mid y)$ and $P(y)$ ), capturing how the data itself is generated, and then use Bayes’ theorem to derive $P(y \mid x)$ for classification. Examples include Naive Bayes, hidden Markov models, VAEs, and GANs. Because generative models learn the data distribution, they can also sample new data; discriminative models can only separate existing classes.
(1) Modeling Target: Discriminative models learn $P(y \mid x)$ directly; generative models learn $P(x, y)$ or $P(x \mid y) P(y)$ .
(2) Decision Mechanism: Discriminative models find a decision boundary; generative models compare class-conditional densities through Bayes’ rule.
(3) Capabilities: Discriminative models only classify; generative models can also generate samples, handle missing features, and score outliers.

Figure 1: Left: a discriminative model cares only about the boundary between classes. Right: a generative model fits a distribution to each class, capturing how the data itself looks.
Mathematical Formulation:
$\text{discriminative:} \quad \hat{y} = \arg\max_y \, P(y \mid x)$
$\text{generative:} \quad P(y \mid x) = \frac{P(x \mid y)\, P(y)}{P(x)}$
Where:
$\hat{y}$ is the predicted label and $y$ ranges over the class set for input $x$ .
$P(y \mid x)$ is the posterior, the quantity a discriminative model learns directly.
$P(x \mid y)$ is the class-conditional likelihood (how data of class $y$ looks) and $P(y)$ the class prior; together they define the joint $P(x, y)$ a generative model learns.
$P(x)$ is the evidence normalizer, computable as $\sum_y P(x \mid y) P(y)$ over all classes.
Login to view more content
February 26, 2025