Category: Easy

ML0068 Type I vs Type II Errors
What are Type I and Type II errors, and how do they trade off?
Answer
A Type I error (false positive) rejects a true null hypothesis: you announce an effect that is not there. A Type II error (false negative) fails to reject a false null: a real effect goes undetected. Their rates are $\alpha$ , the significance level you choose up front, and $\beta$ , with power equal to $1 - \beta$ . For a fixed sample size the two trade off through the decision threshold: making the test stricter lowers $\alpha$ but raises $\beta$ , and only more data or lower metric variance reduces both at once.
(1) Type I ( $\alpha$ ): rejecting a true null, the “convicting the innocent” error. It is controlled by fixing $\alpha$ before looking at data and by correcting when many hypotheses are tested at once.
(2) Type II ( $\beta$ ): missing a real effect. Power $1 - \beta$ grows with sample size, true effect size, and lower metric variance.
(3) Costs Set the Threshold: the right balance is a business decision. Gmail blocks over 99.9% of spam while misrouting only about 0.05% of legitimate mail, because a false positive (lost mail) costs far more than a false negative (spam seen in the inbox).

Figure 1: Sampling distribution of the test statistic under $H_0$ (left) and $H_1$ (right) with one decision threshold. The right tail of $H_0$ is $\alpha$ (Type I), the part of $H_1$ left of the threshold is $\beta$ (Type II), and the rest of $H_1$ is power.
Mathematical Formulation:
$\alpha = \Pr(\text{reject } H_0 \mid H_0 \text{ true})$
$\beta = \Pr(\text{fail to reject } H_0 \mid H_1 \text{ true})$
$\text{power} = 1 - \beta$
Where:
$H_0$ is the null hypothesis (no effect) and $H_1$ the alternative (a real effect exists).
$\alpha$ is the significance level chosen before the test; $\beta$ is the false negative rate under a specific true effect.
Power is the probability of detecting the effect when it is real, and it rises with sample size, effect size, and reduced variance.

Figure 2: Moving the threshold at fixed sample size swaps the errors: stricter (left panel) shrinks $\alpha$ and inflates $\beta$ ; looser (right panel) does the reverse. Shrinking both at once requires more data or lower variance, which pulls the two curves apart.
Decision \ Reality H0 True (No Effect) H1 True (Real Effect)
Reject H0 Type I error, rate $\alpha$ Correct detection, power $1 - \beta$
Fail to Reject H0 Correct, rate $1 - \alpha$ Type II error, rate $\beta$
Login to view more content
August 10, 2026

Decision \ Reality	H0 True (No Effect)	H1 True (Real Effect)
Reject H0	Type I error, rate $\alpha$	Correct detection, power $1 - \beta$
Fail to Reject H0	Correct, rate $1 - \alpha$	Type II error, rate $\beta$

ML0067 Probability vs Likelihood

What is the difference between probability and likelihood?

Answer

Probability and likelihood use the same joint density $p(x \mid \theta)$ but read it in opposite directions. Probability fixes the parameters $\theta$ and asks how plausible different data $x$ are; likelihood fixes the observed data and asks which parameter values would have made that data plausible. A probability distribution must sum or integrate to 1 over all possible data; a likelihood carries no such constraint over the parameters. Maximum likelihood estimation is exactly this direction flip: hold the sample fixed and choose $\theta$ to make the observed data as probable as possible.

(1) Same Formula, Two Directions: with $\theta$ fixed, $p(x \mid \theta)$ is a probability over data; with $x$ fixed at the observed sample, the same expression becomes the likelihood $L(\theta \mid x)$ , a function of $\theta$ .
(2) Normalization: probabilities integrate to 1 over all possible data, while likelihoods do not integrate to 1 over $\theta$ , so a likelihood is never “the probability of the parameter”.
(3) Where Each Is Used: probability drives prediction and simulation of unseen data; likelihood drives parameter estimation. Amazon’s DeepAR forecaster is trained by maximizing the likelihood of observed demand series, then at inference emits probability distributions over future demand.

Two panels: left, a probability density over data values with parameters fixed; right, a likelihood curve over the parameter theta with data fixed, peaking at the maximum likelihood estimate

Figure 1: The same density viewed two ways. With $\theta$ fixed, the curve over data values is a probability distribution and integrates to 1 (left). With the data fixed at the observed sample, the curve over $\theta$ is the likelihood (right), whose peak is the maximum likelihood estimate.

Mathematical Formulation:
$L(\theta \mid x) = p(x \mid \theta)$
$\hat{\theta}_{\mathrm{MLE}} = \arg\max_{\theta}\ \sum_{i=1}^{N} \log p(x_i \mid \theta)$

Where:

$x$ is the observed data and $\theta$ the model parameters.
$p(x \mid \theta)$ is the density of the data under parameters $\theta$ ; $L(\theta \mid x)$ is the same expression read as a function of $\theta$ .
$x_i$ with $i\in\{1,\ldots,N\}$ indexes independent samples, so the joint likelihood factorizes into a product and the logarithm turns it into a sum.

Feature	Probability	Likelihood
What Varies	The data $x$	The parameters $\theta$
What Is Fixed	The parameters $\theta$	The observed data $x$
Normalization	Binomial example: P(X = 3) = C(10,3)0.3^30.7^7 = 0.267	No constraint over $\theta$
Answers	“What data should I expect?”	“Which parameters fit the data I saw?”
Typical Use	Prediction, simulation, p-values	Estimation (MLE), model comparison

August 10, 2026

ML0065 Random Forest III
How to choose the number of features in a random forest?
Answer
Select the number of features considered at each split (the $m$ , or max_features) by starting from the default heuristics, then tuning with cross-validation or out-of-bag (OOB) error to find the best value for your specific dataset. The choice trades bias against variance and accuracy against training cost.
(1) Default Heuristics: Classification: $m = \sqrt{p}$ ; regression: $m = p/3$ ; solid starting points.
(2) Bias-Variance Trade-Off: Smaller $m$ adds randomness: less correlated trees (lower variance) but potentially higher bias; larger $m$ strengthens each tree (lower bias) but correlates their errors (higher variance).
(3) Systematic Search: Grid or randomized search over a range of values with cross-validation is the most robust method; OOB error offers a validation-free alternative unique to bagged models.

Figure 1: CV accuracy across max_features on a 30-feature dataset: accuracy is poor when m is tiny (trees too weak), then plateaus with a noisy peak at m = 15; the $\sqrt{p} \approx 5.5$ heuristic lands inside the good region, and CV refinement picks the best value on the plateau.
Mathematical Formulation:
$m = \sqrt{p} \quad \text{(classification)}$
$m = \frac{p}{3} \quad \text{(regression)}$
Where:
$p$ is the total number of features in the dataset.
$m$ is the number of features randomly drawn and considered at each split (max_features in most libraries).
These are heuristics, not optima; cross-validation or OOB error refines them per dataset.
Login to view more content
July 19, 2025
ML0063 Random Forest
How does the random forest algorithm operate? Please outline its key steps.
Answer
Random Forest builds an ensemble of decision trees, each trained on a bootstrapped sample of the data with a random feature subset considered at each split. This combination reduces variance, combats overfitting, and improves predictive accuracy; the final output aggregates all trees’ predictions: majority vote for classification, averaging for regression.
(1) Bootstrap Sampling: Create multiple subsets of the training data by sampling with replacement (bootstrap samples).
(2) Grow Decision Trees: Train an unpruned decision tree on each bootstrap sample.
(3) Random Feature Selection: At every split in every tree, consider only a random subset of features; this increases diversity between trees.
(4) Aggregate: Classification: each tree votes for a class and the majority wins; regression: the tree outputs are averaged.

Figure 1: Three trees, three different jagged boundaries: each overfits its own bootstrap sample in its own way. The ensemble’s boundary (bottom right) averages the votes and lands smoother and closer to the true structure: the trees’ individual errors cancel.
Mathematical Formulation:
$\hat{y} = \mathrm{mode}\big\{ T_b(x) \big\},\quad b = 1, \ldots, B$
$\hat{y} = \frac{1}{B} \sum_{b=1}^{B} T_b(x)$
Where:
$T_b(x)$ is the prediction of the $b$ -th tree for input $x$ .
$B$ is the total number of trees in the forest.
First line: classification by majority vote (mode); second line: regression by averaging.

Figure 2: The full pipeline: B bootstrap replicas of the training set feed B independently grown trees (each with random feature subsets at its splits), and a majority-vote / averaging box fuses their outputs into one robust prediction.
Login to view more content
July 4, 2025
ML0062 Decision Tree
Please explain how a decision tree works.
Answer
A decision tree partitions the input space into regions by recursively splitting on the feature that best separates the target variable. Each split aims to improve the “purity” of the resulting subsets, measured by criteria such as Gini impurity or entropy. Predictions follow the sequence of splits down to a leaf, returning the most common class (classification) or the average target (regression).
(1) Structure: A tree of nodes: internal nodes test a feature, branches represent the outcomes, leaves give predictions.
(2) Splitting Criterion: Choose the best feature and threshold by maximizing purity: information gain (entropy), Gini impurity, or variance reduction for regression.
(3) Recursive Growth: Starting at the root, split the data, then recurse on each subset until stopping criteria are met (max depth, min samples, or pure leaves).
(4) Prediction: A new sample travels from root to leaf following the feature tests; the leaf’s label or value is returned.

Figure 1: A decision tree from two viewpoints: on the left, the axis-aligned rectangular regions it carves into the feature space; on the right, the tree itself: each node shows its test, impurity, and class counts, and each leaf is a final answer.
Mathematical Formulation:
$\mathrm{Gini}(t) = 1 - \sum_{k=1}^{K} p_k^2$
$\mathrm{Entropy}(t) = -\sum_{k=1}^{K} p_k \log_2(p_k)$
$\mathrm{Information\ Gain} = \mathrm{Entropy}(\mathrm{Parent}) - \sum_{i} \frac{N_i}{N} \, \mathrm{Entropy}(\mathrm{Child}_i)$
Where:
$t$ is a tree node; $K$ the number of classes; $p_k$ the proportion of class $k$ samples in node $t$ .
Gini = 0 means the node is pure (one class only) and grows with mixing; entropy = 0 at perfect purity and is maximal when classes are uniformly mixed.
In the gain formula, $N$ is the parent’s sample count and $N_i$ child $i$ ‘s: the split chosen is the one maximizing this weighted impurity drop.

Figure 2: The two classification criteria compared for a binary node: both peak at $p = 0.5$ (maximally mixed) and vanish at pure nodes; they nearly always rank candidate splits in the same order, which is why Gini (no logarithms) is the common default.
Login to view more content
June 30, 2025

ML0061 KNN and K-means

What are the key differences between KNN and K-means?

Answer

KNN (K-Nearest Neighbors) is a supervised algorithm that classifies data using the labels of its nearest neighbors: prediction from historical labeled data. K-means is an unsupervised clustering technique that groups data purely by similarity, using no labels at all. Despite the shared “K”, they solve different problems.

(1) Learning Type: KNN is supervised (classification/regression); K-means is unsupervised (clustering).
(2) Objective: KNN predicts a new sample’s label from the majority vote (or average) of its K nearest neighbors; K-means partitions the dataset into K clusters by minimizing intra-cluster distance.
(3) Training: KNN has no explicit training: it stores the entire dataset; K-means iteratively learns cluster centroids.
(4) Prediction Cost: KNN is expensive at prediction (distance to every training point, sort, take the top K, then vote/average); K-means is cheap: distances to K centroids, assign the nearest.
(5) Distance Use And Output: KNN uses distance to find neighbors and outputs a label or value; K-means uses distance to assign points to centroids and outputs cluster assignments plus centroids.

Feature	K-Nearest Neighbors (KNN)	K-Means
Type	Supervised Learning	Unsupervised Learning
Task	Classification, Regression	Clustering
Data Required	Labeled data	Unlabeled data
Training Phase	Stores all training data (lazy learner)	Iterative centroid calculation
Prediction Phase	Finds K nearest neighbors and assigns label/value	Assigns new points to closest cluster centroid
Compute Cost	High at prediction (distance calculations for each new point)	High at training (iterative updates); low at prediction (centroid assignment)
Goal	Predict label/value for new data	Group data into K clusters
Output	Class label or predicted value	Data points assigned to clusters

Mathematical Formulation:
$\hat{y} = \arg\max_{c \in \mathcal{C}} \; \sum_{i=1}^{K} \mathbb{1}(y_i = c)$
$\min_{C_1, \ldots, C_K} \; \sum_{k=1}^{K} \sum_{x \in C_k} \|x - \mu_k\|^2$

Where:

KNN (first line): $\hat{y}$ is the predicted class, $\mathcal{C}$ the class set, $y_i$ the label of the $i$ -th nearest neighbor, and $\mathbb{1}(\cdot)$ the vote-counting indicator.
K-means (second line): $C_k$ is cluster $k$ and $\mu_k$ its centroid; the algorithm minimizes total within-cluster squared distance: no labels appear anywhere.

Two panels on crescent moon data showing KNN separating the moons correctly while K-means cuts through them

Figure 1: Same data, different goals: with labels, KNN’s local votes trace the two crescents perfectly (left); without labels, K-means can only split by nearest-centroid geometry and slices through both moons (right). KNN performs well here thanks to local decision-making; K-means fails because it assumes spherical clusters with linear boundaries.

June 18, 2025

ML0059 K-means II
K-Means is widely used for clustering. Can you discuss its main benefits as well as its disadvantages?
Answer
K-means partitions data into $K$ clusters where each point belongs to the cluster with the nearest mean. It is computationally efficient and easy to implement, but it requires specifying the number of clusters in advance, assumes spherical clusters, and is sensitive to initialization and outliers.
(1) Simple And Efficient: Fast to compute, easy to implement, and scalable to large datasets.
(2) Unsupervised And Interpretable: Needs no labels, and the centroids are intuitive summaries of each cluster.
(3) Strong Assumptions: Works best on compact, well-separated spherical clusters; the user must fix $K$ beforehand.
(4) Fragility: Poor initial centroids can lock in suboptimal results, outliers drag centroids, and plain Euclidean distance excludes non-numeric features.
Mathematical Formulation:
$J = \sum_{i=1}^{K} \sum_{x_j \in C_i} \|x_j - \mu_i\|^2$
Where:
$K$ is the number of clusters; $C_i$ the set of points in cluster $i$ .
$\mu_i$ is the centroid of cluster $i$ ; $x_j$ a data point assigned to it.
$J$ is the within-cluster sum of squared distances: the objective both steps of the algorithm drive down monotonically.

Figure 1: The assumption made visible: on spherical, well-separated blobs (left) K-means nails the structure; on two crescents (right) its linear, centroid-based split slices through both moons: the spherical-cluster assumption failing in action.
Login to view more content
June 12, 2025
ML0057 K-means
Please explain how K-means works.
Answer
K-means is an iterative unsupervised algorithm that partitions data into $K$ clusters by minimizing intra-cluster distances (within-cluster variance). It alternates between assigning points to the nearest centroid and recomputing centroids until convergence. It is fast and easy to implement, but sensitive to initialization and to non-convex cluster shapes.
(1) Initialization: Choose $K$ initial centroids (randomly, or with K-means++).
(2) Assignment Step: Assign every point to its closest centroid by Euclidean distance.
(3) Update Step: Recompute each centroid as the mean of the points assigned to it.
(4) Convergence: Repeat assignment and update until the centroids stabilize or a stopping criterion is met.

Figure 1: K-means (K=3) converged on blob data: colors mark the final assignments and the red X’s are the centroids: each is the mean of its cluster, and every point belongs to its nearest centroid’s cluster.
Mathematical Formulation:
$d(x, c_k) = \sqrt{\sum_{i=1}^{n} (x_i - c_{k,i})^2}$
$c_k = \frac{1}{|C_k|} \sum_{x \in C_k} x$
Where:
$x$ is a data point and $c_k$ the centroid of cluster $k$ (first formula: Euclidean assignment distance, $n$ = number of features).
$C_k$ is the set of points currently assigned to cluster $k$ ; the update sets the centroid to their component-wise mean, the minimizer of squared distances for a fixed assignment.
Login to view more content
June 12, 2025
ML0055 KNN Regression
Please explain how KNN Regression works.
Answer
K-Nearest Neighbors (KNN) Regression is a simple, instance-based algorithm that predicts a continuous output by finding the $K$ nearest training points and averaging their target values. Its simplicity makes it easy to understand and implement, but performance is sensitive to $K$ and the distance metric, and prediction is computationally expensive on large datasets since every query compares against all training samples. Like its classification sibling, KNN regression is a “lazy learner”: it builds no explicit model and simply memorizes the training data.
(1) Instance-Based: No explicit model is learned; predictions come from stored training data and similarity.
(2) Distance-Based: Find the $K$ nearest neighbors of the query point (commonly Euclidean distance).
(3) Averaging Neighbors: The prediction is the mean of those neighbors’ target values.
(4) Sensitive To K And Metric: Small $K$ tracks noise; large $K$ oversmooths.
(5) No Training Phase: All computation happens during prediction.

Figure 1: KNN regression (K=5) on a noisy sine wave: each prediction averages the 5 nearest targets, producing a locally-flat, step-like curve that follows the trend but clips the peaks and troughs, the signature of local averaging.
Mathematical Formulation:
$\hat{y} = \frac{1}{K} \sum_{i=1}^{K} y_i$
$\hat{y}(x_q) = \frac{\sum_{i=1}^{K} w_i \cdot y_i}{\sum_{i=1}^{K} w_i}$
$w_i = \frac{1}{d(x_q, x_i) + \epsilon}$
Where:
$\hat{y}$ is the prediction for the query point; $y_i$ the target of the $i$ -th nearest neighbor; $K$ the number of neighbors (first formula: uniform average).
In the weighted variant, $w_i$ is inversely proportional to the distance $d(x_q, x_i)$ between query and neighbor: closer points influence the prediction more.
$\epsilon$ is a small constant avoiding division by zero when a neighbor sits exactly on the query.
Login to view more content
June 12, 2025
ML0054 KNN Classification
Please explain how KNN classification works.
Answer
K-Nearest Neighbors (KNN) is a simple, non-parametric algorithm that predicts a label by majority vote among the $K$ nearest neighbors of a test point, using a chosen distance metric. It is intuitive and effective for small datasets, though less efficient on large-scale data.
(1) Instance-Based Method: KNN learns no explicit model; it stores the training data and predicts from similarity.
(2) Distance-Based Classification: For a test point, it computes the distance to every training point (e.g., Euclidean).
(3) Majority Vote: It selects the $K$ closest neighbors and assigns the most frequent label among them.
(4) Sensitive To K And Metric: Performance depends on the choice of $K$ and the distance measure (Euclidean, Manhattan, …).
(5) No Training Phase: All computation happens at prediction time: hence “lazy learning”.

Figure 1: KNN (K=5) decision regions: the boundary wiggles because each query point polls its 5 nearest neighbors: local votes trace the class structure instead of fitting a global line. Note only Feature 1 is informative here; the vertical wiggle is the noise feature leaking into the vote.
Mathematical Formulation:
$\mathrm{distance}(x, y) = \sqrt{\sum_{i=1}^{n} (x_i - y_i)^2}$
$\hat{y} = \arg\max_{c \in \mathcal{C}} \; \sum_{i=1}^{K} \mathbb{1}(y_i = c)$
Where:
$x_i$ and $y_i$ are the $i$ -th features of the query and training points; $n$ is the number of features (first formula: Euclidean distance).
$\hat{y}$ is the predicted class for the query point; $\mathcal{C}$ is the set of all classes; $K$ is the number of neighbors polled.
$y_i$ is the label of the $i$ -th neighbor and $\mathbb{1}(y_i = c)$ the indicator counting that neighbor’s vote for class $c$ .
Login to view more content
June 11, 2025