Interview for Machine Learning

Author: admin

DL0025 Attention Mechanism
Please explain the concept of “Attention Mechanism.”
Answer
The attention mechanism is a technique in neural networks that allows the model to focus on specific parts of the input sequence when making predictions. It addresses the limitation of traditional sequence-to-sequence models that compress an entire input sequence into a single fixed-size context vector, which can lose information, especially for long sequences.
Attention lets the model dynamically decide which parts of the input are most important for each output step. For each output token, attention computes a weighted sum over all input tokens. These weights represent how much “attention” the model should pay to each input.
Key Components:
Query (Q): Represents what we are looking for or the current element being processed.
Key (K): Represents what information is available from the input.
Value (V): The actual information content to be extracted if a key matches the query.
Each output uses a query to compare with keys and then uses the scores to weight values.
Calculation (Scaled Dot-Product Attention):
Similarity Score: Calculated by taking the dot product of the Query with each Key.
Scaling: The scores are scaled down by the square root of the dimension of the keys ( $d_k$ ) to reduce variance and prevent large values from pushing the Softmax function into regions with tiny gradients.
Normalization: Normalized into a probability distribution using the Softmax function. Ensures the weights sum to 1.
Weighted Sum: Multiplied by the Values to get the final attention output.
$\mbox{Attention}(Q, K, V) = \mbox{Softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$
Where:
$Q, K, V$ : Matrices of queries, keys, and values.
$d_k$ : Dimension of key vectors.
$\mbox{Softmax}$ : Converts similarity scores to probabilities.
The plot below shows how much “attention” each input token receives in a simplified attention mechanism. It uses softmax-normalized weights over a 5-token sentence.
Login to view more content
July 20, 2025
ML0065 Random Forest III
How to choose the number of features in a random forest?
Answer
Select the number of features ( $m$ ) using rules of thumb (default heuristics), then tune via cross-validation or out-of-bag (OOB) error to find the best value for your specific dataset.
Default Heuristics:
Classification: $m = \sqrt{p}$
Regression: $m = \frac{p}{3}$
Where:
$p$ = total number of features,
$m$ = number of features considered at each split.
Bias-Variance Trade-off:
(1) Smaller max_features will increase randomness, leading to less correlated trees (reducing variance) but potentially higher bias.
(2) Larger max_features will decrease randomness, leading to more correlated trees (increasing variance) but potentially lower bias.
Grid Search/Randomized Search:
This is the most robust method. Define a range of possible max_features values and use cross-validation to evaluate the model’s performance for each value.
Out-of-Bag (OOB) Error:
Random Forests can estimate the generalization error internally using OOB samples. You can monitor the OOB error as you vary max_features to find the optimal value.
The figure below shows the cross-validation accuracy curve when using different numbers of features.
Login to view more content
July 19, 2025
ML0064 Random Forest II
Please explain the benefits and drawbacks of random forest.
Answer
Random Forest is a powerful ensemble method that reduces overfitting and improves predictive accuracy by combining many decision trees. However, it trades interpretability and computational efficiency for these benefits and may require careful tuning when dealing with large, imbalanced, or sparse datasets.
Benefits of random forest:
(1) Reduces Overfitting: Aggregating many trees lowers variance.
(2) Robust to Noise and Outliers: Less sensitive to anomalous data.
(3) Handles High Dimensionality: Works well with many input features.
(4) Estimates Feature Importance: Helps identify influential variables.
(5) Built-in Bagging: Bootstrap sampling improves generalization.
Drawbacks of random forest:
(1) Less Interpretability: Hard to visualize or explain compared to a single decision tree.
(2) Computational Cost: Training and prediction can be slower with many trees.
(3) Memory Usage: Large forests can consume significant resources.
(4) Biased with Imbalanced Data: Class imbalance can lead to biased predictions.
(5) Not Always Optimal for Sparse Data: May underperform compared to other algorithms on very sparse datasets.
The example below demonstrates that the random forest sometimes underperforms on the imbalanced dataset.
Login to view more content
July 8, 2025
ML0063 Random Forest
How does the random forest algorithm operate? Please outline its key steps.
Answer
Random Forest builds an ensemble of decision trees using bootstrapped samples and random feature subsets at each split. This combination reduces variance, combats overfitting, and improves predictive accuracy. The final output aggregates the predictions of all trees (majority vote for classification, averaging for regression).
(1) Bootstrap Sampling: Create multiple subsets of the original training data by sampling with replacement (bootstrap samples).
(2) Grow Decision Trees: For each bootstrap sample, train an unpruned decision tree.
(3) Random Feature Selection: At each split in a tree, randomly select a subset of features. The split is chosen only among this random subset (increases diversity).
(4) Aggregate Results with Voting or Averaging:
Classification: Each tree votes for a class label. The majority vote is used.
$\hat{y} = \mathrm{mode}\, { T_b(x) },\quad b=1,\ldots,B$
Where:
$T_b(x)$ = prediction of the b-th tree.
$B$ = total number of trees.
Regression: Each tree predicts a numeric value. The average is used.
$\hat{y} = \frac{1}{B}\sum_{b=1}^{B} T_b(x)$
Where:
$T_b(x)$ = prediction of the b-th tree.
$B$ = total number of trees.
The example below shows the decision boundary differences between three decision trees and their random forest ensemble.
Login to view more content
July 4, 2025
ML0062 Decision Tree
Please explain how a decision tree works.
Answer
A decision tree partitions the input space into regions by recursively splitting on features that best separate the target variable. Each split aims to improve the “purity” of the resulting subsets, as measured by criteria such as Gini impurity or Entropy. Predictions are made by following the sequence of splits down to a leaf node and returning the most common class (classification) or average target (regression).
Structure: A tree of nodes where each internal node tests a feature, branches represent feature outcomes, and leaves give predictions.
Splitting Criterion: Chooses the best feature (and threshold) by maximizing purity—e.g., Information Gain, Gini Impurity, or Variance Reduction.
Recursive Growth: Starting at the root, data is split, then the process recurses on each subset until stopping criteria (max depth, min samples, or pure leaves) are met.
Prediction: A new sample “travels” from root to leaf by following feature-test branches; the leaf’s label or value is returned.
The example below demonstrates using a Decision Tree on a 2-feature dataset for classification.
Login to view more content
June 30, 2025
ML0061 KNN and K-means
What are the key differences between KNN and K-means?
Answer
KNN(K-Nearest Neighbors) is a supervised algorithm that classifies data by considering the labels of its nearest neighbors, emphasizing prediction based on historical data. In contrast, K-Means is an unsupervised clustering technique that groups data together based solely on their similarity, without using any labels.
Here are the key differences between KNN and K-Means:
(1) Learning Type
KNN: Supervised learning algorithm (used for classification/regression).
K-Means: Unsupervised learning algorithm (used for clustering).
(2) Objective
KNN: Predict the label of a new sample based on the majority vote (or average) of its K nearest neighbors.
K-Means: Partition the dataset into K clusters by minimizing intra-cluster distance.
(3) Training
KNN: No explicit training; It simply stores the entire training dataset.
K-Means: Involves an iterative training process to learn cluster centroids.
(4) Prediction
KNN: Computationally expensive, computes the distance from the test point to every training point. Sorts the distances and selects the top $K$ nearest neighbors. The majority votes for classification. Average of values for regression.
K-Means: Fast and simple for inference, compute the distance of any new data point to each of the $K$ centroids. Assign it to the nearest centroid (i.e., predicted cluster).
(5) Distance Metric Use
KNN: Used to find neighbors.
K-Means: Used to assign points to the nearest cluster center.
(6) Output
KNN: Outputs a label (classification) or value (regression).
K-Means: Outputs cluster assignments and centroids.
The table below summarizes the comparison between KNN and K-Means.
Login to view more content
June 18, 2025
ML0060 K Selection in K-Means
How to select K in K-Means?
Answer
To select the optimal number of clusters $K$ in K-Means, use the visual plot like the elbow method, quantitative metrics like the silhouette score, or statistical methods like the gap statistic. These help balance model fit and generalization without overfitting.
Elbow Method:
(1) Plot the within-cluster sum of squares (WCSS) vs. $K$ .
(2) Choose the “elbow” point where the rate of improvement slows.
WCSS can be calculated using the following equation:
$\text{WCSS}(K) = \sum_{k=1}^{K} \sum_{x_i \in C_k} |x_i - \mu_k|^2$
Where:
$C_k$ is cluster $k$ ,
$\mu_k$ is its centroid.
Here is one plot example to demonstrate the location of the elbow point.
Silhouette Score:
The silhouette score measures how well each point lies within its cluster. It ranges from -1 (wrong clustering) to 1 (well-clustered).
(1) Calculate the average silhouette score for different $K$ values.
(2) Choose the $K$ that yields the highest average silhouette score.
Silhouette coefficient for point $i$ can be calculated by the following equation.
$s(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}$
Where:
$a(i)$ = intra-cluster distance,
$b(i)$ = nearest-cluster distance.
Gap Statistic:
(1) Compares clustering against a random reference distribution.
(2) Choose $K$ that maximizes the gap between observed and expected WCSS.
Login to view more content
June 14, 2025
ML0059 K-means II
K-Means is widely used for clustering. Can you discuss its main benefits as well as its disadvantages?
Answer
K-Means clustering is a widely used unsupervised learning algorithm that partitions data points into K clusters, where each point belongs to the cluster with the nearest mean. While it is computationally efficient and easy to implement, it relies on prior specification of the number of clusters, assumes spherical clusters, and is sensitive to initialization and outliers.
Objective Function of K-means:
$J = \sum_{i=1}^{K} \sum_{x_j \in C_i} |x_j - \mu_i|^2$
Where:
$K$ is the number of clusters.
$C_i$ is the set of points in cluster $i$ .
$\mu_i$ is the centroid of cluster $i$ .
$x_j$ is a data point assigned to cluster $i$ .
Main Benefits of K-means:
(1) Simple & Efficient: Fast to compute, easy to implement.
(2) Scalable: Handles large datasets well.
(3) Unsupervised Learning: Requires no labeled data.
(4) Interpretable: Cluster centroids are intuitive and interpretable.
(5) Works Well on Spherical Clusters: Performs best when clusters are compact and well-separated.
Main Disadvantages of K-means:
(1) Must Specify $K$ : The number of clusters must be known in advance.
(2) Sensitive to Initialization: Poor starting points may lead to suboptimal clustering.
(3) Assumes Spherical Clusters: Fails on clusters with irregular shapes or varying densities.
(4) Affected by Outliers: Outliers can skew centroids and degrade performance.
(5) Only Uses Euclidean Distance: Not suitable for non-numeric or categorical features.
Examples below demonstrate K-Means performance on spherical clusters and Irregular shape clusters.
Login to view more content
June 12, 2025
ML0058 K-means++
Please explain how K-means++ works.
Answer
K-means++ is an improved way to initialize centroids in K-means. K-means++ selects initial centroids one by one using a weighted probability based on squared distances from already chosen centroids. This spreads out the centroids more effectively, reducing the chances of poor clustering and helping the algorithm converge faster and more reliably.
K-means++ Steps:
(1) Choose the first centroid $\mu_1$ uniformly at random from the dataset.
(2) For each point $x_i$ , compute its squared distance to the nearest chosen centroid:
$D(x_i)^2 = \min_{1 \le j \le m} |x_i - \mu_j|^2$
Where:
$\mu_j$ is one of the already chosen centroids.
(3) Choose the next centroid $\mu_{m+1}$ with probability:
$P(x_i) = \frac{D(x_i)^2}{\sum_j D(x_j)^2}$
Where:
$D(x_i)^2$ is the squared distance from point $x_i$ to its nearest already chosen centroid.
$\sum_j D(x_j)^2$ is the sum of minimum squared distances from all data points to their nearest chosen centroid.
(4) Repeat until $K$ centroids are chosen.
(5) Then proceed with standard K-means clustering.
Below shows an example for K-means++ clustering.
Login to view more content
June 12, 2025
ML0057 K-means
Please explain how K-means works.
Answer
K-means is an iterative unsupervised algorithm that groups data into $K$ clusters by minimizing intra-cluster distances. It alternates between assigning points to the nearest centroid and updating centroids until convergence. It is fast and easy to implement, but sensitive to initialization and non-convex cluster shapes.

Goal of K-means: Partition data into $K$ clusters by minimizing within-cluster variance.
K-means Steps:
(1) Initialization: Randomly choose $K$ centroids.
(2) Assignment step: Assign each point to the closest centroid using Euclidean distance, given by:
$d(x, c_k) = \sqrt{\sum_{i=1}^{n} (x_i - c_{k,i})^2}$
Where:
$x$ is the data point.
$c_k$ is the cluster center.
(3) Update Step: Compute new cluster centers as the mean of all points assigned to that cluster by:
$c_k = \frac{1}{C_k} \sum_{x \in C_k} x$
Where:
$C_k$ represents the set of points assigned to cluster $k$ .
(4) Convergence: Repeat assignment and update steps until cluster centers stabilize or a stopping criterion is met.
Below shows an example for K-means clustering.
Login to view more content
June 12, 2025