Interview for Machine Learning

Tag: Basics

ML0045 Multi-Layer Perceptron
What is a Multi-Layer Perceptron (MLP)? How does it overcome Perceptron limitations?
Answer
A Multi-Layer Perceptron (MLP) is a feedforward neural network with one or more hidden layers between the input and output layers. Its hidden layers use non-linear activation functions (ReLU, sigmoid, or tanh) to model complex relationships, and it is trained with backpropagation, which adjusts the weights to minimize errors. MLPs are used for classification, regression, and function approximation.
(1) Hidden Layers + Non-Linearity: Each hidden unit applies a non-linear activation to its weighted sum, letting the network compose many linear maps into a curved function.
(2) Overcomes The Perceptron: Unlike a single-layer perceptron (restricted to linearly separable problems), an MLP learns non-linear decision boundaries and handles problems such as XOR.
(3) Universal Approximation: With enough neurons and layers, an MLP can approximate any continuous function, making it a powerful general-purpose model.

Figure 1: An MLP for 3-class classification: 4 inputs, two hidden layers (6 and 4 units) with non-linear activations, and 3 softmax-style outputs. Every layer is fully connected to the next.
Mathematical Formulation:
$h_l = f\big(W_l\, h_{l-1} + b_l\big)$
$\hat{y} = g\big(W_L\, h_{L-1} + b_L\big)$
Where:
$h_l$ is the activation vector of layer $l$ ( $h_0 = x$ is the input).
$W_l, b_l$ are the weight matrix and bias of layer $l$ , learned by backpropagation.
$f(\cdot)$ is the hidden non-linearity (ReLU/sigmoid/tanh); $g(\cdot)$ is the output map (softmax for classification, identity for regression).

Figure 2: The payoff: on XOR, the perceptron’s single line misclassifies half the positives, while the MLP’s hidden units carve a curved region that separates the classes perfectly.
Login to view more content
May 26, 2025
ML0044 Perceptron
Describe the Perceptron and its limitations.
Answer
The perceptron is a simple linear classifier that computes a weighted sum of input features, adds a bias, and applies a step function to produce a binary decision. It works well only for data that is linearly separable, where a straight line (or hyperplane in higher dimensions) can separate the classes.
(1) Linear Score: The perceptron combines inputs linearly as $w^T x + b$ ; geometry-wise this defines one hyperplane.
(2) Step Activation: A threshold turns the score into a hard 0/1 output. Note this non-linearity at the output still leaves the decision boundary linear.
(3) Limitations: It cannot solve non-linearly-separable problems like XOR, a single layer cannot model complex patterns, and it outputs bare binary values with no confidence or probability.

Figure 1: Perceptron structure: inputs $x_1, x_2, x_3$ and a constant 1 (for the bias) feed through weights into a single unit whose step activation emits the binary output.
Mathematical Formulation:
$y = f(w^T x + b)$
$f(z) = 1 \text{ if } z \geq 0;\quad f(z) = 0 \text{ otherwise}$
Where:
$y$ is the predicted output (0 or 1).
$w$ is the weight vector, $x$ the input vector, $b$ the bias term.
$f(\cdot)$ is the step activation; $z = w^T x + b$ is the linear pre-activation whose sign alone decides the class.

Figure 2: What one hyperplane can and cannot do: the left data is linearly separable and a perceptron solves it; the right XOR pattern defeats every single straight line, the classic motivation for hidden layers.
Login to view more content
May 26, 2025
DL0015 Cold Start
What is a “cold start” problem in deep learning?
Answer
The cold start problem is the difficulty of making reliable predictions for new entities (users, items, or contexts) that have little or no historical data. Models that depend on past interactions, especially recommender systems built on collaborative filtering, have no signal to learn a meaningful representation of a brand-new user or item, so their predictions degrade to near-random or popularity-biased guesses until data accumulates.
(1) Missing Interaction History: Collaborative filtering infers taste from a user-item matrix; a new row (user) or column (item) is empty, so the model cannot locate the entity in its embedding space.
(2) Feedback Loop Risk: Poor early predictions reduce engagement, which further slows data collection for the new entity. The problem compounds itself.
(3) Three Flavors: New-user cold start, new-item cold start, and new-system cold start (no data at all) each demand different remedies.

Figure 1: The new user row and new item column contain no interactions. Collaborative filtering has nothing to condition on for them.
Mitigation Strategies: The common theme is supplying side information until interaction data accumulates: transfer learning borrows representations from related domains; hybrid models mix collaborative signals with content features; and active onboarding explicitly gathers a few preferences from new users.
(1) Transfer Learning / Pre-trained Models: Initialize from embeddings or models trained on similar tasks so the new domain starts from useful structure rather than random weights.
(2) Hybrid Recommendation Models: Combine collaborative filtering with content-based features (user demographics, item metadata) so predictions remain reasonable with zero interactions.
(3) Active Learning / User Onboarding: Ask new users to rate a handful of popular or diverse items, turning cold start into a short warm-up phase.

Figure 2: All three strategies inject auxiliary signal into the recommender so cold entities get reasonable predictions before their interaction rows fill in.
Mathematical Formulation:
$\hat{r}_{ui} = \mu + b_u + b_i + p_u^{\top} q_i$
Where:
$\hat{r}_{ui}$ is the predicted rating of user $u$ for item $i$ ; $\mu$ is the global mean rating.
$p_u$ and $q_i$ are the learned latent factor vectors for user $u$ and item $i$ ; $b_u$ , $b_i$ are bias terms.
Cold start means $p_u$ or $q_i$ was never trained: with an empty interaction row/column, the factors stay at random init, so $\hat{r}_{ui}$ is meaningless.
Login to view more content
May 26, 2025
ML0042 Early Stopping
What is Early Stopping? How is it implemented?
Answer
Early stopping is a regularization technique that halts training when the model’s performance on a validation set stops improving, thus avoiding overfitting. It monitors a metric such as validation loss or validation accuracy and stops after a defined number of stagnant epochs (the patience). This ensures efficient training and better generalization.
(1) Split Data: Reserve a validation set separate from the training set.
(2) Evaluate Each Epoch: After every training epoch, measure performance on the validation set.
(3) Track Improvement: If performance improves, save the model and reset the patience counter; if not, increment the counter; when it reaches the patience, stop training.
(4) Restore Best Weights: After stopping, reload the weights from the epoch that yielded the best validation performance, not the final epoch.

Figure 1: Early stopping in action: training loss falls monotonically, but validation loss bottoms out at epoch 60 (ideal stop) and then rises as the model overfits. With patience 10, training actually halts at epoch 70 and the weights from epoch 60 are restored.
Mathematical Formulation:
$t^* = \arg\min_{t} \; \mathcal{L}_{\text{val}}\big(\theta_t\big)$
$\text{stop at } t^* + p \text{ if no epoch in } (t^*,\, t^* + p] \text{ beats } \mathcal{L}_{\text{val}}(\theta_{t^*})$
Where:
$\theta_t$ are the model weights after epoch $t$ ; $\mathcal{L}_{\text{val}}$ is the validation loss.
$t^*$ is the epoch with the best validation loss, the checkpoint whose weights are restored at the end.
$p$ is the patience: how many consecutive non-improving epochs are tolerated before stopping.
Login to view more content
May 25, 2025
ML0041 Concept of NN
Please explain the concept of a Neural Network.
Answer
A neural network (NN) is a machine learning model composed of layers of interconnected neurons. It learns patterns in data by adjusting weights through training, enabling tasks like classification, regression, and more. Neural networks are inspired by biology (they are computer systems modeled after the human brain’s network of neurons), and they excel at identifying complex, non-linear patterns, which makes them suitable for image recognition, natural language processing, and data classification.
(1) Layered Structure: A neural network consists of an input layer, one or more hidden layers, and an output layer; data flows from input to output through the hidden layers.
(2) Neurons And Activation: Each neuron computes a weighted sum of its inputs, adds a bias, and applies an activation function. Weights and biases are learnable parameters adjusted during training, and activation functions (e.g., ReLU, sigmoid) introduce the non-linearity that lets the network model complex relationships.
(3) Learning Process: The network learns by adjusting weights and biases through training algorithms such as backpropagation, minimizing the error between its predictions and the actual results.

Figure 1: A feedforward neural network: every neuron in one layer connects to every neuron in the next. Each connection carries a learned weight; each hidden and output neuron adds a bias and applies an activation function.
Mathematical Formulation:
$a_j = f\Big(\sum_{i} w_{ji}\, x_i + b_j\Big)$
Where:
$x_i$ are the inputs to the neuron (raw features for the input layer, previous-layer activations otherwise).
$w_{ji}$ is the weight on the connection from input $i$ to neuron $j$ ; $b_j$ is the neuron’s bias; both are learned during training.
$f(\cdot)$ is the activation function (ReLU, sigmoid, tanh, …); $a_j$ is the neuron’s output passed to the next layer.
Login to view more content
May 24, 2025
DL0014 Mixed Precision Training
Can you explain the primary benefits of using mixed precision training in deep learning?
Answer
Mixed precision training runs the compute-heavy parts of a model in FP16 while keeping an FP32 master copy of the weights, so training gets the speed and memory of half precision without sacrificing final accuracy. Modern GPU/TPU tensor cores execute FP16 matrix math several times faster than FP32, and halving activation memory lets you train larger models or use larger batches on the same hardware.
(1) Faster Training: FP16 tensor-core matmuls deliver up to an order of magnitude more throughput than FP32 on supported hardware (e.g., ~312 vs ~19.5 TFLOPS on an A100).
(2) Reduced Memory Usage: FP16 activations and working weight copies occupy half the bytes, freeing room for larger batch sizes or deeper models (master weights and optimizer states stay FP32, so total training memory falls by less than half).
(3) Maintained Accuracy: FP32 master weights plus loss scaling keep small gradient values representable, so final model quality matches full-precision training.

Figure 1: FP16 trades exponent range and mantissa precision for half the storage: gradients below $6.1 \times 10^{-5}$ would underflow to zero without loss scaling.
The Training Loop: Weights are stored in FP32 as the master copy. Each step casts them to FP16 for the forward and backward passes, multiplies the loss by a scale factor $S$ so that FP16 gradients stay in range, then divides the gradients by $S$ and applies the optimizer update to the FP32 master weights.

Figure 2: FP16 does the heavy math while the FP32 master copy absorbs tiny updates; loss scaling $S$ shifts gradients into FP16’s representable range.
Measured Benefits: On tensor-core hardware the speedup is substantial, and the halved activation memory (the dominant term at large batch sizes) directly translates into larger feasible models or batches.

Figure 3: Roughly 16x tensor-core throughput and half the activation memory are the headline wins; with Adam states kept in FP32, per-parameter training memory drops only modestly.
Mathematical Formulation:
$\mathcal{L}' = S \cdot \mathcal{L}$
$g_{fp32} = \frac{1}{S}\,\nabla_{\theta}\mathcal{L}'$
$\theta \leftarrow \theta - \eta\, g_{fp32}$
Where:
$S$ is the loss-scale factor (e.g., $2^{15}$ , or dynamically adjusted); $\mathcal{L}'$ is the scaled loss used for backprop in FP16.
$\nabla_{\theta}\mathcal{L}'$ are the scaled FP16 gradients; dividing by $S$ restores the true gradient $g_{fp32}$ .
$\theta$ is the FP32 master weight set and $\eta$ the learning rate; updates always land on the master copy.
Costs to Manage: FP16’s narrow range causes gradient underflow and occasional activation overflow, requiring loss scaling and careful debugging of NaN/Inf values; efficiency also depends on hardware with fast FP16 paths.
Login to view more content
May 22, 2025
ML0040 Bias and Variance
Can you explain the bias-variance tradeoff?
Answer
The bias-variance tradeoff decomposes a model’s expected prediction error into three parts: squared bias, variance, and irreducible noise. Bias is the error from overly simplified assumptions: a high-bias model misses the real pattern and underfits. Variance is the error from sensitivity to the particular training sample: a high-variance model wiggles to fit noise and overfits. The tradeoff arises because increasing model complexity typically decreases bias but increases variance, while simplifying does the reverse: total error as a function of complexity is U-shaped, and the goal is the sweet spot that minimizes the sum. Practically, high bias shows as large training error; high variance shows as a large gap between training and validation error, and each has its own remedies (more capacity/features for bias; more data, regularization, or simpler models for variance).
(1) Bias: Error from wrong assumptions: underfitting, poor fit on both train and test data.
(2) Variance: Error from sample sensitivity: overfitting, big train/test gap.
(3) Tradeoff: Complexity trades one for the other; total error $= \text{Bias}^2 + \text{Variance} + \sigma^2$ is U-shaped: minimize the sum.

Figure 1: The classic tradeoff curve: bias² falls and variance rises as complexity grows; total error is their U-shaped sum, and the best model sits at the minimum, not at maximum complexity.
Mathematical Formulation:
$\mathbb{E}\big[(y - \hat{f}(x))^2\big] = \underbrace{\big(\mathbb{E}[\hat{f}(x)] - f(x)\big)^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}\big[(\hat{f}(x) - \mathbb{E}[\hat{f}(x)])^2\big]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{noise}}$
Where:
$f(x)$ is the true relationship and $\hat{f}(x)$ the model’s prediction; expectations are over training sets.
$\text{Bias}^2$ measures how far the average model is from the truth; $\text{Variance}$ how much predictions scatter around that average.
$\sigma^2$ is the irreducible error: noise in the data itself that no model can eliminate.

Figure 2: The tradeoff on real-shaped data: the high-bias model is too rigid to follow the curve (both errors high); the high-variance model chases every noisy point (train error low, test error high); the balanced model tracks the true function and minimizes test error.
Login to view more content
May 21, 2025
ML0039 Distributed Training
What are the two main distributed training approaches for machine learning?
Answer
The two main distributed training approaches are data parallelism and model parallelism, and they answer different bottlenecks. In data parallelism, the dataset is split across many devices, each holding a complete copy of the model; every device computes gradients on its own mini-batch, and after each step the gradients are aggregated and synchronized so all copies stay identical. It scales training when the data is large but the model still fits in one device’s memory. In model parallelism, the model itself is too large for one device, so different parts of the model (e.g., different layers) are placed on different devices, and data flows through them in sequence, with activations passed between devices. It is the enabling technique for today’s very large models, at the cost of more complex communication. Modern large-scale training often combines both (plus pipeline and tensor parallelism) in hybrid strategies.
(1) Data Parallelism: Full model copy per device, different data shards, gradients synchronized after each step: scales with data.
(2) Model Parallelism: The model is split across devices, data flows sequentially through the parts: scales with model size.
(3) Trade-off: Data parallel communicates gradients per step; model parallel communicates activations between stages.

Figure 1: The two approaches: data parallelism (left) replicates the whole model on every device and splits the data, synchronizing gradients after each step; model parallelism (right) splits the model itself across devices and passes activations down the chain.
Mathematical Formulation:
$g = \frac{1}{D} \sum_{d=1}^{D} g_d$
$g_d = \nabla_\theta \mathcal{L}_d(\theta) \quad \text{(data-parallel gradient sync)}$
$h^{(d+1)} = f^{(d+1)}\big(h^{(d)};\, \theta^{(d+1)}\big)$
$\theta^{(d+1)} \text{ lives on device } d+1 \text{ (model parallel)}$
Where:
$D$ is the number of devices; $g_d$ the gradient computed by device $d$ on its data shard; $g$ the synchronized average applied to every replica.
$h^{(d)}$ is the activation handed from device $d$ to device $d+1$ ; $\theta^{(d)}$ the slice of parameters owned by device $d$ .
Login to view more content
May 21, 2025
ML0035 Model Comparison
How to compare different machine learning models?
Answer
Comparing machine learning models rigorously means more than reading one accuracy number off one test run. A sound comparison fixes the evaluation protocol first: choose metrics that match the task and its costs (accuracy or F1/ROC-AUC for classification, RMSE/MAE for regression), then evaluate every model on the same train/validation/test splits (or better, the same cross-validation folds), so differences are attributable to the models, not the data lottery. Because training has randomness (shuffles, weight initialization), each model should be run multiple times with different seeds and its mean and spread reported; when two models look close, a paired statistical test (e.g., a paired t-test or Wilcoxon test over per-fold scores) tells whether the gap is significant or noise. Finally, break ties and inform deployment with secondary criteria: training/inference cost, robustness to perturbations, and interpretability.
(1) Right Metrics: Task- and cost-appropriate metrics (F1/ROC-AUC, RMSE, …), never a single default number.
(2) Controlled Comparison: Identical splits/folds, multiple seeds, cross-validation; statistical tests for close calls.
(3) Secondary Criteria: Latency, memory, robustness, interpretability decide between statistical ties.

Figure 1: Comparing two classifiers on the same test set with ROC curves: the random forest’s curve dominates logistic regression’s at nearly every threshold (AUC 0.98 vs 0.90), a richer comparison than any single-threshold metric.
Mathematical Formulation:
$\bar{s}_m = \frac{1}{K} \sum_{k=1}^{K} s_{m,k}$
$\mathrm{Var}(s_m) = \frac{1}{K-1} \sum_{k=1}^{K} \big(s_{m,k} - \bar{s}_m\big)^2$
$t = \frac{\bar{d}}{\mathrm{std}(d) / \sqrt{K}}$
$d_k = s_{1,k} - s_{2,k} \quad \text{(paired test over folds)}$
Where:
$s_{m,k}$ is the score of model $m$ on fold (or seed) $k$ ; $\bar{s}_m$ and $\mathrm{Var}(s_m)$ summarize central performance and stability.
$d_k$ is the per-fold score difference between two models: pairing removes fold-to-fold variance, so the $t$ -statistic tests whether the mean gap is distinguishable from zero.
Login to view more content
May 11, 2025
DL0012 Zero Padding
Why is zero padding used in deep learning?
Answer
Zero padding adds rows and columns of zeros around the input before a convolution. In CNNs it preserves spatial dimensions, prevents border information from being under-sampled, allows larger kernels and deeper stacks, and gives explicit control over output size. Beyond CNNs, padding standardizes variable-length sequences so NLP and time-series models can process them in batches.
(1) Preserves Spatial Dimensions: Without padding (“valid” convolution), a $k \times k$ kernel shrinks the feature map by $k - 1$ in total per dimension ( $(k-1)/2$ per side) each layer; padding with $p = (k-1)/2$ keeps the size unchanged.
(2) Retains Boundary Information: Padded borders let the kernel center on edge pixels, so corners and boundaries are processed as thoroughly as the interior.
(3) Controls Output Size: Padding decouples output dimensions from kernel size, enabling deeper networks and predictable feature-map shapes.
Mathematical Formulation:
$n_{out} = \left\lfloor \frac{n_{in} + 2p - k}{s} \right\rfloor + 1$
Where:
$n_{out}$ and $n_{in}$ are the output and input spatial sizes.
$p$ is the padding width added to each side, $k$ is the kernel size, and $s$ is the stride.
“Same” padding for stride 1 uses $p = (k-1)/2$ , giving $n_{out} = n_{in}$ .

Figure 1: Padding a $4 \times 4$ input to $6 \times 6$ lets a $3 \times 3$ kernel output the same $4 \times 4$ size instead of shrinking to $2 \times 2$ .
Beyond CNNs: In NLP and time-series tasks, zero padding extends shorter sequences to a uniform length for efficient batching. Because padded positions carry no information, models combine padding with attention masks so Transformer self-attention ignores those positions entirely.

Figure 2: Valid shrinks the map, same preserves it, and sequence padding plus a mask enables batched NLP inputs.
Login to view more content
May 11, 2025