Interview for Machine Learning

Author: admin

DL0006 Layer Freeze in TL
What are the common strategies for layer freezing in transfer learning?
Answer
Layer freezing controls which pre-trained weights are updated during fine-tuning. Frozen layers keep their learned representations intact, while trainable layers adapt to the target task. The right choice balances leveraging general features against adapting higher-level representations, and depends mainly on target dataset size and task similarity.
(1) Freeze All but the Output Layer(s): Train only the final classification/regression layers; a good starting point for similar tasks and small datasets.
(2) Freeze Early Layers: Early layers capture general features (edges, textures), so train only the later, task-specific layers; effective for moderately similar tasks.
(3) Fine-Tune All Layers with a Low Learning Rate: Adapt all weights slowly; use with caution on small datasets to avoid catastrophic forgetting.
(4) Gradual Unfreezing: Start with frozen layers and progressively unfreeze during training, avoiding large early updates that can destroy learned features.
(5) Backbone-Freeze, Then Low-LR Fine-Tune: Freeze the backbone until the new head converges, then unfreeze it and continue with a reduced learning rate.

Figure 1: The four common strategies differ in how many blocks stay frozen; freeze more when data is scarce and tasks are similar.
Mathematical Formulation:
$\theta = \theta_{frozen} \cup \theta_{trainable}$
$\Delta\theta_{frozen} = 0$
$\theta_{trainable} \leftarrow \theta_{trainable} - \eta\,\nabla_{\theta_{trainable}}\mathcal{L}$
Where:
$\theta_{frozen}$ is the parameter subset kept fixed at its pre-trained values; only $\theta_{trainable}$ receives gradient updates.
$\eta$ is the learning rate; strategies (3) and (5) use a reduced $\eta$ (e.g., 0.1×) to protect pre-trained features.

Figure 2: Gradual unfreezing starts with the head and unfreezes deeper blocks step by step with a reduced learning rate.
Login to view more content
April 30, 2025
DL0005 Transfer Learning
Why use transfer learning in deep learning instead of training from scratch?
Answer
Transfer learning reuses knowledge from a pre-trained model to improve performance, reduce training time and data requirements, and lower computational cost on a new but related task. Instead of learning low-level features from random weights, the model starts from representations that already capture generalizable patterns such as edges, textures, and shapes, so far less target data is needed to reach strong accuracy.
(1) Leverages Existing Knowledge & Reduced Data Requirements: Pre-trained weights encode useful representations learned from large datasets, so good performance is possible with significantly less task-specific data.
(2) Faster Convergence & Training Time: Starting from pre-trained weights is a much better initialization than random weights, leading to faster convergence and often better local optima.
(3) Improved Performance on Limited-Data Tasks: When data is scarce, transfer learning typically yields higher accuracy and better generalization than training from scratch.

Figure 1: The backbone’s weights are copied from pre-training; only the new head (and optionally upper layers) must be learned from limited target data.
Mathematical Formulation:
$\theta^* = \arg\min_{\theta}\ \mathcal{L}_{target}(\theta;\ \theta_{init} = \theta_{pretrained})$
$\eta_l = \eta_{base} \cdot \gamma^{\,L-l}$
Where:
$\theta_{pretrained}$ is the weight set learned on the source task; fine-tuning minimizes the target loss starting from this initialization.
$\eta_l$ is the learning rate of layer $l$ out of $L$ ; a decay factor $\gamma$ below 1 gives earlier (more general) layers smaller updates than later (task-specific) layers.

Figure 2: Transfer learning dominates in the low-data regime; the advantage shrinks as the target dataset grows.
Faster Convergence: Because pre-trained weights are already a strong initialization, the model reaches its accuracy plateau in far fewer epochs than training from random weights.

Figure 3: Transfer learning converges faster and plateaus higher; training from scratch improves slowly over many epochs.
Login to view more content
April 30, 2025
DL0004 Small Kernels
What are the key advantages of using small convolutional kernels, such as 3×3, over utilizing a few larger kernels in deep learning architectures?
Answer
Using small convolutional kernels instead of a few larger kernels lets a network reach the same receptive field with fewer parameters, more depth, and more non-linearity. A stack of $3\times3$ convolutions matches the spatial coverage of a single large kernel while inserting an activation between each layer, which increases the network’s representational power. This design is the foundation of architectures such as VGG.
(1) Deeper Networks & More Non-Linearity: Stacking multiple $3\times3$ layers (e.g., three of them) creates a deeper network with more non-linear activation functions than a single large kernel.
(2) Reduced Parameters: Multiple small kernels achieve the same receptive field as a larger one with fewer parameters: two stacked $3\times3$ layers use $18\cdot C_{in}\cdot C_{out}$ weights versus $25\cdot C_{in}\cdot C_{out}$ for one $5\times5$ layer.
(3) Computational Efficiency: Fewer parameters generally mean lower FLOPs during training and inference.
(4) Gradual Receptive Field Expansion: Successive $3\times3$ convolutions progressively build a larger receptive field while preserving local detail capture, ideal for textures and edges.

Figure 1: Two stacked $3\times3$ convolutions see the same $5\times5$ input region as one $5\times5$ convolution, but with an extra non-linearity in between.
Kernel Composition: Two stacked $3\times3$ kernels compose into a single equivalent $5\times5$ kernel (their full discrete convolution), which is exactly why the receptive fields match.

Figure 2: Two $3\times3$ kernels compose into one equivalent $5\times5$ kernel; stacking keeps them separate: 18 weights plus an extra non-linearity instead of a 25-weight merge.
Mathematical Formulation:
$P_k = k^2 \cdot C_{in} \cdot C_{out}$
$RF_l = 1 + l\cdot(k-1)$
$2 \times 3^2 = 18 \text{ vs } 5^2 = 25$
$3 \times 3^2 = 27 \text{ vs } 7^2 = 49$
Where:
$P_k$ is the parameter count of one $k\times k$ convolution (ignoring biases).
$C_{in}$ and $C_{out}$ are the input and output channel counts.
$RF_l$ is the receptive field after $l$ stacked layers; two $3\times3$ layers give $RF=5$ , three give $RF=7$ .

Figure 3: For an equal receptive field, stacked $3\times3$ convolutions cut parameters by 28% (vs $5\times5$ ) and 45% (vs $7\times7$ ).
Login to view more content
April 29, 2025
DL0003 1×1 Convolution
What are the benefits of using 1×1 convolutional layers in deep learning architectures?
Answer
A 1×1 convolution (pointwise convolution) operates on the channel dimension at each spatial location, enabling dimensionality control, cross-channel feature fusion, and non-linear mixing, all with minimal computational cost. In architectures like ResNet’s bottleneck block, a 1×1 conv first reduces channels (e.g., 256→64), a 3×3 conv processes the reduced representation, and another 1×1 conv expands back (64→256), restoring capacity while keeping compute manageable. This design allows deeper networks without prohibitive parameter growth.
(1) Dimensionality Control: 1×1 convolutions can reduce or expand the number of feature maps, trading off representational capacity and computational cost as needed.
(2) Depth With Controlled Cost: By reducing channel dimensionality before expensive spatial convolutions, 1×1 convs enable deeper architectures without the quadratic growth in channel count that a plain $3 \times 3$ block incurs.
(3) Cross-Channel Feature Fusion: Each output pixel is a learned linear combination of all input channels at that location, enabling rich channel-wise interactions.
(4) Non-Linear Mixing: When followed by activations (ReLU, etc.), 1×1 convs introduce non-linear channel mixing that enhances model expressiveness.

Figure 1: A 1×1 convolution performs a per-pixel weighted sum across all input channels: each output channel is a learned linear combination of every input channel at the same spatial position, preserving $H\times W$ .
Mathematical Formulation:
$y_{i,j,c_{out}} = \sum_{c_{in}=1}^{C_{in}} w_{c_{in},c_{out}} \cdot x_{i,j,c_{in}} + b_{c_{out}}$
$\text{FLOPs}_{1\times1} = H \times W \times C_{in} \times C_{out}$
Where:
$x_{i,j,c_{in}}$ is the input activation at spatial position $(i,j)$ in channel $c_{in}$ ; $y_{i,j,c_{out}}$ is the output.
$w_{c_{in},c_{out}}$ is the 1×1 kernel weight connecting input channel $c_{in}$ to output channel $c_{out}$ ; there is no spatial kernel dimension.
$C_{in}$ and $C_{out}$ are the input and output channel counts; $b_{c_{out}}$ is the bias for each output channel.
$H$ and $W$ are the spatial height and width of the feature map; the operation is equivalent to a fully-connected layer applied per-pixel across channels.

Figure 2: ResNet bottleneck block: a 1×1 conv reduces channels (256→64), a 3×3 conv processes the compressed representation, and another 1×1 conv expands back (64→256), with a residual skip connection.
Login to view more content
April 29, 2025
DL0002 All Ones Init
What are the potential consequences of initializing all weights to one in a deep learning model?
Answer
Initializing all weights to one (or any constant non-zero value) creates a symmetry problem: every neuron in the same layer receives identical gradients, so they learn the same features and cannot develop diverse representations. This limits the network to a single effective feature per layer, severely reducing its representational capacity. Training becomes slow or fails to converge because the optimizer cannot differentiate between neurons. Additionally, constant positive weights can push activations into saturated regions of sigmoid or tanh, worsening vanishing gradients.
(1) Symmetry Problem: All neurons in a layer receive the same input and the same gradient, so they evolve identically throughout training.
(2) Limited Representational Capacity: The network behaves as if each layer has only one neuron, making it impossible to capture complex, varied patterns in the data.
(3) Slow or Failed Convergence: Because neurons cannot differentiate, the loss landscape lacks the diversity needed for gradient descent to find useful minima.
(4) Activation Saturation: Uniform positive weights shift pre-activation values into the flat tails of sigmoid and tanh, producing vanishing gradients that stall learning.

Figure 1: The symmetry problem: with all-ones weights every neuron computes the same function and receives the same gradient, while random initialization lets each neuron learn a distinct feature.
Mathematical Formulation:
$\frac{\partial \mathcal{L}}{\partial w_{ij}^{(l)}} = \delta_i^{(l)} \cdot a_j^{(l-1)}$
$w_{ij}^{(l)} = 1 \quad \forall\, i,j \implies \delta_i^{(l)} = \delta_j^{(l)} \quad \forall\, i,j$
Where:
$w_{ij}^{(l)}$ is the weight from neuron $j$ in layer $l-1$ to neuron $i$ in layer $l$ .
$\delta_i^{(l)}$ is the error signal (delta) for neuron $i$ in layer $l$ .
$a_j^{(l-1)}$ is the activation of neuron $j$ in the previous layer.
$\mathcal{L}$ is the loss function; when all weights are equal, all delta signals are identical, so every weight updates by the same amount.

Figure 2: Training loss comparison: ones initialization converges slowly and plateaus at a high loss, while random initialization descends rapidly to a low minimum.
Login to view more content
April 28, 2025
DL0001 Residual Connection
Why are residual connections important in deep neural networks?
Answer
A residual connection (skip connection) adds a block input to a learned residual transformation, so the block learns an update rather than an entirely new mapping. For a shape-preserving block, the local Jacobian becomes $I+J_F$ , which supplies a direct identity component and creates shorter paths through the computational graph. This usually improves gradient propagation and optimization, but it does not guarantee nonzero or constant gradients: products of residual-block Jacobians can still shrink, grow, or cancel. Residual connections also address the degradation problem, in which adding layers to a plain network can increase training error because the deeper model is difficult to optimize even though an identity extension exists. These properties enabled effective training of ResNet architectures with hundreds of layers.
(1) Shorter Gradient Paths: An identity shortcut changes the local block Jacobian from $J_F$ to $I+J_F$ . The identity component gives backpropagation additional routes, although the product across many blocks can still vanish or explode.
(2) Identity Mapping Fallback: When an additional shape-preserving block is not useful, its residual branch $F_l(x_l)$ can approach zero, making $x_{l+1}\approx x_l$ easier to represent than in a plain nonlinear stack.
(3) Easier Deep Optimization: Residual parameterization helps deeper models avoid the degradation problem, where added layers increase training error because optimization fails to recover a useful identity extension.

Figure 1: A shape-preserving block uses an identity shortcut, while a block that changes resolution or channel width uses a learned projection $S_l$ so the two paths have compatible shapes.
Mathematical Formulation:
$x_{l+1}=F_l(x_l;W_l)+x_l$
$\frac{\partial x_{l+1}}{\partial x_l}=J_{F_l}+I$
$x_{l+1}=F_l(x_l;W_l)+S_lx_l$
Where:
$x_l$ is the input to residual block $l$ , and $x_{l+1}$ is its output.
$F_l(x_l;W_l)$ is the learned residual branch parameterized by weights $W_l$ ; it learns an update to the shortcut representation.
$J_{F_l}=\partial F_l/\partial x_l$ is the residual-branch Jacobian, and $I$ is the identity operator with the same feature dimension as $x_l$ .
$S_l$ is an identity operator when input and output shapes match; otherwise it can be a learned projection, such as a strided $1\times1$ convolution, that aligns spatial and channel dimensions.
Across $L$ residual blocks, backpropagation contains products of factors $I+J_{F_l}$ . These factors often improve conditioning relative to plain Jacobian products, but they do not impose a nonzero lower bound on gradient magnitude.

Figure 2: A plain stack multiplies full layer Jacobians, whereas a residual stack multiplies factors $I+J_{F_l}$ . The identity components provide additional gradient routes but do not guarantee stable magnitude.
Login to view more content
April 23, 2025
ML0031 Linear Regression
What are the advantages and disadvantages of linear regression?
Answer
Linear regression models the target as a weighted sum of the input features plus an intercept, fitting the weights by minimizing squared error. Its advantages make it the default baseline: it is simple and interpretable: each coefficient directly states the strength and direction of a feature’s relationship with the target; it is computationally cheap, with a closed-form solution (normal equations) and fast training even on large data; and it works well when the true relationship is approximately linear. Its disadvantages stem from the same simplicity: it assumes linearity, so it underfits genuinely non-linear relationships; it is sensitive to outliers because squared errors let extreme points dominate the fit; multicollinearity between correlated features makes coefficient estimates unstable and hard to interpret; and the model cannot capture interactions or complexity unless you explicitly engineer such features. Simple linear regression uses one feature; multiple linear regression extends the same idea to many.
(1) Interpretable & Fast: Coefficients read directly as feature effects; closed-form or cheap iterative fitting.
(2) Linearity Assumption: Underfits non-linear patterns: the model class is a hyperplane.
(3) Fragilities: Outlier-sensitive (squared loss), unstable under multicollinearity, no built-in interactions or nonlinearity.

Figure 1: Least-squares fit: the line minimizes the sum of squared vertical residuals (orange segments). Note how the one distant outlier pulls the fitted line toward itself, the sensitivity that comes with squaring errors.
Mathematical Formulation:
$h_\theta(x) = \theta_0 + \sum_{j=1}^{p} \theta_j x_j = \theta^{\top} x$
$\hat{\theta} = \arg\min_{\theta} \sum_{i=1}^{n} \big(y_i - h_\theta(x_i)\big)^2 = (X^{\top}X)^{-1} X^{\top} y$
Where:
$h_\theta(x)$ is the predicted value for feature vector $x$ .
$\theta_0$ is the intercept (bias); $\theta_j$ the weight of feature $x_j$ ; $p$ the number of features.
$X$ is the $n \times (p+1)$ design matrix and $y$ the target vector; the closed form exists when $X^{\top}X$ is invertible (no perfect multicollinearity).
Login to view more content
April 18, 2025
ML0030 Sigmoid
What are the advantages and disadvantages of using a sigmoid activation function?
Answer
The sigmoid activation squashes any real input into the range $(0, 1)$ with a smooth S-curve. Its advantages: it is smooth and differentiable everywhere, and its bounded output has a natural probability interpretation, which keeps it the standard output activation for binary classification and for independent per-label probabilities in multi-label tasks (and for gates in LSTM/GRU cells, which need values in $(0,1)$ ). Its disadvantages drove it out of hidden layers: in the tails the function saturates and its derivative (at most 0.25, near zero for most inputs) makes stacked sigmoid layers a prime cause of the vanishing gradient problem; its outputs are not zero-centered, so downstream weight updates are biased in one direction and convergence slows; and the exponential computation is more expensive than ReLU’s simple threshold.
(1) Probability Output: Range $(0,1)$ , ideal for binary classification output layers and gating mechanisms.
(2) Vanishing Gradient: Derivative peaks at 0.25 and saturates in the tails; deep stacks lose gradient fast.
(3) Not Zero-Centered: Always-positive outputs bias weight updates and slow convergence; also costlier to compute than ReLU.

Figure 1: Sigmoid and its derivative $\sigma(x)(1-\sigma(x))$ : a useful probability-shaped output, but the derivative only reaches 0.25 at best and is near zero in both tails: the mechanism behind vanishing gradients.
Mathematical Formulation:
$\sigma(x) = \frac{1}{1 + e^{-x}}$
$\sigma'(x) = \sigma(x)\,\big(1 - \sigma(x)\big) \leq 0.25$
Where:
$x$ is the neuron’s pre-activation (weighted sum plus bias).
$\sigma'(x)$ is the derivative, computed in one line from the output itself, maximal (0.25) at $x = 0$ , approaching 0 as $|x|$ grows.
Login to view more content
April 18, 2025
ML0029 Tanh
What are the advantages and disadvantages of using the tanh activation function?
Answer
The tanh (hyperbolic tangent) activation squashes its input into the range $(-1, 1)$ with a smooth S-shape. Its headline advantage over sigmoid is that its output is zero-centered: activations average near zero, so the gradients flowing into the next layer do not systematically push all weights in one direction, which makes optimization easier and convergence faster. It is also smooth and infinitely differentiable, and its maximum derivative of 1 (versus sigmoid’s 0.25) gives stronger gradients near the active region. The disadvantages are the flip side of saturation: for large positive or negative inputs the function flattens and its derivative approaches zero, so deep stacks of tanh units suffer the vanishing gradient problem; and like sigmoid it requires exponentials, making it computationally more expensive than ReLU. Because its output is not in $(0, 1)$ , it is also unsuitable as a probability output; its modern niche is mainly inside RNN/LSTM cell dynamics rather than feedforward hidden layers.
(1) Zero-Centered: Output range $(-1, 1)$ centers activations, easing optimization versus sigmoid’s all-positive outputs.
(2) Stronger Gradient: Maximum derivative 1 at the origin (4× sigmoid’s), but it still saturates in the tails.
(3) Costs: Vanishing gradients in deep stacks, exponential computation, and no probability interpretation of the output.

Figure 1: Tanh and its derivative $1 - \tanh^2(x)$ : the zero-centered S-curve has a healthy gradient (peak 1) only in the narrow central region; in both tails the derivative collapses toward zero.
Mathematical Formulation:
$\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} = 2\,\sigma(2x) - 1$
$\frac{d}{dx}\tanh(x) = 1 - \tanh^2(x) \in (0, 1]$
Where:
$x$ is the neuron’s pre-activation; $\sigma(\cdot)$ is the logistic sigmoid, showing tanh is just a rescaled sigmoid.
$1 - \tanh^2(x)$ is the derivative: maximal (1) at $x = 0$ , decaying toward 0 as $|x|$ grows, the saturation that drives vanishing gradients.
Login to view more content
April 17, 2025
ML0028 Softmax
What is the Softmax activation function, and what is its purpose?
Answer
Softmax is the activation used in the output layer for multi-class classification: it converts a vector of raw scores (logits) into a normalized probability distribution over the classes. Each output is the exponentiated logit divided by the sum of exponentiated logits, so every output lies in $(0, 1)$ and all outputs sum to exactly 1, a genuine probability distribution that can be read as the model’s confidence in each class. Softmax assumes classes are mutually exclusive (one true class per sample); for multi-label problems where an input can belong to several classes at once, per-class sigmoid outputs are used instead, since each class needs an independent probability. Paired with cross-entropy loss, softmax gives the clean gradient $p - y$ , which is why the combination is the standard for multi-class training.
(1) Purpose: Turn arbitrary logits into a probability distribution: outputs in $(0,1)$ , summing to 1.
(2) Amplification: Exponentiation sharpens differences: the largest logit dominates the distribution.
(3) Scope: Multi-class (mutually exclusive) problems with cross-entropy loss; multi-label problems use independent sigmoids instead.

Figure 1: Softmax in action: raw logits (top) of arbitrary scale and sign are exponentiated and normalized into a probability distribution (bottom) that sums to 1; the relative order is preserved but differences are amplified.
Mathematical Formulation:
$\mathrm{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$
$\sum_{i=1}^{K} \mathrm{Softmax}(z_i) = 1$
$\mathrm{Softmax}(z_i / T) = \frac{e^{z_i / T}}{\sum_{j=1}^{K} e^{z_j / T}} \quad \text{(with temperature } T \text{)}$
Where:
$z_i$ is the raw score (logit) for class $i$ , and $K$ the number of classes.
The exponential makes every term positive; dividing by the total normalizes the vector to sum 1.
$T$ is an optional temperature: a value below 1 sharpens the distribution, $T > 1$ softens it (used in distillation and sampling).

Figure 2: Temperature controls sharpness: low temperature pushes probability mass onto the top class (hard distribution), high temperature flattens toward uniform: the same logits, three very different distributions.
Login to view more content
April 12, 2025