Interview for Machine Learning

Category: Medium

DL0060 Depthwise Separable Convolution
What is depthwise separable convolution, and where is it commonly used?
Answer
Depthwise separable convolution factorizes a standard convolution into a depthwise spatial operation and a pointwise channel-mixing operation. The depthwise stage applies one $k\times k$ filter per input channel, so it learns spatial patterns independently; the $1\times1$ pointwise stage combines those channel responses into new output channels. This factorization can reduce multiply-accumulates and parameters by almost the kernel area when channel counts are large. It is common in MobileNet, Xception, EfficientNet-style mobile blocks, and edge models where latency, power, and model size matter.
(1) Depthwise Stage: With depth multiplier one, $C_{in}$ independent $k\times k$ filters produce $C_{in}$ spatially filtered maps and do not mix channels.
(2) Pointwise Stage: A $1\times1$ convolution applies a learned $C_{in}\times C_{out}$ channel transformation independently at every spatial position.
(3) Real Hardware: Lower arithmetic cost does not guarantee proportional speedup because memory traffic, kernel launch overhead, vectorization, and accelerator support can dominate.

Figure 1: Tensor-level view of spatial filtering per channel followed by 1×1 channel mixing, with parameters and MACs.
Mathematical Formulation:
$\mathrm{MACs}_{std}=HWk^2C_{in}C_{out}$
$\mathrm{MACs}_{sep}=HW\left(k^2C_{in}+C_{in}C_{out}\right)$
$\frac{\mathrm{MACs}_{sep}}{\mathrm{MACs}_{std}}=\frac{1}{C_{out}}+\frac{1}{k^2}$
Where:
$\mathrm{MACs}_{std}$ and $\mathrm{MACs}_{sep}$ count multiply-accumulates for standard and depthwise-separable convolution.
$H$ and $W$ are output height and width, and $k$ is the square spatial-kernel width.
$C_{in}$ and $C_{out}$ are input and output channel counts; the formula assumes depth multiplier $1$ .
For $k=3$ and large $C_{out}$ , the ratio approaches $1/9$ .

Figure 2: Architecture decision based on efficiency target, channel interaction, hardware kernels, and measured latency.
Login to view more content
July 20, 2026
DL0059 Transposed Convolution
What is a transposed convolution (sometimes called deconvolution), and when is it used?
Answer
A transposed convolution is a learnable linear operator whose matrix is the transpose of the matrix representing a corresponding forward convolution. Operationally, each input value scatters a scaled kernel into an output grid, with stride controlling the spacing and overlapping contributions summed. It can increase spatial resolution, but it is not a mathematical inverse and does not recover information destroyed by downsampling. It is commonly used in semantic-segmentation decoders, generative models, learned reconstruction, and other architectures that map low-resolution features to higher-resolution outputs.
(1) Learned Upsampling: Unlike fixed nearest-neighbor or bilinear interpolation, the layer learns how feature values contribute to surrounding output positions.
(2) Output Geometry: Kernel size, stride, padding, dilation, and output padding jointly determine the output shape; output padding resolves shape ambiguity but does not add zero-valued borders.
(3) Artifact Risk: Uneven overlap occurs when the kernel size is not divisible by stride and can produce checkerboard artifacts, especially in image generators.

Figure 1: A stride-2 transposed convolution decomposed into sparse expansion and learned filtering, including output-shape arithmetic.
Mathematical Formulation:
$H_{out}=(H_{in}-1)s-2p+d(k-1)+p_{out}+1$
Where:
$H_{in}$ and $H_{out}$ are the input and output sizes along one spatial axis.
$s$ is stride, $p$ is padding, $d$ is dilation, and $k$ is kernel size.
$p_{out}$ is output padding used to choose among otherwise ambiguous valid output sizes; it does not append learned pixels.
The formula applies independently to width and, for 3D layers, depth by replacing $H$ with the corresponding axis size.

Figure 2: Selection guide based on learned reconstruction, artifact sensitivity, pooling indices, and deployment constraints.
Login to view more content
July 20, 2026
DL0058 Feature Pyramid Network
What is a Feature Pyramid Network (FPN) in the context of CNNs, and why is it useful for dense prediction?
Answer
A feature pyramid network (FPN) builds a hierarchy of semantically strong feature maps at multiple spatial resolutions. A bottom-up backbone produces progressively smaller maps with richer semantics; a top-down pathway upsamples deep features and merges them with same-resolution lateral projections from earlier stages. The resulting $P_2\text{--}P_5$ maps combine localization detail with high-level context. Detection, instance segmentation, and keypoint heads can then select a pyramid level appropriate to each object or region scale.
(1) Bottom-up Hierarchy: Backbone stages $C_2\text{--}C_5$ reduce spatial resolution while increasing receptive field and semantic abstraction.
(2) Top-down Fusion: Nearest-neighbor upsampling and 1×1 lateral projections align channel widths before element-wise addition; a 3×3 convolution commonly smooths each merged map.
(3) Scale Assignment: Small objects use high-resolution levels such as $P_2$ or $P_3$ , while large objects use coarser levels such as $P_4$ or $P_5$ .

Figure 1: FPN construction showing tensor resolution, lateral 1×1 projection, top-down upsampling, fusion, and 3×3 smoothing.
Mathematical Formulation:
$P_l=\mathrm{Conv}_{3\times3}\!\left(\mathrm{Conv}_{1\times1}(C_l)+\mathrm{Up}_2(P_{l+1})\right)$
$k=\left\lfloor k_0+\log_2\!\left(\frac{\sqrt{wh}}{224}\right)\right\rfloor$
Where:
$C_l$ is the bottom-up backbone feature and $P_l$ is the fused pyramid output at level $l$ .
$\mathrm{Conv}_{1\times1}$ aligns channel width, $\mathrm{Up}_2$ doubles spatial resolution, and $\mathrm{Conv}_{3\times3}$ smooths the merged feature.
$k$ is the assigned pyramid level, $k_0$ is the reference level, and $w,h$ are the region width and height.
$224$ is the canonical reference scale and $\lfloor\cdot\rfloor$ maps the continuous log-scale value to a discrete level.

Figure 2: How object size directs prediction heads toward fine, medium, or coarse pyramid levels.
Login to view more content
July 20, 2026
DL0057 2D VS 3D Convolutions
What is the difference between 2D and 3D convolutions, and when would you use each?
Answer
A 2D convolution slides a kernel across height and width, while aggregating all input channels at each spatial location. A 3D convolution slides across depth or time as well as height and width, so it learns joint spatiotemporal or volumetric features. Use 2D convolution for ordinary images, per-frame video processing, or slice-wise analysis when cross-slice context is unnecessary. Use 3D convolution for videos, CT/MRI volumes, or occupancy grids when local relationships along the third axis carry essential information and the additional memory and compute are affordable.
(1) Kernel Geometry: A 2D kernel has spatial extent $k_h\times k_w$ ; a 3D kernel adds $k_d$ and jointly traverses depth or time.
(2) Data Semantics: The third axis should represent an ordered neighborhood such as adjacent frames or slices, not an unordered feature channel.
(3) Trade-off: 3D convolution captures motion or volumetric continuity directly but costs roughly $k_d$ times more than a comparable 2D layer and stores larger activation volumes.

Figure 1: Kernel geometry and output formation for 2D spatial convolution and 3D spatiotemporal convolution.
Mathematical Formulation:
$Y_{o,d,h,w}=\sum_{c=1}^{C_{in}}\sum_{\delta_d,\delta_h,\delta_w}K_{o,c,\delta_d,\delta_h,\delta_w}X_{c,d+\delta_d,h+\delta_h,w+\delta_w}$
Where:
$X$ and $Y$ are the input and output tensors, while $K$ is the learned 3D convolution kernel.
$o$ indexes output channels and $c\in\{1,\ldots,C_{in}\}$ indexes input channels.
$d,h,w$ index output depth/time, height, and width; $\delta_d,\delta_h,\delta_w$ range over the kernel support along those axes.
For a 2D convolution, $d$ and $\delta_d$ are removed, leaving only spatial indices $h,w$ ; stride, padding, and dilation modify each active index mapping.

Figure 2: Selection guide based on third-axis semantics, required context, resource budget, and deployment constraints.
Login to view more content
July 20, 2026
DL0055 Vision Transformer
Explain the Vision Transformer (ViT). How does it convert an image into a class prediction?
Answer
A Vision Transformer converts an image into a sequence of fixed-size patch tokens and processes them with a Transformer encoder. A learned class token is prepended, positional embeddings preserve patch order, and global self-attention lets every patch exchange information with every other patch. After the encoder stack, the class-token representation is passed to a prediction head. Compared with a CNN, a ViT has weaker built-in locality and translation bias, but it can model long-range interactions directly and scales effectively with data and compute.
(1) Tokenization: An image of shape $H\times W\times C$ is divided into $N=(H/P)(W/P)$ non-overlapping $P\times P$ patches; each flattened patch is projected to width $D$ .
(2) Global Context: Multi-head self-attention mixes information across all patch positions, while residual connections and MLP sublayers refine each token.
(3) Classification: A learned [CLS] token aggregates evidence across the encoder stack; its final state is normalized and mapped to class logits.

Figure 1: ViT architecture with tensor shapes, patch tokenization, the Transformer encoder stack, and the classification path.
Mathematical Formulation:
$z_0=[x_{\mathrm{cls}};x_p^1E;x_p^2E;\ldots;x_p^NE]+E_{\mathrm{pos}}$
$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^T}{\sqrt{d_h}}\right)V$
Where:
$z_0$ is the initial token sequence supplied to the Transformer encoder.
$x_{\mathrm{cls}}$ is the learned class token, and $x_p^i$ is flattened image patch $i$ .
$E\in\mathbb{R}^{P^2C\times D}$ projects each $P\times P\times C$ patch to width $D$ , while $E_{\mathrm{pos}}$ supplies positional embeddings.
$N=(H/P)(W/P)$ is the patch count for an image of height $H$ , width $W$ , and channel count $C$ .
$Q$ , $K$ , and $V$ are query, key, and value matrices; $d_h$ is the per-head query/key width.

Figure 2: ViT inference flow from image validation and patch embedding to encoder processing and class prediction.
Login to view more content
July 20, 2026
DL0054 Deformable Attention
What is Deformable Attention and how does it reduce computational complexity for object detection tasks?
Answer
Deformable Attention is a sparse attention mechanism that learns dynamic sampling locations instead of attending to all spatial positions uniformly. It uses learned 2D offsets from reference points to sample only the most relevant features, reducing complexity from $O(N^2)$ to $O(NK)$ where K is a small constant (typically 4). This makes it ideal for high-resolution feature maps in object detection where full attention is computationally prohibitive — for a 1024×1024 feature map ( $N \approx 10^6$ positions), standard attention requires ~1T operations per head while deformable attention needs only ~4M.
(1) Sparse Sampling: Instead of computing attention over all $N \times N$ positions, deformable attention samples only K reference points per query, typically K=4 or 8, reducing the key-value set from N to K.
(2) Learned Offsets: The model predicts $\Delta p_{mk}$ offsets from each reference point $p_k$ using a lightweight linear layer on query features, requiring only $O(NC)$ additional computation where C is channel dimension.
(3) Bilinear Interpolation: When offsets point to non-integer locations, bilinear interpolation computes feature values from the 4 nearest pixels, enabling sub-pixel precision sampling without modifying the feature map.

Figure 1: Deformable attention learns K=4 sampling offsets per query point instead of dense N×N attention
Mathematical Formulation:
$y(p) = \sum_{m=1}^{M} W_m \left[ \sum_{k=1}^{K} A_{mk} \cdot W_m' x(p + p_k + \Delta p_{mk}) \right]$
Where:
$p$ is the reference position (query location on the feature map)
$M$ is the number of attention heads
$K$ is the number of sampled keys per head (typically 4)
$p_k$ are fixed reference offsets (uniformly initialized)
$\Delta p_{mk}$ are learned deformable offsets (2D, predicted per head per key)
$A_{mk}$ is the attention weight (normalized, not from softmax over all positions)
$W_m, W_m'$ are projection matrices for each head

Figure 2: Complexity comparison shows O(NK) grows linearly while O(N²) becomes prohibitive for large feature maps
The offsets $\Delta p_{mk}$ are predicted by a linear projection from query features: $\Delta p_{mk} = W_\text{offset} \cdot q_m(p)$ , where $W_\text{offset} \in \mathbb{R}^{C \times 2K}$ . The attention weights $A_{mk}$ are computed via a separate softmax over only K elements, not the full N positions. In Deformable DETR, multi-scale deformable attention extends this to sample across multiple feature map resolutions simultaneously, enabling the model to capture both small and large objects efficiently.
Login to view more content
July 13, 2026
DL0053 Gated Attention
What is Gated Attention and how does it improve transformer architectures over standard scaled dot-product attention?
Answer
Gated Attention (arXiv:2505.06708) applies a head-specific sigmoid gate after Scaled Dot-Product Attention (SDPA) to dynamically modulate attention output. Unlike standard attention where all heads contribute equally, gated attention introduces query-dependent sparse gating that suppresses irrelevant heads and activates only salient ones. This mitigates the attention sink problem where standard transformers concentrate disproportionate attention on the first few tokens, and enhances long-context extrapolation by maintaining diverse attention patterns across sequence lengths.
(1) Post-SDPA Gating: The gate is applied after SDPA computation, not before — each attention head $h_i$ is multiplied by a sigmoid gate $g_i = \sigma(W_g \cdot q_i)$ where $W_g$ is a head-specific projection.
(2) Sparsity Induction: The sigmoid gate produces values in $[0, 1]$ , and empirical measurements show mean gate activation of ~0.116, meaning most heads are heavily suppressed — introducing beneficial sparsity without hard pruning.
(3) Attention Sink Mitigation: Standard attention allocates ~46.7% of attention mass to the first token; gated attention reduces this to ~4.8%, distributing attention more uniformly across tokens.

Figure 1: Gated attention applies a head-specific sigmoid gate after SDPA to modulate attention output before the residual connection
Mathematical Formulation:
$\text{GatedAttn}(Q, K, V) = \text{Concat}(g_1 \odot h_1, \ldots, g_H \odot h_H) W_O$
$g_i = \sigma(W_g^{(i)} \cdot q_i + b_g^{(i)})$
Where:
$h_i = \text{SDPA}(q_i, k_i, v_i)$ is the output of the $i$ -th attention head
$g_i \in [0, 1]$ is the head-specific gate score
$W_g^{(i)} \in \mathbb{R}^{d_k \times 1}$ is a learned projection from query to scalar gate
$\sigma$ is the sigmoid function
$\odot$ denotes element-wise multiplication
$W_O$ is the standard output projection

Figure 2: Gate activation distribution across 8 attention heads shows heavy suppression (mean ~0.116) with sparse high-activation regions
The gate projection $W_g^{(i)}$ adds only $O(d_k)$ parameters per head — a negligible overhead of ~0.1% of total model parameters — yet significantly improves long-context performance. On the RULER benchmark at 128K context length, gated attention improves needle-in-haystack retrieval accuracy from ~72% to ~94% compared to standard attention.
Login to view more content
July 13, 2026
DL0052 Rotary Positional Embedding
What is Rotary Positional Embedding (RoPE)?
Answer
Rotary Positional Embedding (RoPE) encodes position by rotating each pair of query/key dimensions by an angle proportional to the token’s position, instead of adding a position vector. Because a rotation at position $m$ composed with one at position $n$ depends only on the difference, the attention dot product between tokens automatically reflects their relative distance, with no learned parameters and good length generalization. RoPE is the default in GPT-NeoX, LLaMA, PaLM, Qwen, and most modern LLMs.
(1) Geometric Encoding: Each 2D slice of a query or key vector is rotated by $m\theta_i$ , where $m$ is the token position and $\theta_i$ the per-dimension frequency.
(2) Relative From Absolute: The dot product of two rotated vectors depends on $(m - n)\theta_i$ only: absolute rotations yield relative-position attention.
(3) Parameter-Free and Efficient: No embedding table, no attention bias, just a fixed trigonometric transform applied to Q and K.
Mathematical Formulation:
$f(x, m) = \begin{pmatrix} \cos(m\theta) & -\sin(m\theta) \\ \sin(m\theta) & \cos(m\theta) \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}$
$\theta_i = 10000^{-2i/d}$
Where:
$m$ is the token’s absolute position in the sequence; $x_1, x_2$ are one 2D pair of the query/key vector.
$\theta_i$ is the rotation frequency of the $i$ -th dimension pair, geometrically decreasing across pairs so low dimensions capture short-range and high dimensions long-range offsets.

Figure 1: Similarity vs relative distance: RoPE decays smoothly with distance (a built-in recency prior), while sinusoidal absolute PE stays roughly flat.
Why It Won: RoPE is shift-invariant (the same rotation rule works at any position), adds zero parameters or memory, and, unlike learned absolute embeddings, degrades gracefully beyond the training length, which is why context-extension methods (NTK scaling, YaRN) all build on rescaling its frequencies.
Login to view more content
January 2, 2026
DL0051 Sparsity in NN
Explain the concept of “Sparsity” in neural networks.
Answer
Sparsity means that many of a network’s weights or activations are exactly zero. Sparse models are smaller to store, cheaper to run, and often more interpretable. It arises naturally (ReLU zeroes negative activations) or is induced deliberately through L1 regularization, pruning, or structured sparsity patterns designed for hardware acceleration.
(1) Weight Sparsity: Pruning or L1 drives unimportant weights to zero, so the model keeps accuracy with a fraction of its parameters.
(2) Activation Sparsity: ReLU produces zeros for all negative pre-activations, so only a subset of neurons “fire” per input.
(3) Hardware Payoff: Only structured sparsity (e.g., NVIDIA’s 2:4 pattern: 2 zeros in every 4 values) maps to real speedups; scattered zeros alone rarely accelerate dense GPU kernels.
Mathematical Formulation:
$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_i |w_i|$
$\mathrm{ReLU}(x) = \max(0, x)$
Where:
$w_i$ is the $i$ -th model weight; the L1 penalty pushes small weights to exact zeros, with $\lambda$ controlling sparsity strength.
$x$ is a neuron’s pre-activation input; ReLU’s hard zero is the simplest source of activation sparsity.

Figure 1: L1-induced weight sparsity: without L1 the weights form a broad bell; with L1 they collapse into a spike at exactly zero.
Hardware Support (NVIDIA 2:4): Since Ampere (A100) and continuing in Hopper/Blackwell, Sparse Tensor Cores require every block of 4 values to contain at least 2 zeros (50% structured sparsity); they then skip the zero multiplies, doubling effective GEMM throughput and roughly halving the stored weights plus memory bandwidth.
Login to view more content
December 30, 2025
DL0050 Knowledge Distillation
Describe the process and benefits of knowledge distillation.
Answer
Knowledge distillation (KD) trains a small student model to imitate a large, accurate teacher model. The key trick is learning from the teacher’s temperature-softened output distribution (“dark knowledge”: e.g., a cat image looks a bit like a dog, nothing like a truck) rather than only from one-hot hard labels. The student ends up much smaller and faster while retaining most of the teacher’s accuracy, which is why KD is the standard route to deployable models.
(1) Soft Targets: Teacher logits are passed through softmax with a temperature $T > 1$ , exposing inter-class similarity structure that hard labels hide.
(2) Combined Loss: The student minimizes a mix of distillation loss (KL to the teacher’s soft targets) and ordinary cross-entropy on the true labels.
(3) Benefits: Compression and latency for edge/real-time deployment, plus a regularization effect: students often generalize better than the same architecture trained on hard labels alone.
Mathematical Formulation:
$q_i(T) = \mathrm{softmax}(z_i / T) = \frac{e^{z_i / T}}{\sum_{j=1}^{K} e^{z_j / T}}$
$\mathcal{L} = \alpha\, \mathcal{L}_{\text{CE}}(y, q^{\text{student}}) + (1 - \alpha)\, T^2\, \mathrm{KL}\!\left(q^{\text{teacher}}(T) \,\|\, q^{\text{student}}(T)\right)$
Where:
$z_i$ is the logit for class $i$ , $K$ the number of classes, and $T > 0$ the temperature; higher $T$ yields a smoother distribution.
$\alpha$ balances hard-label CE against distillation; the $T^2$ factor rescales the KL term because softening shrinks its gradients by $1/T^2$ .

Figure 1: Temperature smoothing: at $T = 1$ only the winner class is visible; at higher $T$ the class-similarity ratios (“dark knowledge”) emerge.
Training Setup: The teacher runs in inference mode (frozen); only the student’s weights update. Both models see the same inputs, and the two losses are computed on the student’s outputs only.

Figure 2: The KD setup: the student learns from both the teacher’s soft targets (distillation loss) and the ground truth (student loss).
Practical Caveats: A weak or biased teacher transfers its errors; $T$ and $\alpha$ need tuning; and an extremely small student may lack the capacity to absorb the teacher. Intermediate-feature distillation and task-specific data help close the gap.
Login to view more content
October 28, 2025