Interview for Machine Learning

Category: Medium

DL0012 Zero Padding
Why is zero padding used in deep learning?
Answer
Zero padding in deep learning, particularly in CNNs, is a technique of adding layers of zeros around the input to convolutional layers. This is crucial for maintaining the spatial dimensions of feature maps, preventing loss of information at the image or feature map borders, enabling the use of larger receptive fields via larger kernels, and providing control over the output size of convolutional operations. Ultimately, it helps in building deeper and more effective neural networks by preserving important spatial information throughout the network.
Here are the benefits of using zero padding for CNN.
(1) Preserves Spatial Dimensions: Prevents feature maps from shrinking after convolution.
Below is a 2D image example for using zero padding in CNN.

(2) Retains Boundary Information: Ensures edge pixels are processed adequately.
(3) Enables Larger Kernels: Allows using bigger filters without excessive size reduction.
(4) Controls Output Size: Provides a mechanism to manage the dimensions of output feature maps.
Beyond CNNs, zero padding plays a vital role in deep learning by standardizing variable-length sequences in tasks like NLP and time-series modeling. It ensures inputs have uniform dimensions for efficient batching and computation, enhances frequency resolution in spectral analyses, and allows for effective loss masking to focus learning on actual data.
(1) Standardizing Variable-Length Inputs: In NLP and time-series analysis, zero padding ensures that sequences of varying lengths have a uniform size. This uniformity is crucial for batch processing and for models like recurrent neural networks (RNNs) or transformers.
(2) Attention Masking in Transformers: Padding tokens in Transformer inputs are assigned zero values and then excluded via padding masks in self-attention layers, preventing the model from attending to irrelevant positions in the sequence.
Login to view more content
May 11, 2025
DL0010 Receptive Field
What is the receptive field in convolutional neural networks, and how do you calculate it?
Answer
In convolutional neural networks (CNNs), the receptive field of a neuron is the region of the input image that can affect that neuron’s activation. Receptive field Increases in deeper layers, allowing the network to learn hierarchical features.
Use the following iterative formula to calculate the Receptive field:
$RF_l = RF_{l-1} + (k_l - 1) \times \prod_{i=1}^{l-1} s_i$
Where:
$RF_l$ represents the receptive field size in layer $l$ . $RF_0 = 1$ for the input layer.
$k_l$ represents the kernel size of layer $l$ .
$s_i$ represents the stride of layer $i$ .
The following image shows an example of receptive field size growth in a CNN.
K means kernel size, S means stride, and D means dilation rate.
Login to view more content
May 1, 2025
DL0003 1×1 Convolution
What are the benefits of using 1×1 convolutional layers in deep learning architectures?
Answer
A 1×1 convolution, also known as a pointwise convolution, is a convolutional operation where the kernel size is 1×1, which plays several crucial roles in deep learning architectures.
(1) Dimensionality control: 1×1 convolution can reduce or expand the number of feature maps, trading off representational capacity and computational cost.
For example, Bottleneck designs: In architectures like ResNet’s bottleneck block, a 1×1 conv first reduces channels (e.g., 256→64), then a 3×3 conv processes those, and finally another 1×1 conv expands back (64→256) to restore capacity while keeping compute manageable.
(2) Increased Network Depth with Controlled Cost: Allows for the design of deeper networks by reducing channel dimensionality before computationally expensive spatial convolutions.
(3) Cross-Channel Feature Fusion: Enables interaction and combination of information across different feature channels at the same spatial location.
(4) Non-linear mixing: When followed by activations (ReLU, etc.), 1×1 convolutions introduce non‐linear channel mixing that enhances model expressiveness.
Login to view more content
April 29, 2025
DL0002 All Ones Init
What are the potential consequences of initializing all weights to one in a deep learning model?
Answer
Below are the key consequences of initializing all weights in a deep-learning model to one (a constant non-zero value), illustrating why random, scaled initializations (e.g., Xavier/He) are essential.
(1) Symmetry Problem: Neurons receive identical gradients, causing them to learn the same features rather than developing distinct representations.
(2) Limited Representational Capacity: The network cannot capture complex, varied patterns because all neurons behave identically.
(3) Slow/No Convergence: The lack of Representational Capacity further makes it difficult for the model to update to the optimal weights. (The below image shows an example for training loss comparison for ones initialization vs random initialization)

(4) Activation Saturation: Can push neurons into saturated regions of activation functions (e.g., sigmoid, tanh), leading to vanishing gradients.
Login to view more content
April 28, 2025
DL0001 Residual Connection
Why are residual connections important in deep neural networks?
Answer
A residual connection (skip connection) adds a block input to a learned residual transformation, so the block learns an update rather than an entirely new mapping. For a shape-preserving block, the local Jacobian becomes $I+J_F$ , which supplies a direct identity component and creates shorter paths through the computational graph. This usually improves gradient propagation and optimization, but it does not guarantee nonzero or constant gradients: products of residual-block Jacobians can still shrink, grow, or cancel. Residual connections also address the degradation problem, in which adding layers to a plain network can increase training error because the deeper model is difficult to optimize even though an identity extension exists. These properties enabled effective training of ResNet architectures with hundreds of layers.

Figure 1: A shape-preserving block uses an identity shortcut, while a block that changes resolution or channel width uses a learned projection $S_l$ so the two paths have compatible shapes.
(1) Shorter Gradient Paths: An identity shortcut changes the local block Jacobian from $J_F$ to $I+J_F$ . The identity component gives backpropagation additional routes, although the product across many blocks can still vanish or explode.
(2) Identity Mapping Fallback: When an additional shape-preserving block is not useful, its residual branch $F_l(x_l)$ can approach zero, making $x_{l+1}\approx x_l$ easier to represent than in a plain nonlinear stack.
(3) Easier Deep Optimization: Residual parameterization helps deeper models avoid the degradation problem, where added layers increase training error because optimization fails to recover a useful identity extension.
Mathematical Formulation:
$x_{l+1}=F_l(x_l;W_l)+x_l$
$\frac{\partial x_{l+1}}{\partial x_l}=J_{F_l}+I$
$x_{l+1}=F_l(x_l;W_l)+S_lx_l$
Where:
$x_l$ is the input to residual block $l$ , and $x_{l+1}$ is its output.
$F_l(x_l;W_l)$ is the learned residual branch parameterized by weights $W_l$ ; it learns an update to the shortcut representation.
$J_{F_l}=\partial F_l/\partial x_l$ is the residual-branch Jacobian, and $I$ is the identity operator with the same feature dimension as $x_l$ .
$S_l$ is an identity operator when input and output shapes match; otherwise it can be a learned projection, such as a strided $1\times1$ convolution, that aligns spatial and channel dimensions.
Across $L$ residual blocks, backpropagation contains products of factors $I+J_{F_l}$ . These factors often improve conditioning relative to plain Jacobian products, but they do not impose a nonzero lower bound on gradient magnitude.

Figure 2: A plain stack multiplies full layer Jacobians, whereas a residual stack multiplies factors $I+J_{F_l}$ . The identity components provide additional gradient routes but do not guarantee stable magnitude.
Login to view more content
April 23, 2025