Tag: Basics

ML0034 Backpropagation
What is backpropagation?
Answer
Backpropagation (backward propagation of errors) is the algorithm by which neural networks learn: it efficiently computes how much every weight and bias contributed to the prediction error, so an optimizer can adjust each parameter in the direction that reduces the loss. At its core it is the chain rule of calculus applied systematically: after a forward pass computes the output and the loss, a backward pass starts at the output layer, computes the error signal $\delta = \partial L / \partial z$ there, and recursively propagates it backward: each layer’s error is a weighted sum of the next layer’s errors times the local activation derivative, and each weight’s gradient is simply its neuron’s error signal times the input it received. This reuses intermediate results so the cost of computing all gradients is roughly one extra forward pass, which is what makes training deep networks tractable. Gradients then feed an optimizer (SGD, Adam) that performs the actual update $w \leftarrow w - \eta \, \partial L / \partial w$ .
(1) Forward Pass: Inputs flow through the network producing a prediction; all intermediate activations are cached.
(2) Backward Pass: The loss error propagates backward via the chain rule: each layer’s $\delta$ is built from the next layer’s.
(3) Gradient & Update: $\partial L / \partial w_i = \delta \cdot x_i$ ; the optimizer subtracts a learning-rate-scaled step.

Figure 1: The two passes: activations flow forward (blue) through each layer to the loss; error signals $\delta$ flow backward (orange) along the same edges, and every weight’s gradient combines the forward activation with the backward error.
Mathematical Formulation:
$z = \sum_{i} w_i x_i + b$
$a = f(z)$
$\delta = \frac{\partial L}{\partial z} = \frac{\partial L}{\partial a}\, f'(z)$
$\delta_j = \Big( \sum_{k} \delta_k \, w_{kj} \Big) f'(z_j)$
$\frac{\partial L}{\partial w_i} = \delta \cdot x_i$
$\frac{\partial L}{\partial b} = \delta$
Where:
$z$ is the pre-activation and $a = f(z)$ the activation of a neuron; $L$ the loss.
$\delta$ is the error signal: how much the loss changes per unit change of $z$ ; at hidden neuron $j$ it sums contributions $\delta_k w_{kj}$ from all downstream neurons $k$ .
$x_i$ is the input feeding weight $w_i$ : the gradient is proportional to it; the bias gradient equals $\delta$ itself.

Figure 2: One numeric step through a neuron: forward values (blue) compute $z = wx + b$ and $a = \sigma(z)$ ; backward gradients (orange) multiply local derivatives down the chain: $\partial L/\partial w = 0.16 \times 0.5 = 0.08$ , so the weight updates as $w \leftarrow w - \eta \cdot 0.08$ .
Login to view more content
May 8, 2025
ML0033 All Zeros Init
How does initializing all weights and biases to zero affect a neural network’s training?
Answer
Initializing every weight and bias to zero creates a fatal symmetry problem: all neurons in a layer compute the same output, receive the same gradient, and update identically, so they remain identical forever. The layer effectively behaves as a single neuron no matter how wide it is, the network cannot learn diverse features, and its representational capacity collapses. The pathology goes further: with zero weights, activations propagate as zero through the layers (for ReLU the forward signal is exactly 0), and gradients backpropagating through zero weights vanish entirely, so training can stall from the very first step. Proper initialization is therefore not a cosmetic choice: random schemes like Xavier/He break the symmetry and set the activation scale so signals flow through the full depth of the network.
(1) Symmetry: Identical weights → identical outputs → identical gradients → identical neurons forever; width is wasted.
(2) No Learning Signal: Zero weights kill forward activations and backward gradients (especially with ReLU), stalling training.
(3) Fix: Small random initialization (Xavier/He) breaks symmetry and keeps activation/gradient variance stable across layers.

Figure 1: Zero versus random initialization on a binary task: with all-zero weights, train and validation accuracy stay pinned at the 0.5 chance level: the network learns nothing; random initialization breaks symmetry and both curves climb above 0.9.
Mathematical Formulation:
$z_j = \sum_{i} w_{ji} x_i + b_j = 0 \quad \forall j \;\; (\text{all-zero init})$
$\frac{\partial L}{\partial w_{ji}} = \delta_j \, x_i$
$\delta_j = f'(z_j) \sum_k \delta_k w_{kj} = 0$
Where:
$w_{ji}$ is the weight from input $i$ to neuron $j$ , $b_j$ its bias, $z_j$ its pre-activation.
$\delta_j$ is the backpropagated error signal of neuron $j$ ; with downstream weights $w_{kj} = 0$ it collapses to 0, so every weight gradient is 0.
Even where a gradient exists (first layer before the collapse), every neuron in a layer gets the same gradient and stays identical: the symmetry itself is the deeper failure.
Login to view more content
May 6, 2025
ML0032 Non-Linear Activation
Why use non-linear activation functions in neural networks in machine learning, and what limitations would a network face if only linear activation functions were used?
Answer
Non-linear activations are what make a deep network more than a single linear map. Without them, depth is an illusion: composing two linear transformations yields another linear transformation, so a 100-layer network with linear activations collapses algebraically into an equivalent one-layer linear model: it can only ever learn linearly separable relationships, no matter how many parameters it has. Inserting a non-linearity (ReLU, sigmoid, tanh, GELU) after each layer breaks that collapse: the stack can now carve input space into complex regions, build hierarchical representations, and, by the universal approximation theorem, approximate any continuous function to arbitrary accuracy given enough units. The benefits therefore compound: non-linearity introduces the ability to model complex patterns, and it is what lets additional layers add genuine representational power rather than redundant reparameterization.
(1) Introduce Non-Linearity: Enable learning curved decision boundaries and complex input-output patterns.
(2) Universal Approximation: A network with non-linear activations can approximate any continuous function; a linear one cannot.
(3) Depth Matters: With only linear activations, any multilayer network equals a single linear map: layers add nothing.

Figure 1: The limitation in one picture: on V-shaped data, a network with only linear activations can only draw a straight line (red) and misses the structure entirely, while the same-depth network with ReLU activations tracks the true curve (green).
Mathematical Formulation:
$y = W_2 (W_1 x + b_1) + b_2 = \underbrace{W_2 W_1}_{W'} x + \underbrace{W_2 b_1 + b_2}_{b'}$
$h_l = f\big(W_l\, h_{l-1} + b_l\big)$
$f \text{ non-linear} \;\Rightarrow\; h_L \not\equiv W' x + b'$
Where:
$W_l, b_l$ are the weight matrix and bias of layer $l$ ; $h_l$ its output.
The first line shows the collapse: two linear layers reduce to a single effective $(W', b')$ , the core argument against linear-only depth.
$f$ is the non-linear activation (ReLU, sigmoid, …); once it sits between layers, the composition no longer simplifies to one affine map.

Figure 2: Training loss on the same task: the linear-only network plateaus at high error no matter how long it trains (the function class cannot fit the data), while the non-linear network keeps descending to a much lower loss.
Login to view more content
May 5, 2025
DL0011 Fully Connected Layer
Can you explain what a fully connected layer is?
Answer
A fully connected (FC) layer, also called a dense layer, is a layer in which every neuron connects to all neurons of the previous layer. It computes a weighted sum of its inputs, adds a bias, and applies an activation function, letting the network learn global combinations of features. FC layers typically appear at the end of a network, mapping extracted features to class scores.
(1) Dense Connectivity: With $n_{in}$ inputs and $n_{out}$ neurons, the layer holds $n_{in} \times n_{out}$ weights plus $n_{out}$ biases.
(2) Parameter Heavy: FC layers often dominate model size (e.g., flattening a $7 \times 7 \times 512$ map into 1000 classes needs ~25M parameters), which raises overfitting risk.
(3) Loses Spatial Structure: Inputs must be flattened, discarding spatial layout; Global Average Pooling (GAP) is a parameter-free alternative.
Mathematical Formulation:
$y = \sigma(Wx + b)$
$\#\text{params} = (n_{in} + 1) \times n_{out}$
Where:
$x \in \mathbb{R}^{n_{in}}$ is the flattened input vector and $y \in \mathbb{R}^{n_{out}}$ is the output vector.
$W \in \mathbb{R}^{n_{out} \times n_{in}}$ is the weight matrix and $b \in \mathbb{R}^{n_{out}}$ is the bias vector.
$\sigma$ is a non-linear activation such as ReLU (or softmax at the classifier output).

Figure 1: In an FC layer, every input neuron connects to every output neuron; each output is $\sigma(\sum_j w_j x_j + b)$ .
Flatten + FC vs GAP + FC: For a CNN head with $7 \times 7 \times 8$ feature maps and 6 classes, flattening feeds 392 values to the FC layer, while GAP feeds only 8, a 44× parameter reduction with less overfitting:
Approach FC Input Size FC Parameters Spatial Info
Flatten + FC $7 \times 7 \times 8 = 392$ $(392 + 1) \times 6 = 2358$ Discarded by flattening
GAP + FC $8$ (one per channel) $(8 + 1) \times 6 = 54$ Summarized per channel

Figure 2: GAP collapses each feature map to one value before the FC layer, cutting parameters from 2358 to 54 in this example.
Login to view more content
May 3, 2025
DL0010 Receptive Field
What is the receptive field in convolutional neural networks, and how do you calculate it?
Answer
The receptive field (RF) of a neuron is the region of the input image that can influence that neuron’s activation. It grows with network depth, so deeper layers see larger context and learn more hierarchical features. The RF is computed layer by layer: each layer expands the field according to its kernel size, scaled by the cumulative stride of all preceding layers.
(1) Definition: The RF is the input region that affects one activation in a given layer; a neuron with a large RF integrates global context.
(2) Growth Rule: Each layer adds $(k_l - 1) \times \prod s_i$ to the RF, where $k_l$ is the kernel size and the product runs over all previous strides.
(3) Design Implication: Stacked small kernels grow the RF parameter-efficiently; strided and dilated convolutions grow it much faster.
Mathematical Formulation:
$RF_l = RF_{l-1} + (k_l - 1) \times \prod_{i=1}^{l-1} s_i$
Where:
$RF_l$ is the receptive field size after layer $l$ , with $RF_0 = 1$ at the input layer.
$k_l$ is the kernel size of layer $l$ .
$s_i$ is the stride of layer $i$ , and the product accumulates all strides before layer $l$ .

Figure 1: With $k=3, s=1$ per layer, one neuron’s RF grows from 3 to 5 to 7 input cells as layers stack.
Stride and Dilation Effects: A layer with stride 2 doubles the jump between adjacent neurons, so every subsequent layer adds twice as much to the RF. Dilated convolutions enlarge the kernel’s span by inserting gaps, growing the RF without reducing resolution, which is useful in segmentation.

Figure 2: Worked example: a stride-2 layer at $l=3$ doubles the increment contributed by every later layer, jumping the RF from 7 to 11.
Real CNN Stacks: In practice, convolutions, pooling, and dilation mix freely: pooling multiplies the jump between neurons, and dilated kernels widen the span, so the RF can reach 22 within just five layers.

Figure 3: In a realistic stack, max pooling doubles the jump and dilation (D=2) widens the kernel, pushing the RF from 1 to 22 in five layers.
Login to view more content
May 1, 2025

Approach	FC Input Size	FC Parameters	Spatial Info
Flatten + FC	$7 \times 7 \times 8 = 392$	$(392 + 1) \times 6 = 2358$	Discarded by flattening
GAP + FC	$8$ (one per channel)	$(8 + 1) \times 6 = 54$	Summarized per channel

DL0009 Pooling

Please compare max pooling and average pooling in deep learning, and explain in which scenarios you would prefer one over the other.

Answer

Max pooling selects the maximum value within each window of the feature map, keeping the strongest activation in every region. Average pooling computes the mean of all values in the window, producing a smoothed, holistic summary. Both downsample the feature map, but they preserve different information: max pooling keeps sharp, distinctive activations, while average pooling retains the overall activation distribution.

(1) Operation Difference: Max pooling takes the peak activation per window; average pooling takes the window mean: one is sharp and selective, the other smooth and inclusive.
(2) Information Retention: Max pooling may discard weaker features but ignores minor noisy activations; average pooling keeps more of the overall distribution but can average noise into the output.
(3) Typical Use Cases: Prefer max pooling for classification and object detection where feature presence matters; prefer average pooling for segmentation and Global Average Pooling (GAP) where a holistic summary matters.

Mathematical Formulation:
$n_{out} = \left\lfloor \frac{n_{in} - k}{s} \right\rfloor + 1$

Where:

$n_{out}$ and $n_{in}$ are the output and input spatial sizes (identical formula for max and average pooling).
$k$ is the pooling window size and $s$ is the stride (commonly $s = k$ ).

Side-by-side worked example of max pooling and average pooling applied to the same 4x4 input with 2x2 windows, producing different 2x2 outputs.

Figure 1: With 2×2 windows and stride 2, max pooling keeps each region’s peak value while average pooling smooths each region into its mean.

Max Pooling vs Average Pooling in Summary:

Characteristic	Max Pooling	Average Pooling
Operation	Selects maximum value	Calculates average value
Focus	Most prominent features	Overall, smooth representation of features
Noise Sensitivity	Robust to small fluctuations; ignores minor noisy activations	Can incorporate noise if noisy activations are averaged in
Information Retention	May lose weaker feature information	Retains more overall distribution information
Common Use Cases	Object detection, classification	Segmentation, global average pooling

When to Prefer Which: Choose max pooling when the presence of a feature matters more than its exact magnitude, as in detecting edges, textures, or objects. Choose average pooling when you need a balanced regional summary, e.g., aggregating context for segmentation or collapsing feature maps with GAP before the classifier.

Decision flow for choosing max pooling or average pooling based on whether sharp feature presence or a smooth holistic summary is needed.

Figure 2: Decision guide: sharp feature presence points to max pooling; a smooth holistic summary points to average pooling.

May 1, 2025

DL0008 Hyperparameter Tuning
What are the common strategies for hyperparameter tuning in deep learning?
Answer
Hyperparameter tuning optimizes the configuration settings that control the learning process, such as learning rate, batch size, and architecture choices. Because each evaluation requires training a model, the goal is to find strong configurations with as few trials as possible, which makes sample-efficient search strategies essential.
(1) Manual/Heuristic Search: Start with values from prior work or common practice and iteratively adjust based on validation performance.
(2) Grid Search: Exhaustively evaluate all combinations over a predefined discrete grid; simple but scales poorly with dimensionality.
(3) Random Search: Randomly sample values from predefined ranges; covers more distinct values per hyperparameter than grid search for the same budget.
(4) Bayesian Optimization: Use a probabilistic surrogate model to intelligently suggest the next configuration, balancing exploration vs exploitation.

Figure 1: With 9 trials, grid search tests only 3 distinct learning rates, while random search covers 9 distinct values, which matters when one hyperparameter is much more important than the others.
Mathematical Formulation:
$EI(x) = \mathbb{E}\left[\max\left(f(x) - f(x^*),\ 0\right)\right]$
$x_{next} = \arg\max_{x}\ \alpha(x;\ \mathcal{D})$
Where:
$EI(x)$ is the Expected Improvement acquisition function: the expected amount by which $f(x)$ exceeds the best observed score $f(x^*)$ (for maximization).
$\alpha(x;\mathcal{D})$ is the acquisition function (e.g., EI or UCB) computed from the surrogate model fitted on past trials $\mathcal{D}$ ; maximizing it picks the next configuration to evaluate.

Figure 2: Bayesian optimization iterates: fit a surrogate model, pick the next point via an acquisition function, evaluate it, and update the model, repeating until the trial budget is spent.
Hyperparameter Interactions: Validation accuracy is non-monotonic in learning rate, and the optimum shifts with batch size. That is one reason joint search beats tuning one hyperparameter at a time.

Figure 3: Accuracy peaks in the same best lr region for both batch sizes, but the curves differ: hyperparameters interact and should be tuned jointly.
Login to view more content
April 30, 2025
DL0006 Layer Freeze in TL
What are the common strategies for layer freezing in transfer learning?
Answer
Layer freezing controls which pre-trained weights are updated during fine-tuning. Frozen layers keep their learned representations intact, while trainable layers adapt to the target task. The right choice balances leveraging general features against adapting higher-level representations, and depends mainly on target dataset size and task similarity.
(1) Freeze All but the Output Layer(s): Train only the final classification/regression layers; a good starting point for similar tasks and small datasets.
(2) Freeze Early Layers: Early layers capture general features (edges, textures), so train only the later, task-specific layers; effective for moderately similar tasks.
(3) Fine-Tune All Layers with a Low Learning Rate: Adapt all weights slowly; use with caution on small datasets to avoid catastrophic forgetting.
(4) Gradual Unfreezing: Start with frozen layers and progressively unfreeze during training, avoiding large early updates that can destroy learned features.
(5) Backbone-Freeze, Then Low-LR Fine-Tune: Freeze the backbone until the new head converges, then unfreeze it and continue with a reduced learning rate.

Figure 1: The four common strategies differ in how many blocks stay frozen; freeze more when data is scarce and tasks are similar.
Mathematical Formulation:
$\theta = \theta_{frozen} \cup \theta_{trainable}$
$\Delta\theta_{frozen} = 0$
$\theta_{trainable} \leftarrow \theta_{trainable} - \eta\,\nabla_{\theta_{trainable}}\mathcal{L}$
Where:
$\theta_{frozen}$ is the parameter subset kept fixed at its pre-trained values; only $\theta_{trainable}$ receives gradient updates.
$\eta$ is the learning rate; strategies (3) and (5) use a reduced $\eta$ (e.g., 0.1×) to protect pre-trained features.

Figure 2: Gradual unfreezing starts with the head and unfreezes deeper blocks step by step with a reduced learning rate.
Login to view more content
April 30, 2025
DL0005 Transfer Learning
Why use transfer learning in deep learning instead of training from scratch?
Answer
Transfer learning reuses knowledge from a pre-trained model to improve performance, reduce training time and data requirements, and lower computational cost on a new but related task. Instead of learning low-level features from random weights, the model starts from representations that already capture generalizable patterns such as edges, textures, and shapes, so far less target data is needed to reach strong accuracy.
(1) Leverages Existing Knowledge & Reduced Data Requirements: Pre-trained weights encode useful representations learned from large datasets, so good performance is possible with significantly less task-specific data.
(2) Faster Convergence & Training Time: Starting from pre-trained weights is a much better initialization than random weights, leading to faster convergence and often better local optima.
(3) Improved Performance on Limited-Data Tasks: When data is scarce, transfer learning typically yields higher accuracy and better generalization than training from scratch.

Figure 1: The backbone’s weights are copied from pre-training; only the new head (and optionally upper layers) must be learned from limited target data.
Mathematical Formulation:
$\theta^* = \arg\min_{\theta}\ \mathcal{L}_{target}(\theta;\ \theta_{init} = \theta_{pretrained})$
$\eta_l = \eta_{base} \cdot \gamma^{\,L-l}$
Where:
$\theta_{pretrained}$ is the weight set learned on the source task; fine-tuning minimizes the target loss starting from this initialization.
$\eta_l$ is the learning rate of layer $l$ out of $L$ ; a decay factor $\gamma$ below 1 gives earlier (more general) layers smaller updates than later (task-specific) layers.

Figure 2: Transfer learning dominates in the low-data regime; the advantage shrinks as the target dataset grows.
Faster Convergence: Because pre-trained weights are already a strong initialization, the model reaches its accuracy plateau in far fewer epochs than training from random weights.

Figure 3: Transfer learning converges faster and plateaus higher; training from scratch improves slowly over many epochs.
Login to view more content
April 30, 2025
DL0002 All Ones Init
What are the potential consequences of initializing all weights to one in a deep learning model?
Answer
Initializing all weights to one (or any constant non-zero value) creates a symmetry problem: every neuron in the same layer receives identical gradients, so they learn the same features and cannot develop diverse representations. This limits the network to a single effective feature per layer, severely reducing its representational capacity. Training becomes slow or fails to converge because the optimizer cannot differentiate between neurons. Additionally, constant positive weights can push activations into saturated regions of sigmoid or tanh, worsening vanishing gradients.
(1) Symmetry Problem: All neurons in a layer receive the same input and the same gradient, so they evolve identically throughout training.
(2) Limited Representational Capacity: The network behaves as if each layer has only one neuron, making it impossible to capture complex, varied patterns in the data.
(3) Slow or Failed Convergence: Because neurons cannot differentiate, the loss landscape lacks the diversity needed for gradient descent to find useful minima.
(4) Activation Saturation: Uniform positive weights shift pre-activation values into the flat tails of sigmoid and tanh, producing vanishing gradients that stall learning.

Figure 1: The symmetry problem: with all-ones weights every neuron computes the same function and receives the same gradient, while random initialization lets each neuron learn a distinct feature.
Mathematical Formulation:
$\frac{\partial \mathcal{L}}{\partial w_{ij}^{(l)}} = \delta_i^{(l)} \cdot a_j^{(l-1)}$
$w_{ij}^{(l)} = 1 \quad \forall\, i,j \implies \delta_i^{(l)} = \delta_j^{(l)} \quad \forall\, i,j$
Where:
$w_{ij}^{(l)}$ is the weight from neuron $j$ in layer $l-1$ to neuron $i$ in layer $l$ .
$\delta_i^{(l)}$ is the error signal (delta) for neuron $i$ in layer $l$ .
$a_j^{(l-1)}$ is the activation of neuron $j$ in the previous layer.
$\mathcal{L}$ is the loss function; when all weights are equal, all delta signals are identical, so every weight updates by the same amount.

Figure 2: Training loss comparison: ones initialization converges slowly and plateaus at a high loss, while random initialization descends rapidly to a low minimum.
Login to view more content
April 28, 2025