Author: admin

ML0035 Model Comparison
How to compare different machine learning models?
Answer
Comparing machine learning models rigorously means more than reading one accuracy number off one test run. A sound comparison fixes the evaluation protocol first: choose metrics that match the task and its costs (accuracy or F1/ROC-AUC for classification, RMSE/MAE for regression), then evaluate every model on the same train/validation/test splits (or better, the same cross-validation folds), so differences are attributable to the models, not the data lottery. Because training has randomness (shuffles, weight initialization), each model should be run multiple times with different seeds and its mean and spread reported; when two models look close, a paired statistical test (e.g., a paired t-test or Wilcoxon test over per-fold scores) tells whether the gap is significant or noise. Finally, break ties and inform deployment with secondary criteria: training/inference cost, robustness to perturbations, and interpretability.
(1) Right Metrics: Task- and cost-appropriate metrics (F1/ROC-AUC, RMSE, …), never a single default number.
(2) Controlled Comparison: Identical splits/folds, multiple seeds, cross-validation; statistical tests for close calls.
(3) Secondary Criteria: Latency, memory, robustness, interpretability decide between statistical ties.

Figure 1: Comparing two classifiers on the same test set with ROC curves: the random forest’s curve dominates logistic regression’s at nearly every threshold (AUC 0.98 vs 0.90), a richer comparison than any single-threshold metric.
Mathematical Formulation:
$\bar{s}_m = \frac{1}{K} \sum_{k=1}^{K} s_{m,k}$
$\mathrm{Var}(s_m) = \frac{1}{K-1} \sum_{k=1}^{K} \big(s_{m,k} - \bar{s}_m\big)^2$
$t = \frac{\bar{d}}{\mathrm{std}(d) / \sqrt{K}}$
$d_k = s_{1,k} - s_{2,k} \quad \text{(paired test over folds)}$
Where:
$s_{m,k}$ is the score of model $m$ on fold (or seed) $k$ ; $\bar{s}_m$ and $\mathrm{Var}(s_m)$ summarize central performance and stability.
$d_k$ is the per-fold score difference between two models: pairing removes fold-to-fold variance, so the $t$ -statistic tests whether the mean gap is distinguishable from zero.
Login to view more content
May 11, 2025
DL0012 Zero Padding
Why is zero padding used in deep learning?
Answer
Zero padding adds rows and columns of zeros around the input before a convolution. In CNNs it preserves spatial dimensions, prevents border information from being under-sampled, allows larger kernels and deeper stacks, and gives explicit control over output size. Beyond CNNs, padding standardizes variable-length sequences so NLP and time-series models can process them in batches.
(1) Preserves Spatial Dimensions: Without padding (“valid” convolution), a $k \times k$ kernel shrinks the feature map by $k - 1$ in total per dimension ( $(k-1)/2$ per side) each layer; padding with $p = (k-1)/2$ keeps the size unchanged.
(2) Retains Boundary Information: Padded borders let the kernel center on edge pixels, so corners and boundaries are processed as thoroughly as the interior.
(3) Controls Output Size: Padding decouples output dimensions from kernel size, enabling deeper networks and predictable feature-map shapes.
Mathematical Formulation:
$n_{out} = \left\lfloor \frac{n_{in} + 2p - k}{s} \right\rfloor + 1$
Where:
$n_{out}$ and $n_{in}$ are the output and input spatial sizes.
$p$ is the padding width added to each side, $k$ is the kernel size, and $s$ is the stride.
“Same” padding for stride 1 uses $p = (k-1)/2$ , giving $n_{out} = n_{in}$ .

Figure 1: Padding a $4 \times 4$ input to $6 \times 6$ lets a $3 \times 3$ kernel output the same $4 \times 4$ size instead of shrinking to $2 \times 2$ .
Beyond CNNs: In NLP and time-series tasks, zero padding extends shorter sequences to a uniform length for efficient batching. Because padded positions carry no information, models combine padding with attention masks so Transformer self-attention ignores those positions entirely.

Figure 2: Valid shrinks the map, same preserves it, and sequence padding plus a mask enables batched NLP inputs.
Login to view more content
May 11, 2025
ML0034 Backpropagation
What is backpropagation?
Answer
Backpropagation (backward propagation of errors) is the algorithm by which neural networks learn: it efficiently computes how much every weight and bias contributed to the prediction error, so an optimizer can adjust each parameter in the direction that reduces the loss. At its core it is the chain rule of calculus applied systematically: after a forward pass computes the output and the loss, a backward pass starts at the output layer, computes the error signal $\delta = \partial L / \partial z$ there, and recursively propagates it backward: each layer’s error is a weighted sum of the next layer’s errors times the local activation derivative, and each weight’s gradient is simply its neuron’s error signal times the input it received. This reuses intermediate results so the cost of computing all gradients is roughly one extra forward pass, which is what makes training deep networks tractable. Gradients then feed an optimizer (SGD, Adam) that performs the actual update $w \leftarrow w - \eta \, \partial L / \partial w$ .
(1) Forward Pass: Inputs flow through the network producing a prediction; all intermediate activations are cached.
(2) Backward Pass: The loss error propagates backward via the chain rule: each layer’s $\delta$ is built from the next layer’s.
(3) Gradient & Update: $\partial L / \partial w_i = \delta \cdot x_i$ ; the optimizer subtracts a learning-rate-scaled step.

Figure 1: The two passes: activations flow forward (blue) through each layer to the loss; error signals $\delta$ flow backward (orange) along the same edges, and every weight’s gradient combines the forward activation with the backward error.
Mathematical Formulation:
$z = \sum_{i} w_i x_i + b$
$a = f(z)$
$\delta = \frac{\partial L}{\partial z} = \frac{\partial L}{\partial a}\, f'(z)$
$\delta_j = \Big( \sum_{k} \delta_k \, w_{kj} \Big) f'(z_j)$
$\frac{\partial L}{\partial w_i} = \delta \cdot x_i$
$\frac{\partial L}{\partial b} = \delta$
Where:
$z$ is the pre-activation and $a = f(z)$ the activation of a neuron; $L$ the loss.
$\delta$ is the error signal: how much the loss changes per unit change of $z$ ; at hidden neuron $j$ it sums contributions $\delta_k w_{kj}$ from all downstream neurons $k$ .
$x_i$ is the input feeding weight $w_i$ : the gradient is proportional to it; the bias gradient equals $\delta$ itself.

Figure 2: One numeric step through a neuron: forward values (blue) compute $z = wx + b$ and $a = \sigma(z)$ ; backward gradients (orange) multiply local derivatives down the chain: $\partial L/\partial w = 0.16 \times 0.5 = 0.08$ , so the weight updates as $w \leftarrow w - \eta \cdot 0.08$ .
Login to view more content
May 8, 2025
ML0033 All Zeros Init
How does initializing all weights and biases to zero affect a neural network’s training?
Answer
Initializing every weight and bias to zero creates a fatal symmetry problem: all neurons in a layer compute the same output, receive the same gradient, and update identically, so they remain identical forever. The layer effectively behaves as a single neuron no matter how wide it is, the network cannot learn diverse features, and its representational capacity collapses. The pathology goes further: with zero weights, activations propagate as zero through the layers (for ReLU the forward signal is exactly 0), and gradients backpropagating through zero weights vanish entirely, so training can stall from the very first step. Proper initialization is therefore not a cosmetic choice: random schemes like Xavier/He break the symmetry and set the activation scale so signals flow through the full depth of the network.
(1) Symmetry: Identical weights → identical outputs → identical gradients → identical neurons forever; width is wasted.
(2) No Learning Signal: Zero weights kill forward activations and backward gradients (especially with ReLU), stalling training.
(3) Fix: Small random initialization (Xavier/He) breaks symmetry and keeps activation/gradient variance stable across layers.

Figure 1: Zero versus random initialization on a binary task: with all-zero weights, train and validation accuracy stay pinned at the 0.5 chance level: the network learns nothing; random initialization breaks symmetry and both curves climb above 0.9.
Mathematical Formulation:
$z_j = \sum_{i} w_{ji} x_i + b_j = 0 \quad \forall j \;\; (\text{all-zero init})$
$\frac{\partial L}{\partial w_{ji}} = \delta_j \, x_i$
$\delta_j = f'(z_j) \sum_k \delta_k w_{kj} = 0$
Where:
$w_{ji}$ is the weight from input $i$ to neuron $j$ , $b_j$ its bias, $z_j$ its pre-activation.
$\delta_j$ is the backpropagated error signal of neuron $j$ ; with downstream weights $w_{kj} = 0$ it collapses to 0, so every weight gradient is 0.
Even where a gradient exists (first layer before the collapse), every neuron in a layer gets the same gradient and stays identical: the symmetry itself is the deeper failure.
Login to view more content
May 6, 2025
ML0032 Non-Linear Activation
Why use non-linear activation functions in neural networks in machine learning, and what limitations would a network face if only linear activation functions were used?
Answer
Non-linear activations are what make a deep network more than a single linear map. Without them, depth is an illusion: composing two linear transformations yields another linear transformation, so a 100-layer network with linear activations collapses algebraically into an equivalent one-layer linear model: it can only ever learn linearly separable relationships, no matter how many parameters it has. Inserting a non-linearity (ReLU, sigmoid, tanh, GELU) after each layer breaks that collapse: the stack can now carve input space into complex regions, build hierarchical representations, and, by the universal approximation theorem, approximate any continuous function to arbitrary accuracy given enough units. The benefits therefore compound: non-linearity introduces the ability to model complex patterns, and it is what lets additional layers add genuine representational power rather than redundant reparameterization.
(1) Introduce Non-Linearity: Enable learning curved decision boundaries and complex input-output patterns.
(2) Universal Approximation: A network with non-linear activations can approximate any continuous function; a linear one cannot.
(3) Depth Matters: With only linear activations, any multilayer network equals a single linear map: layers add nothing.

Figure 1: The limitation in one picture: on V-shaped data, a network with only linear activations can only draw a straight line (red) and misses the structure entirely, while the same-depth network with ReLU activations tracks the true curve (green).
Mathematical Formulation:
$y = W_2 (W_1 x + b_1) + b_2 = \underbrace{W_2 W_1}_{W'} x + \underbrace{W_2 b_1 + b_2}_{b'}$
$h_l = f\big(W_l\, h_{l-1} + b_l\big)$
$f \text{ non-linear} \;\Rightarrow\; h_L \not\equiv W' x + b'$
Where:
$W_l, b_l$ are the weight matrix and bias of layer $l$ ; $h_l$ its output.
The first line shows the collapse: two linear layers reduce to a single effective $(W', b')$ , the core argument against linear-only depth.
$f$ is the non-linear activation (ReLU, sigmoid, …); once it sits between layers, the composition no longer simplifies to one affine map.

Figure 2: Training loss on the same task: the linear-only network plateaus at high error no matter how long it trains (the function class cannot fit the data), while the non-linear network keeps descending to a much lower loss.
Login to view more content
May 5, 2025
DL0011 Fully Connected Layer
Can you explain what a fully connected layer is?
Answer
A fully connected (FC) layer, also called a dense layer, is a layer in which every neuron connects to all neurons of the previous layer. It computes a weighted sum of its inputs, adds a bias, and applies an activation function, letting the network learn global combinations of features. FC layers typically appear at the end of a network, mapping extracted features to class scores.
(1) Dense Connectivity: With $n_{in}$ inputs and $n_{out}$ neurons, the layer holds $n_{in} \times n_{out}$ weights plus $n_{out}$ biases.
(2) Parameter Heavy: FC layers often dominate model size (e.g., flattening a $7 \times 7 \times 512$ map into 1000 classes needs ~25M parameters), which raises overfitting risk.
(3) Loses Spatial Structure: Inputs must be flattened, discarding spatial layout; Global Average Pooling (GAP) is a parameter-free alternative.
Mathematical Formulation:
$y = \sigma(Wx + b)$
$\#\text{params} = (n_{in} + 1) \times n_{out}$
Where:
$x \in \mathbb{R}^{n_{in}}$ is the flattened input vector and $y \in \mathbb{R}^{n_{out}}$ is the output vector.
$W \in \mathbb{R}^{n_{out} \times n_{in}}$ is the weight matrix and $b \in \mathbb{R}^{n_{out}}$ is the bias vector.
$\sigma$ is a non-linear activation such as ReLU (or softmax at the classifier output).

Figure 1: In an FC layer, every input neuron connects to every output neuron; each output is $\sigma(\sum_j w_j x_j + b)$ .
Flatten + FC vs GAP + FC: For a CNN head with $7 \times 7 \times 8$ feature maps and 6 classes, flattening feeds 392 values to the FC layer, while GAP feeds only 8, a 44× parameter reduction with less overfitting:
Approach FC Input Size FC Parameters Spatial Info
Flatten + FC $7 \times 7 \times 8 = 392$ $(392 + 1) \times 6 = 2358$ Discarded by flattening
GAP + FC $8$ (one per channel) $(8 + 1) \times 6 = 54$ Summarized per channel

Figure 2: GAP collapses each feature map to one value before the FC layer, cutting parameters from 2358 to 54 in this example.
Login to view more content
May 3, 2025
DL0010 Receptive Field
What is the receptive field in convolutional neural networks, and how do you calculate it?
Answer
The receptive field (RF) of a neuron is the region of the input image that can influence that neuron’s activation. It grows with network depth, so deeper layers see larger context and learn more hierarchical features. The RF is computed layer by layer: each layer expands the field according to its kernel size, scaled by the cumulative stride of all preceding layers.
(1) Definition: The RF is the input region that affects one activation in a given layer; a neuron with a large RF integrates global context.
(2) Growth Rule: Each layer adds $(k_l - 1) \times \prod s_i$ to the RF, where $k_l$ is the kernel size and the product runs over all previous strides.
(3) Design Implication: Stacked small kernels grow the RF parameter-efficiently; strided and dilated convolutions grow it much faster.
Mathematical Formulation:
$RF_l = RF_{l-1} + (k_l - 1) \times \prod_{i=1}^{l-1} s_i$
Where:
$RF_l$ is the receptive field size after layer $l$ , with $RF_0 = 1$ at the input layer.
$k_l$ is the kernel size of layer $l$ .
$s_i$ is the stride of layer $i$ , and the product accumulates all strides before layer $l$ .

Figure 1: With $k=3, s=1$ per layer, one neuron’s RF grows from 3 to 5 to 7 input cells as layers stack.
Stride and Dilation Effects: A layer with stride 2 doubles the jump between adjacent neurons, so every subsequent layer adds twice as much to the RF. Dilated convolutions enlarge the kernel’s span by inserting gaps, growing the RF without reducing resolution, which is useful in segmentation.

Figure 2: Worked example: a stride-2 layer at $l=3$ doubles the increment contributed by every later layer, jumping the RF from 7 to 11.
Real CNN Stacks: In practice, convolutions, pooling, and dilation mix freely: pooling multiplies the jump between neurons, and dilated kernels widen the span, so the RF can reach 22 within just five layers.

Figure 3: In a realistic stack, max pooling doubles the jump and dilation (D=2) widens the kernel, pushing the RF from 1 to 22 in five layers.
Login to view more content
May 1, 2025

Approach	FC Input Size	FC Parameters	Spatial Info
Flatten + FC	$7 \times 7 \times 8 = 392$	$(392 + 1) \times 6 = 2358$	Discarded by flattening
GAP + FC	$8$ (one per channel)	$(8 + 1) \times 6 = 54$	Summarized per channel

DL0009 Pooling

Please compare max pooling and average pooling in deep learning, and explain in which scenarios you would prefer one over the other.

Answer

Max pooling selects the maximum value within each window of the feature map, keeping the strongest activation in every region. Average pooling computes the mean of all values in the window, producing a smoothed, holistic summary. Both downsample the feature map, but they preserve different information: max pooling keeps sharp, distinctive activations, while average pooling retains the overall activation distribution.

(1) Operation Difference: Max pooling takes the peak activation per window; average pooling takes the window mean: one is sharp and selective, the other smooth and inclusive.
(2) Information Retention: Max pooling may discard weaker features but ignores minor noisy activations; average pooling keeps more of the overall distribution but can average noise into the output.
(3) Typical Use Cases: Prefer max pooling for classification and object detection where feature presence matters; prefer average pooling for segmentation and Global Average Pooling (GAP) where a holistic summary matters.

Mathematical Formulation:
$n_{out} = \left\lfloor \frac{n_{in} - k}{s} \right\rfloor + 1$

Where:

$n_{out}$ and $n_{in}$ are the output and input spatial sizes (identical formula for max and average pooling).
$k$ is the pooling window size and $s$ is the stride (commonly $s = k$ ).

Side-by-side worked example of max pooling and average pooling applied to the same 4x4 input with 2x2 windows, producing different 2x2 outputs.

Figure 1: With 2×2 windows and stride 2, max pooling keeps each region’s peak value while average pooling smooths each region into its mean.

Max Pooling vs Average Pooling in Summary:

Characteristic	Max Pooling	Average Pooling
Operation	Selects maximum value	Calculates average value
Focus	Most prominent features	Overall, smooth representation of features
Noise Sensitivity	Robust to small fluctuations; ignores minor noisy activations	Can incorporate noise if noisy activations are averaged in
Information Retention	May lose weaker feature information	Retains more overall distribution information
Common Use Cases	Object detection, classification	Segmentation, global average pooling

When to Prefer Which: Choose max pooling when the presence of a feature matters more than its exact magnitude, as in detecting edges, textures, or objects. Choose average pooling when you need a balanced regional summary, e.g., aggregating context for segmentation or collapsing feature maps with GAP before the classifier.

Decision flow for choosing max pooling or average pooling based on whether sharp feature presence or a smooth holistic summary is needed.

Figure 2: Decision guide: sharp feature presence points to max pooling; a smooth holistic summary points to average pooling.

May 1, 2025

DL0008 Hyperparameter Tuning
What are the common strategies for hyperparameter tuning in deep learning?
Answer
Hyperparameter tuning optimizes the configuration settings that control the learning process, such as learning rate, batch size, and architecture choices. Because each evaluation requires training a model, the goal is to find strong configurations with as few trials as possible, which makes sample-efficient search strategies essential.
(1) Manual/Heuristic Search: Start with values from prior work or common practice and iteratively adjust based on validation performance.
(2) Grid Search: Exhaustively evaluate all combinations over a predefined discrete grid; simple but scales poorly with dimensionality.
(3) Random Search: Randomly sample values from predefined ranges; covers more distinct values per hyperparameter than grid search for the same budget.
(4) Bayesian Optimization: Use a probabilistic surrogate model to intelligently suggest the next configuration, balancing exploration vs exploitation.

Figure 1: With 9 trials, grid search tests only 3 distinct learning rates, while random search covers 9 distinct values, which matters when one hyperparameter is much more important than the others.
Mathematical Formulation:
$EI(x) = \mathbb{E}\left[\max\left(f(x) - f(x^*),\ 0\right)\right]$
$x_{next} = \arg\max_{x}\ \alpha(x;\ \mathcal{D})$
Where:
$EI(x)$ is the Expected Improvement acquisition function: the expected amount by which $f(x)$ exceeds the best observed score $f(x^*)$ (for maximization).
$\alpha(x;\mathcal{D})$ is the acquisition function (e.g., EI or UCB) computed from the surrogate model fitted on past trials $\mathcal{D}$ ; maximizing it picks the next configuration to evaluate.

Figure 2: Bayesian optimization iterates: fit a surrogate model, pick the next point via an acquisition function, evaluate it, and update the model, repeating until the trial budget is spent.
Hyperparameter Interactions: Validation accuracy is non-monotonic in learning rate, and the optimum shifts with batch size. That is one reason joint search beats tuning one hyperparameter at a time.

Figure 3: Accuracy peaks in the same best lr region for both batch sizes, but the curves differ: hyperparameters interact and should be tuned jointly.
Login to view more content
April 30, 2025
DL0007 Batch Norm
Why use batch normalization in deep learning training?
Answer
Batch normalization stabilizes and accelerates training by normalizing each layer’s inputs across the mini-batch: subtract the batch mean and divide by the batch standard deviation. After normalization, a learnable scale $\gamma$ and shift $\beta$ let the network recover the identity transformation if needed. BN is applied after the linear transform (e.g., the convolution) and before the activation (e.g., ReLU).
(1) Stabilizes Learning: Reduces internal covariate shift, making training less sensitive to initialization and hyperparameter choices.
(2) Enables Higher Learning Rates: Larger learning rates can be used without instability, leading to faster convergence.
(3) Improves Generalization: Normalizing per mini-batch introduces noise into activations, preventing over-reliance on specific batches and acting as a mild regularizer.

Figure 1: Each feature channel is normalized using its mini-batch statistics, then rescaled by learnable $\gamma$ and $\beta$ .
Mathematical Formulation:
$BN(x_i) = \gamma \left( \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} \right) + \beta$
Where:
$x_i$ is an individual feature value in the batch.
$\mu_B$ and $\sigma_B^2$ are the mean and variance of that feature across the current mini-batch.
$\epsilon$ is a small constant (e.g., $10^{-5}$ ) added for numerical stability.
$\gamma$ and $\beta$ are learnable scaling and shifting parameters.

Figure 2: Training computes per-batch statistics and updates running averages; inference uses the frozen running averages, producing deterministic outputs.
Placement in a CNN Block: BN sits between the linear transform and the non-linearity: Conv2D → BatchNorm → ReLU.

Figure 3: In a standard CNN block, BatchNorm normalizes the convolution output before the activation function.
Login to view more content
April 30, 2025