Tag: Basics

DL0049 Weight Init

Why is “weight initialization” important in deep neural networks?

Answer

Initialization decides whether signals survive a deep network. If weights are too small, activations and gradients shrink toward zero layer by layer (vanishing); too large, and they blow up (exploding) or saturate sigmoid/tanh into zero-gradient plateaus. Proper schemes like Xavier/Glorot and He scale the variance to the layer’s fan-in/fan-out so activations keep unit-scale statistics, while randomness breaks symmetry so neurons learn different features.

(1) Prevents Vanishing/Exploding Signals: Keeping activation variance constant across layers keeps gradients at a usable scale during backprop.
(2) Breaks Symmetry: Identical initial weights make neurons identical forever; random init gives each a distinct feature to learn.
(3) Matches the Activation: Xavier suits symmetric activations (tanh/sigmoid); He doubles the variance for ReLU, which zeroes half its inputs.

Mathematical Formulation:
$\text{Xavier:}\quad W \sim \mathcal{N}\!\left(0,\; \frac{2}{n_{\text{in}} + n_{\text{out}}}\right) \;\; \text{or} \;\; \mathcal{U}\!\left(-\sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}},\; \sqrt{\frac{6}{n_{\text{in}} + n_{\text{out}}}}\right)$
$\text{He:}\quad W \sim \mathcal{N}\!\left(0,\; \frac{2}{n_{\text{in}}}\right) \;\; \text{or} \;\; \mathcal{U}\!\left(-\sqrt{\frac{6}{n_{\text{in}}}},\; \sqrt{\frac{6}{n_{\text{in}}}}\right)$

Where:

$n_{\text{in}}$ and $n_{\text{out}}$ are the layer’s input and output unit counts (fan-in / fan-out).
Xavier balances both directions for symmetric activations; He drops $n_{\text{out}}$ and doubles variance because ReLU discards half the signal.

Three stacked histogram panels of post-ReLU activation distributions at layers 3 to 6, showing plain random init collapsing to a spike at zero, Xavier shrinking toward zero with depth, and He init keeping a stable spread.

Figure 1: Post-ReLU activations by depth: plain random init collapses and Xavier fades, while He init keeps a healthy spread even at layer 6.

Activation / Architecture	Recommended Init	Why
ReLU (CNNs, ResNet)	He / Kaiming	Doubles variance to compensate for ~50% zeros from ReLU
Tanh / Sigmoid (MLPs)	Xavier / Glorot	Balances fan-in and fan-out for symmetric activations
GELU (BERT-scale Transformers)	Truncated normal (σ ≈ 0.02)	LayerNorm + residuals already stabilize; gentle init suffices
Very deep LLMs (GPT, LLaMA)	Scaled normal (DeepNorm-style)	Residual-branch scaling stops signal growth across 100+ layers

Table 1: Init choice is activation- and architecture-dependent: there is no single best scheme.

October 28, 2025

DL0048 Adam Optimizer
Can you explain how the Adam optimizer works?
Answer
Adam (Adaptive Moment Estimation) combines momentum and RMSprop: it keeps an exponentially decaying average of the gradient (first moment, the direction) and of the squared gradient (second moment, the scale), then divides the former by the square root of the latter. The result is a per-parameter adaptive learning rate: large steps for parameters with small, consistent gradients, small steps for noisy or steep ones, plus bias correction that fixes the zero-initialization of both averages in early steps.
(1) First Moment (Momentum): $m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t$ smooths the gradient direction over time.
(2) Second Moment (RMSprop): $v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$ tracks the gradient magnitude for per-parameter scaling.
(3) Bias Correction + Update: $\hat{m}_t = m_t/(1-\beta_1^t)$ , $\hat{v}_t = v_t/(1-\beta_2^t)$ remove the zero-init bias before the normalized step.
Mathematical Formulation:
$\theta_t = \theta_{t-1} - \alpha\, \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$
Where:
$\theta_t$ are the model parameters at step $t$ , and $g_t$ is the gradient of the loss at that step.
$\alpha$ is the learning rate (default $0.001$ ); $\epsilon \approx 10^{-8}$ prevents division by zero.
$\beta_1 = 0.9$ and $\beta_2 = 0.999$ control the decay of the first and second moment averages (paper defaults).

Figure 1: Adam on a quadratic bowl: adaptive per-parameter steps converge smoothly to the minimum without the zig-zag of plain SGD.
Intuition for the Division: $\hat{m}_t / \sqrt{\hat{v}_t}$ is roughly a signal-to-noise ratio: parameters whose gradients are large but inconsistent (high $v$ ) get small updates, while parameters with small but persistent gradients get amplified. This is what makes Adam robust to ill-scaled features and sparse gradients.
Login to view more content
October 27, 2025
ML0050 Logistic Regression III
Why is Mean Squared Error (L2 Loss) an unsuitable loss function for logistic regression compared to cross-entropy?
Answer
Mean Squared Error (MSE) is unsuitable for logistic regression primarily because, combined with the sigmoid, it produces a non-convex loss landscape: optimization becomes harder and convergence less reliable. It also provides weaker gradients exactly when predictions are confidently wrong, slowing learning. Cross-entropy aligns with the Bernoulli distribution assumption behind binary classification, yields a convex loss for the single-neuron binary setting, and delivers strong gradients throughout.
(1) Wrong Assumption: MSE assumes Gaussian-distributed errors, while logistic regression models a Bernoulli (binary) outcome.
(2) Non-Convex Optimization: MSE on top of the sigmoid creates a non-convex surface: gradient descent can stall in flat regions or poor local behavior.
(3) Gradient Issues: With MSE, confident wrong predictions produce tiny gradients (the sigmoid saturates), slowing learning; cross-entropy keeps the gradient strong precisely there.
(4) Interpretation: Cross-entropy directly compares predicted probabilities to true labels, the natural measure for classification.

Figure 1: The MSE loss surface over $(w, b)$ for a sigmoid classifier: wide, nearly flat plateaus (vanishing gradients where the sigmoid saturates) around a narrow curved valley: a landscape that is awkward and slow for gradient descent, unlike the convex bowl cross-entropy gives.
Mathematical Formulation:
$\mathcal{L}_{\text{MSE}} = \frac{1}{n} \sum_{i=1}^{n} \big(y_i - \sigma(z_i)\big)^2$
$\mathcal{L}_{\text{CE}} = -\frac{1}{n} \sum_{i=1}^{n} \Big[ y_i \log \sigma(z_i) + (1 - y_i) \log \big(1 - \sigma(z_i)\big) \Big]$
$\frac{\partial \mathcal{L}_{\text{CE}}}{\partial z_i} = \sigma(z_i) - y_i$
Where:
$y_i \in \{0, 1\}$ is the true label and $\sigma(z_i)$ the predicted probability for sample $i$ , with score $z_i = \mathbf{w}^{\top}\mathbf{x}_i + b$ .
MSE’s gradient carries an extra factor $\sigma'(z_i) = \sigma(z_i)(1 - \sigma(z_i))$ , which vanishes when the sigmoid saturates, exactly when the prediction is confidently wrong.
Cross-entropy’s gradient $\sigma(z_i) - y_i$ is simply the prediction error: large when the model is confidently wrong, with no saturating factor.
Login to view more content
June 10, 2025

ML0049 Logistic Regression II

Please compare Logistic Regression and Neural Networks.

Answer

Logistic regression is a straightforward, linear model suited to linearly separable data, offering good interpretability and fast, simple training. Neural networks are powerful non-linear models capable of capturing intricate patterns in large datasets, often at the expense of interpretability and higher computational demands.

(1) Expressiveness: Logistic regression draws one hyperplane; neural networks compose hidden layers into arbitrarily curved boundaries.
(2) Workflow: Logistic regression often needs manual feature engineering; neural networks learn feature representations automatically.
(3) Cost: Logistic regression trains in seconds with low overfitting risk when features are well chosen; networks need more data, compute, and regularization.

Logistic regression drawn as a neural network with an input layer feeding one sigmoid output neuron, with no hidden layer

Figure 1: Logistic regression is a neural network: an input layer plus a single output neuron with a sigmoid activation and no hidden layer, the simplest possible network, for binary classification.

Mathematical Formulation:
$\hat{y} = \sigma(\mathbf{w}^{\top} \mathbf{x} + b)$
$\hat{y} = g\big(W_L \, f(\cdots f(W_1 \mathbf{x} + b_1) \cdots) + b_L\big)$

Where:

The first line is logistic regression: one linear score through a sigmoid $\sigma$ , equivalent to a 0-hidden-layer network.
The second line is a general neural network: compositions of weight matrices $W_l$ , biases $b_l$ , and non-linearities $f$ , with output map $g$ .
Setting the number of hidden layers to zero and $g = \sigma$ recovers logistic regression exactly.

Feature	Logistic Regression	Neural Networks (NN)
Model Type	Linear	Non-linear, multi-layer
Architecture	Single-layer (no hidden layers)	Multi-layer (can be deep)
Non-linearity	Can’t model non-linear relationships directly	Can capture complex non-linear patterns
Feature Engineering	Often needed manually	Learns feature representations automatically
Interpretability	High	Low (acts like a black box)
Training Time	Fast, simple to train	Slower, needs more compute & data
Overfitting Risk	Low (if features are well-chosen)	Higher (requires regularization)
Common Use Cases	Simple classification problems (e.g., churn prediction)	Complex tasks (e.g., image, text, speech recognition)

June 10, 2025

ML0048 Logistic Regression

Can you explain logistic regression and how it contrasts with linear regression?

Answer

Logistic regression maps inputs to a probability space for classification, while linear regression estimates continuous outcomes through a direct linear relationship. Logistic regression estimates the probability that a binary outcome ( $y = 1$ ) occurs given an input vector $\mathbf{x}$ , by passing a linear score through the sigmoid function; the output lies between 0 and 1 and is interpreted as the probability of class membership. Note that standard logistic regression is still fundamentally a linear model at its core: it cannot model non-linear relationships directly.

(1) Purpose: Linear regression predicts a continuous output (price, height); logistic regression predicts a probability for classification (spam or not).
(2) Output Range: Linear regression emits any real number (can be negative or above 1); logistic regression squashes to (0, 1) with the sigmoid.
(3) Assumption: Linear regression assumes a linear input-output relationship; logistic regression assumes a linear relationship between inputs and the log-odds of the positive class.

Left panel linear regression line fitting continuous data, right panel sigmoid curve fitting binary zero one data with a 0.5 decision boundary

Figure 1: The contrast in one picture: linear regression fits a straight line through continuous targets (left); logistic regression fits an S-shaped sigmoid to 0/1 labels (right), with the 0.5 probability level acting as the decision boundary.

Mathematical Formulation:
$\Pr(y = 1 \mid \mathbf{x}) = \frac{1}{1 + e^{-(\mathbf{w}^{\top} \mathbf{x} + b)}}$

Where:

$\mathbf{x}$ is the input feature vector.
$\mathbf{w}$ is the weight vector and $b$ the bias; the linear score $\mathbf{w}^{\top}\mathbf{x} + b$ is the log-odds of the positive class.
The sigmoid maps the score to (0, 1); classifying at threshold 0.5 gives a linear decision boundary.

Feature	Logistic Regression	Linear Regression
Type of Problem	Classification	Regression
Dependent Variable	Categorical (e.g., Yes/No, 0/1, True/False)	Continuous (e.g., Price, Temperature, Age)
Output	Probability (0 to 1)	Real number (any value)
Core Function	Sigmoid function (for binary)	Identity function
Loss Function	Log-loss (Cross-entropy)	Mean Squared Error (MSE)

June 9, 2025

DL0018 NaN Values
What are the common causes for a deep learning model to output NaN values?
Answer
NaN outputs almost always trace back to numerical instability somewhere in the pipeline: unstable math operations, exploding gradients, an oversized learning rate, bad weight initialization, or corrupted input data. Once any parameter becomes NaN or Inf, the poisoning propagates through every subsequent layer, so the model’s outputs quickly become NaN everywhere.
(1) Exploding Gradients: Gradients grow without bound through many layers, producing weight updates so large that parameters overflow to Inf/NaN.
(2) Unstable Math Operations: $\log(0)$ , division by zero, or square roots of negative numbers yield Inf/NaN. For example, batch normalization divides by zero variance if a batch is constant and no $\epsilon$ is used.
(3) Improper Learning Rate: A learning rate that is too high makes parameter updates diverge, pushing weights to extreme values that overflow.
(4) Incorrect Weight Initialization: Initializing weights to very large values can overflow activations on the very first forward pass.
(5) Data Issues: Inputs containing NaN, Inf, or extreme unnormalized values inject invalid numbers directly into the computation graph.

Figure 1: Unclipped gradients can explode within a few steps until the loss itself becomes NaN/Inf; gradient clipping keeps training on track.
NaN Loss vs NaN Outputs: The two are stages of the same instability. A NaN loss can even appear while model outputs are still finite. For example, cross-entropy computes $\log(\hat{y})$ , and a prediction extremely close to zero yields $-\infty$ , which turns into NaN after the backward pass.

Figure 2: The five common causes and their standard fixes: every path ends at NaN parameters, hence NaN outputs.
Mathematical Formulation:
$\mathcal{L} = -\log(\hat{y} + \epsilon)$
$\hat{y} \to 0 \ \Rightarrow\ \mathcal{L} \to \infty \ \Rightarrow\ \text{NaN in backprop}$
$g \leftarrow g \cdot \min\left(1,\ \frac{\tau}{\lVert g \rVert}\right)$
Where:
$\hat{y}$ is the predicted probability of the true class; without the $\epsilon$ clamp the loss diverges as $\hat{y}$ approaches 0 and turns NaN in backprop.
$g$ is the gradient vector and $\tau$ the clipping threshold; the second rule (norm clipping) caps update magnitude before gradients can overflow the parameters.
Login to view more content
May 30, 2025
DL0017 Reproducibility
How do you ensure the reproducibility of deep learning experiments?
Answer
Reproducibility means a rerun of your experiment, by you or by someone else, produces the same results. Because deep learning pipelines are full of hidden randomness (weight initialization, data shuffling, dropout, GPU nondeterminism), achieving it requires controlling randomness with fixed seeds and deterministic operations, versioning code and configurations, pinning the software environment, fixing the dataset, and logging everything about each run.
(1) Seed Control and Deterministic Operations: Fix random seeds for Python, NumPy, and your framework (PyTorch/TensorFlow), and enable deterministic algorithms while disabling autotuners that pick nondeterministic kernels.
(2) Code and Configuration Versioning: Track code in Git and store every hyperparameter in versioned config files (YAML/JSON), so a run maps to an exact commit plus config.
(3) Environment and Dependency Control: Pin library versions (requirements.txt, Conda) or containerize with Docker, recording CUDA/cuDNN and hardware details.
(4) Dataset Management: Fix train/validation/test splits, document preprocessing, and use versioned datasets (e.g., DVC) so data never silently changes.
(5) Logging and Experiment Tracking: Record seeds, configs, metrics, and artifacts for every run with tools like MLflow or Weights & Biases.
What Failure Looks Like: Without seed control, two runs of identical code and data can trace visibly different accuracy curves (sometimes differing by a point or more at convergence), making results impossible to validate or compare.

Figure 1: Same code, same data, different curves: unseeded randomness alone is enough to make experiments non-reproducible.
A Practical Pipeline: Treat reproducibility as a chain: each link below removes one source of variability, and the result is reproducible only if every link holds.

Figure 2: Identical results require identical code + environment + data + seeds. A break in any link breaks the chain.
Mathematical Formulation:
$m = F(\text{code},\ \text{config},\ \text{env},\ \text{data},\ \text{seed})$
$\text{reproducible} \iff \forall\, i, j:\ m_i = m_j$
Where:
$F$ is the full experiment pipeline viewed as a function of its five inputs; $m$ is the resulting metric set (accuracy, loss curves).
$m_i, m_j$ are metrics from any two reruns; reproducibility demands zero variance across runs, which holds only when all five inputs are fixed.
Login to view more content
May 29, 2025
ML0047 Parameters
What are the differences between parameters and hyperparameters?
Answer
Parameters are the values a model learns from its training data, while hyperparameters are settings defined by the user that guide the training process and model architecture. Parameters represent the model’s internal knowledge and decision rules; hyperparameters govern how the model learns, influencing its architecture, training dynamics, and ultimately its ability to generalize to unseen data: choosing them well is crucial for building an effective model.
(1) Parameters: Internal variables learned from data (e.g., weights and biases); adjusted during training by the optimization algorithm; capture the model’s learned patterns and information.
(2) Hyperparameters: External configurations set before training (e.g., learning rate, batch size, number of layers); remain fixed during training and are not updated by the learning process; influence how the model learns and its overall structure.
(3) Who Updates Whom: The optimizer updates parameters by minimizing the loss; the practitioner (or a search procedure) updates hyperparameters by watching validation performance.

Figure 1: The two knobs of a model: parameters (weights, biases) live inside the training loop and are updated every step by the optimizer; hyperparameters (learning rate, batch size, depth, …) are set before training and shape how that loop behaves.
Mathematical Formulation:
$\theta_{t+1} = \theta_t - \eta \, \nabla_\theta \mathcal{L}(\theta_t)$
$\lambda^* = \arg\min_{\lambda} \; \mathcal{L}_{\text{val}}\big(\hat{\theta}(\lambda)\big)$
Where:
$\theta$ denotes the parameters (weights and biases), updated every training step $t$ by gradient descent.
$\eta$ is the learning rate, itself a hyperparameter, set before training and never learned by the optimizer.
$\lambda$ stands for hyperparameters generally; $\hat{\theta}(\lambda)$ is the fully trained model under that setting, and the outer minimization over validation loss is hyperparameter search.
Login to view more content
May 29, 2025
DL0016 Learning Rate Warmup
What is learning rate warmup, and why does it help stabilize the early steps of deep network training?
Answer
Learning rate warmup starts training with a very small learning rate and increases it gradually (usually linearly) to the target peak over the first few hundred or thousand steps, after which the normal schedule (e.g., cosine decay) takes over. Its purpose is to stabilize early training: at initialization the model’s gradients are noisy and poorly conditioned, so immediately applying the full learning rate can cause loss spikes, divergence, or permanent damage to early layers.
(1) Stabilizes Early Updates: Random initializations produce unreliable gradient estimates; small early steps prevent destructive weight changes before the model finds its footing.
(2) Protects Adaptive Optimizers: With Adam, second-moment estimates are cold at step zero and can amplify the first updates; warmup bridges this biased-estimate phase.
(3) Enables Higher Peak Rates: Deep networks and Transformers tolerate (and benefit from) a higher peak learning rate once representations have settled, which warmup makes reachable without instability.

Figure 1: A typical schedule: linear warmup for 200 steps to the 0.10 peak, then cosine decay to zero, the standard recipe behind models like BERT and GPT.
Mathematical Formulation:
$\eta_t = \eta_{max} \cdot \frac{t}{T_{warm}}, \quad t \leq T_{warm}$
$p = \frac{t - T_{warm}}{T - T_{warm}}$
$\eta_t = \eta_{max} \cdot \tfrac{1}{2}\big(1 + \cos(\pi p)\big)$
Where:
$\eta_t$ is the learning rate at step $t$ ; $\eta_{max}$ is the target peak learning rate.
$T_{warm}$ is the warmup length in steps (e.g., 200–10,000 depending on model and batch size).
$T$ is the total number of training steps; after warmup, the schedule shown is cosine decay.
$p \in [0, 1]$ is the post-warmup progress used by the third line, which applies once $T_{warm}$ steps have elapsed: $p = 0$ at the end of warmup and $p = 1$ at the final step, so $\eta_t$ decays from the peak to zero.
Why It Matters in Practice: Transformers are famously sensitive: without warmup their early loss can spike or diverge outright, while warmup yields a smooth descent; the effect is smaller but still useful in CNN training at very large batch sizes.

Figure 2: Without warmup the early loss can spike and diverge; warmup keeps the fragile first steps small until gradients become trustworthy.
Login to view more content
May 27, 2025
ML0046 Forward Propagation
Please explain the process of Forward Propagation.
Answer
Forward propagation is the process by which a neural network takes an input and generates a prediction: the input data is passed systematically through each layer of the network. At every neuron, a weighted sum of the previous layer’s outputs (plus a bias) is computed, then a non-linear activation is applied; this repeats layer by layer until the output layer emits the final prediction. It is essentially the prediction phase of the network: information flows in one direction, from input to output, using the learned weights and biases.
(1) Input Layer: The network receives the raw input data.
(2) Layer-Wise Processing: Each neuron computes a linear combination (weighted sum plus bias), then applies a non-linear activation (ReLU, sigmoid, tanh) to introduce non-linearity.
(3) Propagation Through Layers: The output of one layer becomes the input to the next, progressing through all hidden layers.
(4) Output Generation: The final layer applies a task-appropriate function (softmax for classification, a linear function for regression) to produce the prediction.

Figure 1: One direction only: each layer transforms activations into pre-activations ( $z = Wx + b$ ) and back into activations ( $a = f(z)$ ) until the output layer produces the prediction. No gradient information flows here; that is backpropagation’s job.
Mathematical Formulation:
$z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$
$a^{(l)} = f\big(z^{(l)}\big)$
$\hat{y} = a^{(L)}$
Where:
$a^{(l-1)}$ is the previous layer’s activation vector ( $a^{(0)} = x$ , the input).
$W^{(l)}, b^{(l)}$ are layer $l$ ‘s learned weights and biases; $z^{(l)}$ is the pre-activation.
$f(\cdot)$ is the activation function; $L$ is the number of layers and $\hat{y}$ the prediction.

Figure 2: A worked example with concrete numbers: inputs $x = (0.5, 0.1, 0.4)$ flow through fixed weights and biases; each hidden node shows its computed $z$ and sigmoid activation $a$ , ending in the scalar prediction. Every number on the diagram follows from the two formulas above.
Login to view more content
May 27, 2025