Tag: Basics

DL0001 Residual Connection
Why are residual connections important in deep neural networks?
Answer
A residual connection (skip connection) adds a block input to a learned residual transformation, so the block learns an update rather than an entirely new mapping. For a shape-preserving block, the local Jacobian becomes $I+J_F$ , which supplies a direct identity component and creates shorter paths through the computational graph. This usually improves gradient propagation and optimization, but it does not guarantee nonzero or constant gradients: products of residual-block Jacobians can still shrink, grow, or cancel. Residual connections also address the degradation problem, in which adding layers to a plain network can increase training error because the deeper model is difficult to optimize even though an identity extension exists. These properties enabled effective training of ResNet architectures with hundreds of layers.
(1) Shorter Gradient Paths: An identity shortcut changes the local block Jacobian from $J_F$ to $I+J_F$ . The identity component gives backpropagation additional routes, although the product across many blocks can still vanish or explode.
(2) Identity Mapping Fallback: When an additional shape-preserving block is not useful, its residual branch $F_l(x_l)$ can approach zero, making $x_{l+1}\approx x_l$ easier to represent than in a plain nonlinear stack.
(3) Easier Deep Optimization: Residual parameterization helps deeper models avoid the degradation problem, where added layers increase training error because optimization fails to recover a useful identity extension.

Figure 1: A shape-preserving block uses an identity shortcut, while a block that changes resolution or channel width uses a learned projection $S_l$ so the two paths have compatible shapes.
Mathematical Formulation:
$x_{l+1}=F_l(x_l;W_l)+x_l$
$\frac{\partial x_{l+1}}{\partial x_l}=J_{F_l}+I$
$x_{l+1}=F_l(x_l;W_l)+S_lx_l$
Where:
$x_l$ is the input to residual block $l$ , and $x_{l+1}$ is its output.
$F_l(x_l;W_l)$ is the learned residual branch parameterized by weights $W_l$ ; it learns an update to the shortcut representation.
$J_{F_l}=\partial F_l/\partial x_l$ is the residual-branch Jacobian, and $I$ is the identity operator with the same feature dimension as $x_l$ .
$S_l$ is an identity operator when input and output shapes match; otherwise it can be a learned projection, such as a strided $1\times1$ convolution, that aligns spatial and channel dimensions.
Across $L$ residual blocks, backpropagation contains products of factors $I+J_{F_l}$ . These factors often improve conditioning relative to plain Jacobian products, but they do not impose a nonzero lower bound on gradient magnitude.

Figure 2: A plain stack multiplies full layer Jacobians, whereas a residual stack multiplies factors $I+J_{F_l}$ . The identity components provide additional gradient routes but do not guarantee stable magnitude.
Login to view more content
April 23, 2025
ML0031 Linear Regression
What are the advantages and disadvantages of linear regression?
Answer
Linear regression models the target as a weighted sum of the input features plus an intercept, fitting the weights by minimizing squared error. Its advantages make it the default baseline: it is simple and interpretable: each coefficient directly states the strength and direction of a feature’s relationship with the target; it is computationally cheap, with a closed-form solution (normal equations) and fast training even on large data; and it works well when the true relationship is approximately linear. Its disadvantages stem from the same simplicity: it assumes linearity, so it underfits genuinely non-linear relationships; it is sensitive to outliers because squared errors let extreme points dominate the fit; multicollinearity between correlated features makes coefficient estimates unstable and hard to interpret; and the model cannot capture interactions or complexity unless you explicitly engineer such features. Simple linear regression uses one feature; multiple linear regression extends the same idea to many.
(1) Interpretable & Fast: Coefficients read directly as feature effects; closed-form or cheap iterative fitting.
(2) Linearity Assumption: Underfits non-linear patterns: the model class is a hyperplane.
(3) Fragilities: Outlier-sensitive (squared loss), unstable under multicollinearity, no built-in interactions or nonlinearity.

Figure 1: Least-squares fit: the line minimizes the sum of squared vertical residuals (orange segments). Note how the one distant outlier pulls the fitted line toward itself, the sensitivity that comes with squaring errors.
Mathematical Formulation:
$h_\theta(x) = \theta_0 + \sum_{j=1}^{p} \theta_j x_j = \theta^{\top} x$
$\hat{\theta} = \arg\min_{\theta} \sum_{i=1}^{n} \big(y_i - h_\theta(x_i)\big)^2 = (X^{\top}X)^{-1} X^{\top} y$
Where:
$h_\theta(x)$ is the predicted value for feature vector $x$ .
$\theta_0$ is the intercept (bias); $\theta_j$ the weight of feature $x_j$ ; $p$ the number of features.
$X$ is the $n \times (p+1)$ design matrix and $y$ the target vector; the closed form exists when $X^{\top}X$ is invertible (no perfect multicollinearity).
Login to view more content
April 18, 2025
ML0030 Sigmoid
What are the advantages and disadvantages of using a sigmoid activation function?
Answer
The sigmoid activation squashes any real input into the range $(0, 1)$ with a smooth S-curve. Its advantages: it is smooth and differentiable everywhere, and its bounded output has a natural probability interpretation, which keeps it the standard output activation for binary classification and for independent per-label probabilities in multi-label tasks (and for gates in LSTM/GRU cells, which need values in $(0,1)$ ). Its disadvantages drove it out of hidden layers: in the tails the function saturates and its derivative (at most 0.25, near zero for most inputs) makes stacked sigmoid layers a prime cause of the vanishing gradient problem; its outputs are not zero-centered, so downstream weight updates are biased in one direction and convergence slows; and the exponential computation is more expensive than ReLU’s simple threshold.
(1) Probability Output: Range $(0,1)$ , ideal for binary classification output layers and gating mechanisms.
(2) Vanishing Gradient: Derivative peaks at 0.25 and saturates in the tails; deep stacks lose gradient fast.
(3) Not Zero-Centered: Always-positive outputs bias weight updates and slow convergence; also costlier to compute than ReLU.

Figure 1: Sigmoid and its derivative $\sigma(x)(1-\sigma(x))$ : a useful probability-shaped output, but the derivative only reaches 0.25 at best and is near zero in both tails: the mechanism behind vanishing gradients.
Mathematical Formulation:
$\sigma(x) = \frac{1}{1 + e^{-x}}$
$\sigma'(x) = \sigma(x)\,\big(1 - \sigma(x)\big) \leq 0.25$
Where:
$x$ is the neuron’s pre-activation (weighted sum plus bias).
$\sigma'(x)$ is the derivative, computed in one line from the output itself, maximal (0.25) at $x = 0$ , approaching 0 as $|x|$ grows.
Login to view more content
April 18, 2025
ML0029 Tanh
What are the advantages and disadvantages of using the tanh activation function?
Answer
The tanh (hyperbolic tangent) activation squashes its input into the range $(-1, 1)$ with a smooth S-shape. Its headline advantage over sigmoid is that its output is zero-centered: activations average near zero, so the gradients flowing into the next layer do not systematically push all weights in one direction, which makes optimization easier and convergence faster. It is also smooth and infinitely differentiable, and its maximum derivative of 1 (versus sigmoid’s 0.25) gives stronger gradients near the active region. The disadvantages are the flip side of saturation: for large positive or negative inputs the function flattens and its derivative approaches zero, so deep stacks of tanh units suffer the vanishing gradient problem; and like sigmoid it requires exponentials, making it computationally more expensive than ReLU. Because its output is not in $(0, 1)$ , it is also unsuitable as a probability output; its modern niche is mainly inside RNN/LSTM cell dynamics rather than feedforward hidden layers.
(1) Zero-Centered: Output range $(-1, 1)$ centers activations, easing optimization versus sigmoid’s all-positive outputs.
(2) Stronger Gradient: Maximum derivative 1 at the origin (4× sigmoid’s), but it still saturates in the tails.
(3) Costs: Vanishing gradients in deep stacks, exponential computation, and no probability interpretation of the output.

Figure 1: Tanh and its derivative $1 - \tanh^2(x)$ : the zero-centered S-curve has a healthy gradient (peak 1) only in the narrow central region; in both tails the derivative collapses toward zero.
Mathematical Formulation:
$\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} = 2\,\sigma(2x) - 1$
$\frac{d}{dx}\tanh(x) = 1 - \tanh^2(x) \in (0, 1]$
Where:
$x$ is the neuron’s pre-activation; $\sigma(\cdot)$ is the logistic sigmoid, showing tanh is just a rescaled sigmoid.
$1 - \tanh^2(x)$ is the derivative: maximal (1) at $x = 0$ , decaying toward 0 as $|x|$ grows, the saturation that drives vanishing gradients.
Login to view more content
April 17, 2025
ML0028 Softmax
What is the Softmax activation function, and what is its purpose?
Answer
Softmax is the activation used in the output layer for multi-class classification: it converts a vector of raw scores (logits) into a normalized probability distribution over the classes. Each output is the exponentiated logit divided by the sum of exponentiated logits, so every output lies in $(0, 1)$ and all outputs sum to exactly 1, a genuine probability distribution that can be read as the model’s confidence in each class. Softmax assumes classes are mutually exclusive (one true class per sample); for multi-label problems where an input can belong to several classes at once, per-class sigmoid outputs are used instead, since each class needs an independent probability. Paired with cross-entropy loss, softmax gives the clean gradient $p - y$ , which is why the combination is the standard for multi-class training.
(1) Purpose: Turn arbitrary logits into a probability distribution: outputs in $(0,1)$ , summing to 1.
(2) Amplification: Exponentiation sharpens differences: the largest logit dominates the distribution.
(3) Scope: Multi-class (mutually exclusive) problems with cross-entropy loss; multi-label problems use independent sigmoids instead.

Figure 1: Softmax in action: raw logits (top) of arbitrary scale and sign are exponentiated and normalized into a probability distribution (bottom) that sums to 1; the relative order is preserved but differences are amplified.
Mathematical Formulation:
$\mathrm{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$
$\sum_{i=1}^{K} \mathrm{Softmax}(z_i) = 1$
$\mathrm{Softmax}(z_i / T) = \frac{e^{z_i / T}}{\sum_{j=1}^{K} e^{z_j / T}} \quad \text{(with temperature } T \text{)}$
Where:
$z_i$ is the raw score (logit) for class $i$ , and $K$ the number of classes.
The exponential makes every term positive; dividing by the total normalizes the vector to sum 1.
$T$ is an optional temperature: a value below 1 sharpens the distribution, $T > 1$ softens it (used in distillation and sampling).

Figure 2: Temperature controls sharpness: low temperature pushes probability mass onto the top class (hard distribution), high temperature flattens toward uniform: the same logits, three very different distributions.
Login to view more content
April 12, 2025

ML0027 Leaky ReLU

What are the benefits of the Leaky ReLU activation function?

Answer

Leaky ReLU modifies standard ReLU by replacing the hard zero on the negative side with a small linear slope: negative inputs pass through scaled by a small constant $\alpha$ (typically 0.01). This one change directly attacks ReLU’s main weakness: the dying ReLU problem. Because the negative region now carries a small but non-zero gradient, a neuron whose pre-activation goes negative for all inputs still receives learning signal and can be pulled back into the active regime, instead of being frozen at zero output forever. At the same time Leaky ReLU retains everything that made ReLU attractive: the positive side stays the identity with gradient 1 (no vanishing gradients), the computation is still a trivial piecewise-linear threshold, and the output remains unbounded above, preserving ReLU’s scale behavior. In practice the accuracy gain over ReLU is often modest, but it costs nothing and removes a permanent failure mode.

(1) Fixes Dying ReLU: Negative inputs get slope $\alpha$ instead of 0, so gradient always flows and “dead” neurons can recover.
(2) Keeps ReLU’s Strengths: Identity on the positive side (gradient 1, no saturation) and near-identical computational cost.
(3) Costs: Introduces the hyperparameter $\alpha$ (or learns it, as in PReLU); practical accuracy gains over ReLU are often small.

ReLU versus Leaky ReLU curves with the small negative slope highlighted

Figure 1: ReLU versus Leaky ReLU: identical on the positive side, but Leaky ReLU keeps a small slope $\alpha$ in the negative region, enough gradient for a stuck neuron to recover instead of dying.

Mathematical Formulation:
$\mathrm{LeakyReLU}(x) = x \text{ for } x \geq 0$
$\mathrm{LeakyReLU}(x) = \alpha x \text{ otherwise}$
$\alpha \approx 0.01$
$\mathrm{LeakyReLU}'(x) = 1 \text{ for } x > 0$
$\mathrm{LeakyReLU}'(x) = \alpha \text{ otherwise}$

Where:

$x$ is the neuron’s pre-activation (weighted sum plus bias).
$\alpha$ is the negative-side slope, a fixed small constant (0.01 by default) or a learned parameter in PReLU.

Feature	ReLU	Leaky ReLU
Negative Input	Output is 0	Output is a small non-zero value (αx)
Gradient for x<0	0	α (small positive constant)
Dying ReLU Problem	Susceptible	Less susceptible
Zero-Centered Output	No	No (but closer than ReLU)
Computational Cost	Slightly lower	Slightly higher

Table 1: ReLU versus Leaky ReLU: the functions differ only in the negative region, but that small slope is what keeps neurons alive and gradients flowing.

April 12, 2025

ML0026 ReLU
What are the benefits and limitations of the ReLU activation function?
Answer
ReLU (Rectified Linear Unit) is the piecewise-linear activation $\max(0, x)$ : it passes positive inputs unchanged and blocks negative ones to zero. Its benefits made it the default hidden-layer activation of deep learning: in the positive region the gradient is a constant 1, which largely eliminates the vanishing gradient problem that saturating sigmoid/tanh units suffer; negative outputs produce sparse activations, so only a subset of neurons is active at any time, yielding efficient and often more robust representations; and the function is a trivial threshold operation, far cheaper to compute than exponentials. Its main limitation is the dying ReLU problem: a neuron that falls into a regime of consistently negative pre-activations outputs zero and has zero gradient, so it can never recover, permanently shrinking model capacity. ReLU is also unbounded on the positive side (large activations can destabilize training if unmanaged) and non-differentiable at zero, a theoretical wrinkle handled in practice by defining a subgradient of 0 there.
(1) Gradient Health: Derivative is 1 for $x > 0$ : no saturation, strong gradient flow in deep networks.
(2) Efficiency & Sparsity: A single comparison computes it; zero outputs give sparse, efficient representations.
(3) Dying ReLU: Neurons stuck in the negative region output 0 with 0 gradient and may never recover; output is also unbounded above.

Figure 1: ReLU and its derivative: the identity ramp on the positive side keeps gradients at 1 (no vanishing), while the flat zero region on the left kills both signal and gradient, the source of the dying-ReLU failure mode.
Mathematical Formulation:
$\mathrm{ReLU}(x) = \max(0, x)$
$\mathrm{ReLU}'(x) = 1 \text{ for } x > 0, \quad 0 \text{ for } x \leq 0$
Where:
$x$ is the neuron’s pre-activation (weighted sum plus bias).
$\mathrm{ReLU}'(x)$ is the derivative used in backpropagation; at $x = 0$ the function is non-differentiable and implementations assign subgradient 0.
Login to view more content
April 12, 2025
ML0025 Exploding Gradient
What are the typical reasons for exploding gradient?
Answer
Exploding gradients occur when gradients grow exponentially during backpropagation, producing huge weight updates that make training unstable: the loss oscillates, spikes, or diverges to NaN. The mechanism mirrors the vanishing problem: the gradient at an early layer is a chain-rule product of per-layer factors, and when those factors are consistently larger than 1 in magnitude (from poorly scaled weight initialization, deep unnormalized architectures, or activation regimes with derivatives above 1), the product blows up with depth. Recurrent networks are especially vulnerable because the same weight matrix is multiplied once per time step, so long sequences amplify the effect. A learning rate set too high then converts already-large gradients into catastrophic updates.
(1) Deep Chains of Large Factors: Products of weight matrices with spectral norm $> 1$ grow exponentially with depth (or RNN time steps).
(2) Bad Initialization: Weights initialized too large produce outsized activations and derivatives from the start.
(3) Compounding Learning Rate: A high learning rate turns large gradients into weight updates that overshoot and destabilize training.

Figure 1: Gradient norm flowing backward through a deep network: without control it grows geometrically layer by layer (note the log scale); gradient clipping caps the norm at a fixed threshold, keeping updates bounded regardless of depth.
Mathematical Formulation:
$\frac{\partial L}{\partial z_l} = \frac{\partial L}{\partial z_n} \prod_{i=l}^{n-1} W_{i+1} \, f'(z_{i+1})$
$\left\| \prod_i W_i \right\| \sim \prod_i \|W_i\|$
$g \leftarrow g \cdot \min\!\left(1,\; \frac{\tau}{\|g\|}\right) \quad \text{(gradient clipping to threshold } \tau \text{)}$
Where:
$z_l$ is the pre-activation of layer $l$ ; $W_i$ and $f'(z_i)$ are the per-layer weight and activation-derivative factors.
$\|W_i\|$ is the operator (spectral) norm of the weight matrix; when the typical product exceeds 1, the gradient norm grows geometrically with depth.
$g$ is the full gradient vector and $\tau$ the clipping threshold: if $\|g\|$ exceeds $\tau$ , the gradient is rescaled down to norm $\tau$ without changing its direction.
Login to view more content
April 4, 2025
ML0024 Vanishing Gradient
What are the typical reasons for vanishing gradient?
Answer
The vanishing gradient problem occurs when gradients shrink exponentially as they are backpropagated from the output layer toward the early layers of a deep network, so early layers receive almost no learning signal and train extremely slowly. The root cause is the chain rule: the gradient at an early layer is a product of many per-layer factors (weight matrices and activation derivatives), and if those factors are consistently smaller than 1 in magnitude, the product collapses toward zero as depth grows. The classic driver is saturating activation functions (sigmoid’s derivative peaks at only 0.25 and tanh’s at 1, and both are near zero for most of their input range), compounding with poor weight initialization that pushes pre-activations into the saturated tails. Recurrent networks suffer the same effect across time steps.
(1) Saturating Activations: Sigmoid/tanh derivatives are small almost everywhere (sigmoid $\sigma'(z) \leq 0.25$ ); multiplying them across layers shrinks gradients exponentially.
(2) Depth / Chain Rule: The gradient at layer 1 is a product of $n$ per-layer factors; each factor below 1 makes the product vanish as $n$ grows.
(3) Poor Initialization: Too-large or too-small initial weights push activations into saturation, shrinking derivatives from the start.

Figure 1: Gradient magnitude at each layer during backpropagation (log scale): with sigmoid activations the signal decays roughly geometrically: after 20 layers the earliest layers receive gradients orders of magnitude smaller than the output layer.
Mathematical Formulation:
$\frac{\partial L}{\partial z_l} = \frac{\partial L}{\partial z_{n}} \prod_{i=l}^{n-1} W_{i+1} \, f'(z_{i+1})$
$\sigma(z) = \frac{1}{1 + e^{-z}}$
$\sigma'(z) = \sigma(z)\big(1 - \sigma(z)\big) \leq 0.25$
Where:
$z_l$ is the pre-activation of layer $l$ , and $n$ the total number of layers.
$W_{i+1}$ is the weight matrix of layer $i+1$ and $f'(\cdot)$ the activation derivative, the two per-layer factors in the chain product.
$\sigma(z)$ is the sigmoid; its derivative peaks at 0.25 near $z = 0$ and approaches 0 in the saturated tails.

Figure 2: Why sigmoid kills gradients: outside the narrow active region around zero the derivative is essentially zero, so any neuron operating in the flat tails passes almost no gradient backward, and these small factors multiply down the chain.
Login to view more content
April 4, 2025
ML0023 Gradient Descent
What is Gradient Descent in machine learning?
Answer
Gradient descent is the iterative first-order optimization algorithm used to minimize a loss function by repeatedly stepping in the direction of steepest descent, opposite to the gradient. In each iteration the algorithm computes the gradient of the loss with respect to every parameter, then updates the parameters by subtracting a fraction of that gradient, where the fraction is the learning rate. The procedure repeats until the updates become negligibly small; for convex losses this converges to the global minimum, and for the non-convex losses of deep networks it reliably finds good local minima. In practice three variants trade gradient accuracy against computation: batch gradient descent uses the full dataset per step (stable but expensive), stochastic gradient descent uses a single sample (fast but noisy), and mini-batch gradient descent uses small subsets and is the default in modern training.
(1) Update Rule: $\theta \leftarrow \theta - \alpha \nabla_\theta J(\theta)$ : move against the gradient, scaled by the learning rate.
(2) Learning Rate: Too large diverges or oscillates; too small converges slowly; it is the single most important hyperparameter.
(3) Variants: Batch (full data, stable), stochastic (one sample, noisy), mini-batch (compromise used in practice).

Figure 1: Gradient descent on a 1-D quadratic loss: each step moves opposite to the local slope, with step size proportional to the gradient: large steps far from the minimum, automatically shrinking steps near it.
Mathematical Formulation:
$\theta_{t+1} = \theta_t - \alpha \, \nabla_\theta J(\theta_t)$
$\nabla_\theta J(\theta) = \begin{bmatrix} \frac{\partial J}{\partial \theta_1} & \frac{\partial J}{\partial \theta_2} & \cdots & \frac{\partial J}{\partial \theta_p} \end{bmatrix}^{\top}$
Where:
$\theta$ is the parameter vector being optimized (e.g., all network weights and biases).
$\alpha$ is the learning rate, the step-size fraction applied to the gradient at each iteration.
$J(\theta)$ is the loss (cost) function, and $\nabla_\theta J$ its gradient: the vector of partial derivatives pointing in the direction of steepest ascent.
Login to view more content
April 4, 2025