Category: Easy

ML0030 Sigmoid
What are the advantages and disadvantages of using a sigmoid activation function?
Answer
The sigmoid activation squashes any real input into the range $(0, 1)$ with a smooth S-curve. Its advantages: it is smooth and differentiable everywhere, and its bounded output has a natural probability interpretation, which keeps it the standard output activation for binary classification and for independent per-label probabilities in multi-label tasks (and for gates in LSTM/GRU cells, which need values in $(0,1)$ ). Its disadvantages drove it out of hidden layers: in the tails the function saturates and its derivative (at most 0.25, near zero for most inputs) makes stacked sigmoid layers a prime cause of the vanishing gradient problem; its outputs are not zero-centered, so downstream weight updates are biased in one direction and convergence slows; and the exponential computation is more expensive than ReLU’s simple threshold.
(1) Probability Output: Range $(0,1)$ , ideal for binary classification output layers and gating mechanisms.
(2) Vanishing Gradient: Derivative peaks at 0.25 and saturates in the tails; deep stacks lose gradient fast.
(3) Not Zero-Centered: Always-positive outputs bias weight updates and slow convergence; also costlier to compute than ReLU.

Figure 1: Sigmoid and its derivative $\sigma(x)(1-\sigma(x))$ : a useful probability-shaped output, but the derivative only reaches 0.25 at best and is near zero in both tails: the mechanism behind vanishing gradients.
Mathematical Formulation:
$\sigma(x) = \frac{1}{1 + e^{-x}}$
$\sigma'(x) = \sigma(x)\,\big(1 - \sigma(x)\big) \leq 0.25$
Where:
$x$ is the neuron’s pre-activation (weighted sum plus bias).
$\sigma'(x)$ is the derivative, computed in one line from the output itself, maximal (0.25) at $x = 0$ , approaching 0 as $|x|$ grows.
Login to view more content
April 18, 2025
ML0028 Softmax
What is the Softmax activation function, and what is its purpose?
Answer
Softmax is the activation used in the output layer for multi-class classification: it converts a vector of raw scores (logits) into a normalized probability distribution over the classes. Each output is the exponentiated logit divided by the sum of exponentiated logits, so every output lies in $(0, 1)$ and all outputs sum to exactly 1, a genuine probability distribution that can be read as the model’s confidence in each class. Softmax assumes classes are mutually exclusive (one true class per sample); for multi-label problems where an input can belong to several classes at once, per-class sigmoid outputs are used instead, since each class needs an independent probability. Paired with cross-entropy loss, softmax gives the clean gradient $p - y$ , which is why the combination is the standard for multi-class training.
(1) Purpose: Turn arbitrary logits into a probability distribution: outputs in $(0,1)$ , summing to 1.
(2) Amplification: Exponentiation sharpens differences: the largest logit dominates the distribution.
(3) Scope: Multi-class (mutually exclusive) problems with cross-entropy loss; multi-label problems use independent sigmoids instead.

Figure 1: Softmax in action: raw logits (top) of arbitrary scale and sign are exponentiated and normalized into a probability distribution (bottom) that sums to 1; the relative order is preserved but differences are amplified.
Mathematical Formulation:
$\mathrm{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}$
$\sum_{i=1}^{K} \mathrm{Softmax}(z_i) = 1$
$\mathrm{Softmax}(z_i / T) = \frac{e^{z_i / T}}{\sum_{j=1}^{K} e^{z_j / T}} \quad \text{(with temperature } T \text{)}$
Where:
$z_i$ is the raw score (logit) for class $i$ , and $K$ the number of classes.
The exponential makes every term positive; dividing by the total normalizes the vector to sum 1.
$T$ is an optional temperature: a value below 1 sharpens the distribution, $T > 1$ softens it (used in distillation and sampling).

Figure 2: Temperature controls sharpness: low temperature pushes probability mass onto the top class (hard distribution), high temperature flattens toward uniform: the same logits, three very different distributions.
Login to view more content
April 12, 2025
ML0026 ReLU
What are the benefits and limitations of the ReLU activation function?
Answer
ReLU (Rectified Linear Unit) is the piecewise-linear activation $\max(0, x)$ : it passes positive inputs unchanged and blocks negative ones to zero. Its benefits made it the default hidden-layer activation of deep learning: in the positive region the gradient is a constant 1, which largely eliminates the vanishing gradient problem that saturating sigmoid/tanh units suffer; negative outputs produce sparse activations, so only a subset of neurons is active at any time, yielding efficient and often more robust representations; and the function is a trivial threshold operation, far cheaper to compute than exponentials. Its main limitation is the dying ReLU problem: a neuron that falls into a regime of consistently negative pre-activations outputs zero and has zero gradient, so it can never recover, permanently shrinking model capacity. ReLU is also unbounded on the positive side (large activations can destabilize training if unmanaged) and non-differentiable at zero, a theoretical wrinkle handled in practice by defining a subgradient of 0 there.
(1) Gradient Health: Derivative is 1 for $x > 0$ : no saturation, strong gradient flow in deep networks.
(2) Efficiency & Sparsity: A single comparison computes it; zero outputs give sparse, efficient representations.
(3) Dying ReLU: Neurons stuck in the negative region output 0 with 0 gradient and may never recover; output is also unbounded above.

Figure 1: ReLU and its derivative: the identity ramp on the positive side keeps gradients at 1 (no vanishing), while the flat zero region on the left kills both signal and gradient, the source of the dying-ReLU failure mode.
Mathematical Formulation:
$\mathrm{ReLU}(x) = \max(0, x)$
$\mathrm{ReLU}'(x) = 1 \text{ for } x > 0, \quad 0 \text{ for } x \leq 0$
Where:
$x$ is the neuron’s pre-activation (weighted sum plus bias).
$\mathrm{ReLU}'(x)$ is the derivative used in backpropagation; at $x = 0$ the function is non-differentiable and implementations assign subgradient 0.
Login to view more content
April 12, 2025
ML0023 Gradient Descent
What is Gradient Descent in machine learning?
Answer
Gradient descent is the iterative first-order optimization algorithm used to minimize a loss function by repeatedly stepping in the direction of steepest descent, opposite to the gradient. In each iteration the algorithm computes the gradient of the loss with respect to every parameter, then updates the parameters by subtracting a fraction of that gradient, where the fraction is the learning rate. The procedure repeats until the updates become negligibly small; for convex losses this converges to the global minimum, and for the non-convex losses of deep networks it reliably finds good local minima. In practice three variants trade gradient accuracy against computation: batch gradient descent uses the full dataset per step (stable but expensive), stochastic gradient descent uses a single sample (fast but noisy), and mini-batch gradient descent uses small subsets and is the default in modern training.
(1) Update Rule: $\theta \leftarrow \theta - \alpha \nabla_\theta J(\theta)$ : move against the gradient, scaled by the learning rate.
(2) Learning Rate: Too large diverges or oscillates; too small converges slowly; it is the single most important hyperparameter.
(3) Variants: Batch (full data, stable), stochastic (one sample, noisy), mini-batch (compromise used in practice).

Figure 1: Gradient descent on a 1-D quadratic loss: each step moves opposite to the local slope, with step size proportional to the gradient: large steps far from the minimum, automatically shrinking steps near it.
Mathematical Formulation:
$\theta_{t+1} = \theta_t - \alpha \, \nabla_\theta J(\theta_t)$
$\nabla_\theta J(\theta) = \begin{bmatrix} \frac{\partial J}{\partial \theta_1} & \frac{\partial J}{\partial \theta_2} & \cdots & \frac{\partial J}{\partial \theta_p} \end{bmatrix}^{\top}$
Where:
$\theta$ is the parameter vector being optimized (e.g., all network weights and biases).
$\alpha$ is the learning rate, the step-size fraction applied to the gradient at each iteration.
$J(\theta)$ is the loss (cost) function, and $\nabla_\theta J$ its gradient: the vector of partial derivatives pointing in the direction of steepest ascent.
Login to view more content
April 4, 2025
ML0022 Cross Entropy Loss
Explain how Cross Entropy Loss is used for a classification task.
Answer
Cross-entropy loss (log loss) measures the distance between two probability distributions: the model’s predicted class probabilities and the true distribution (a one-hot vector for labeled data). Training a classifier means minimizing this distance, which is exactly maximum-likelihood estimation of the model parameters. In the binary case the loss averages $-[y \log p + (1-y)\log(1-p)]$ ; in the multi-class case only the true class term survives the one-hot sum, reducing the per-sample loss to $-\log p_{\text{true}}$ . The logarithm is the key design choice: it penalizes confident wrong predictions severely: as the probability assigned to the true class approaches 0 the loss explodes toward infinity, giving a strong, well-behaved gradient exactly where the model errs most. It pairs naturally with sigmoid (binary) or softmax (multi-class) output layers, whose gradients under cross-entropy take the clean form $p - y$ .
(1) Definition: Negative log-likelihood of the true labels under the predicted distribution; minimizing it is maximum-likelihood training.
(2) Confidence Sensitivity: Loss grows unboundedly as the true-class probability approaches 0; confident mistakes are punished hardest.
(3) Standard Pairing: Sigmoid output + binary cross-entropy, softmax output + categorical cross-entropy; combined gradient simplifies to $p - y$ .

Figure 1: Per-sample loss $-\log(p)$ as a function of the probability assigned to the true class: near $p = 1$ the loss is almost zero, below $p = 0.1$ it grows steeply: confident errors dominate the gradient.
Mathematical Formulation:
$\mathcal{L}_{\text{binary}} = -\frac{1}{n} \sum_{i=1}^{n} \Big[ y_i \log p_i + (1 - y_i) \log(1 - p_i) \Big]$
$\mathcal{L}_{\text{multi}} = -\frac{1}{n} \sum_{i=1}^{n} \sum_{c=1}^{C} y_{ic} \log p_{ic} = -\frac{1}{n} \sum_{i=1}^{n} \log p_{i,\text{true}}$
Where:
$n$ is the number of samples and $C$ the number of classes.
$y_i \in \{0, 1\}$ is the true binary label; $p_i$ the predicted probability of the positive class.
$y_{ic}$ is the one-hot indicator (1 if sample $i$ belongs to class $c$ ); $p_{ic}$ the predicted probability of class $c$ , so the inner sum keeps only the true-class term.
Login to view more content
March 31, 2025

ML0021 L1 Loss L2 Loss

What are the key differences between L1 loss and L2 loss?

Answer

L1 loss (mean absolute error) measures the average absolute difference between predictions and targets, while L2 loss (mean squared error) measures the average squared difference. That single change in the error function drives every practical difference: squaring amplifies large deviations, so L2 is more sensitive to outliers but enjoys a smooth gradient that shrinks to zero at the optimum; L1 treats all errors linearly, making it robust to outliers, but its gradient is a constant $\pm 1$ everywhere, so optimization can oscillate near the minimum instead of settling. When used as a regularization penalty rather than a regression loss, L1 additionally induces sparsity: it can drive the weights of uninformative features to exactly zero, performing implicit feature selection, whereas L2 only shrinks weights toward zero.

(1) Error Measure: L1 averages absolute errors $|e|$ ; L2 averages squared errors $e^2$ , amplifying large deviations.
(2) Gradient Behavior: L1’s gradient is a constant $\pm 1$ (undefined at 0); L2’s gradient is proportional to the error and vanishes smoothly at the optimum.
(3) Practical Choice: L1 for outlier-robust regression and sparse models; L2 for smooth, stable optimization when Gaussian noise is a reasonable assumption.

Feature	L1 Loss (MAE)	L2 Loss (MSE)
Error Calculation	Absolute difference	Squared difference
Outlier Sensitivity	Less sensitive	More sensitive
Gradient	Constant (+1 or -1)	Proportional to the error
Sparsity	Induces sparsity (feature selection)	Does not inherently induce sparsity
Optimization near minimum	Can be unstable	More stable

Table 1: The five practical differences between L1 and L2 loss; every row follows from the choice of $|e|$ versus $e^2$ as the per-sample penalty.

L1 absolute loss V-shape versus L2 squared loss parabola as functions of the residual

Figure 1: Per-sample penalty as a function of the residual: L1 grows linearly (V-shape, robust to large errors), L2 grows quadratically (small errors are nearly free, large errors are heavily punished).

Mathematical Formulation:
$\mathcal{L}_{L1} = \frac{1}{n} \sum_{i=1}^{n} \left| \hat{y}_i - y_i \right|$
$\frac{\partial \mathcal{L}_{L1}}{\partial \hat{y}_i} = \frac{1}{n}\,\mathrm{sign}(\hat{y}_i - y_i)$
$\mathcal{L}_{L2} = \frac{1}{n} \sum_{i=1}^{n} \left( \hat{y}_i - y_i \right)^2$
$\frac{\partial \mathcal{L}_{L2}}{\partial \hat{y}_i} = \frac{2}{n}\left(\hat{y}_i - y_i\right)$

Where:

$y_i$ is the true target and $\hat{y}_i$ the model prediction for sample $i$ .
$n$ is the number of samples; both losses average over the dataset.
$\mathrm{sign}(\cdot)$ is the sign function: $+1$ for positive residuals, $-1$ for negative, undefined at 0 (subgradient $[-1, 1]$ in practice).

Gradient of L1 loss constant step function versus gradient of L2 loss linear in the residual

Figure 2: Gradient magnitude versus residual: L1’s constant gradient never decays (unstable near the optimum, undefined at zero), while L2’s gradient shrinks linearly and vanishes exactly at the optimum.

March 29, 2025

ML0020 Data Split
How to split the dataset?
Answer
A dataset is typically split into three parts. The training set is used to fit the model: it learns the patterns and relationships from this data. The validation set is used during development to tune hyperparameters and compare model configurations, which prevents overfitting the training data. The test set is touched only once, for a final unbiased evaluation on completely unseen data: an estimate of real-world generalization. Typical ratios scale with dataset size: for small datasets (fewer than ~1,000 samples), 60–70% training / 10–15% validation / 15–25% test, with k-fold cross-validation strongly recommended because small validation estimates are noisy; for medium datasets (1,000–100,000), a common starting point is 70–80% / 10–15% / 10–15%; for large datasets (over ~100,000), even 98% / 1% / 1% leaves plenty of validation and test samples. For imbalanced data, use stratified splits so every part keeps the original class proportions.
(1) Three Roles: Train fits, validation tunes, test judges; each set answers a different question.
(2) Size-Dependent Ratios: Small data needs more training share plus cross-validation; big data can spare 1–2% for evaluation.
(3) Stratification: Preserve class ratios in every split when classes are imbalanced.

Figure 1: Split ratios by dataset size: the smaller the data, the larger the training share (and the more you need cross-validation); big data can evaluate on 1–2%.
Mathematical Formulation:
$D = D_{\mathrm{train}} \cup D_{\mathrm{val}} \cup D_{\mathrm{test}}$
$D_{\mathrm{train}} \cap D_{\mathrm{val}} = D_{\mathrm{train}} \cap D_{\mathrm{test}} = D_{\mathrm{val}} \cap D_{\mathrm{test}} = \emptyset$
Where:
$D$ is the full dataset, partitioned into three disjoint subsets: no sample may appear in two roles.
$D_{\mathrm{train}}$ is the training set used to fit the model parameters.
$D_{\mathrm{val}}$ is the validation set used to tune hyperparameters and trigger early stopping.
$D_{\mathrm{test}}$ is the test set, used exactly once for the final unbiased estimate; reusing it for tuning silently turns it into validation data.
Login to view more content
March 28, 2025
ML0018 Data Normalization
Why is data normalization used in Machine Learning?
Answer
Data normalization scales features into a specific range or distribution: typically [0, 1] with min-max scaling, or mean 0 and standard deviation 1 with standardization, so that every feature contributes comparably to learning. It is used for three reasons. First, better model performance: algorithms built on distance metrics, such as k-nearest neighbors and SVMs, are skewed when one feature spans thousands and another spans decimals, normalization puts them on the same scale. Second, training stability and speed: normalized inputs keep neural network gradients well-conditioned, so training converges faster and more smoothly. Third, fair feature contribution: without scaling, large-magnitude features dominate the objective while small-magnitude ones are ignored, regardless of how informative they actually are.
(1) What It Does: Rescales each feature to a common range (min-max) or distribution (z-score standardization).
(2) Who Needs It: Distance-based models (kNN, SVM), gradient-based learners (neural networks, linear/logistic regression); tree ensembles largely do not.
(3) The Payoff: Faster convergence, stabler training, and features competing on information rather than magnitude.

Figure 1: Before: one axis spans hundreds while the other spans decimals, so distance is decided by the big axis alone. After standardization, both features contribute equally.
Mathematical Formulation:
$x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$
$z = \frac{x - \mu}{\sigma}$
Where:
$x'$ is the min-max normalized value of feature value $x$ , squashed into [0, 1].
$x_{\min}$ and $x_{\max}$ are the feature’s minimum and maximum computed on the training set only.
$z$ is the standardized (z-score) value with mean 0 and standard deviation 1.
$\mu$ and $\sigma$ are the feature’s training-set mean and standard deviation; fitting them on all data would leak test information.
Login to view more content
March 28, 2025
ML0017 Data Augmentation
What are the common data augmentation techniques?
Answer
Data augmentation increases the diversity and effective size of a training set by creating modified versions of the existing data, especially valuable in computer vision and NLP, where collecting and labeling new data is expensive. In computer vision, common techniques are geometric transformations (rotate, flip, crop, scale), color adjustments (brightness, contrast, saturation, color jitter), and noise injection (random noise or blur). In NLP: synonym replacement, back translation (translate to another language and back), and random insertion or deletion of words. For tabular data: SMOTE-style synthetic sample generation and small random noise on numeric features. The benefits: better robustness and generalization, less overfitting (the model cannot memorize a moving target), relief for class imbalance, and lower data-collection cost.
(1) Core Idea: Apply label-preserving transformations so one sample teaches many variations of the same concept.
(2) By Data Type: Geometric/color/noise for images, synonym/back-translation/edit for text, SMOTE/noise for tables.
(3) Benefits: Robustness, regularization against overfitting, imbalance relief, and cost savings, all from data you already have.

Figure 1: One sample, five augmented views. The label never changes (only the appearance does), so the model learns invariance instead of memorizing pixels.
Mathematical Formulation:
$\tilde{x} = T(x; \phi), \quad \tilde{y} = y$
$\min_\theta \; \mathbb{E}_{(x,y)} \, \mathbb{E}_{\phi} \, \ell\big(f(T(x;\phi);\theta), y\big)$
Where:
$T(x;\phi)$ is the augmentation transform applied to sample $x$ with random parameters $\phi$ (e.g., rotation angle, crop offset).
$\tilde{x}$ is the augmented sample and $\tilde{y}$ its label, unchanged because the transforms are label-preserving.
$f(\cdot;\theta)$ is the model with parameters $\theta$ , and $\ell$ is the training loss.
$\mathbb{E}_{\phi}$ is the expectation over random augmentations: training minimizes loss over infinitely many variants, which is what regularizes the model.
Login to view more content
March 28, 2025
ML0016 AUC
What is AUC?
Answer
AUC (Area Under the Curve) measures a binary classifier’s ability to distinguish between the positive and negative classes, computed as the area under its ROC curve, the curve that trades off true positive rate against false positive rate across all thresholds. Its value ranges from 0 to 1: 1.0 is a perfect classifier, 0.5 is random guessing (the diagonal), and below 0.5 is worse than random, which rarely happens and can be flipped into a better model. AUC’s advantages: it is threshold-independent, summarizing performance over every decision threshold at once; it handles imbalanced data well because it evaluates the ranking of predictions rather than absolute counts; and it gives an intuitive single number for comparing different models.
(1) Definition: The area under the ROC curve: one scalar summarizing threshold-free discrimination.
(2) Probabilistic Meaning: AUC equals the probability that a random positive scores higher than a random negative.
(3) Reading The Scale: 1.0 perfect, 0.5 random, below 0.5 inverted.

Figure 1: AUC is literally the shaded area under the ROC curve; the wider the curve bows toward the top-left corner, the larger the area.
Mathematical Formulation:
$\text{AUC} = P\big(s(x^{+}) > s(x^{-})\big)$
$\text{AUC} = \int_{0}^{1} \text{TPR}(\text{FPR}) \, d\text{FPR}$
Where:
$s(x)$ is the model’s score (e.g., predicted probability) for sample $x$ .
$x^{+}$ is a randomly chosen positive sample and $x^{-}$ a randomly chosen negative one.
$\text{TPR}(\text{FPR})$ is the ROC curve itself, true positive rate as a function of false positive rate.
The two lines are equivalent: the geometric area (integral) equals the pairwise-ranking probability, which is why AUC measures ranking quality.
Login to view more content
March 27, 2025