Tag: Basics

ML0020 Data Split
How to split the dataset?
Answer
A dataset is typically split into three parts. The training set is used to fit the model: it learns the patterns and relationships from this data. The validation set is used during development to tune hyperparameters and compare model configurations, which prevents overfitting the training data. The test set is touched only once, for a final unbiased evaluation on completely unseen data: an estimate of real-world generalization. Typical ratios scale with dataset size: for small datasets (fewer than ~1,000 samples), 60–70% training / 10–15% validation / 15–25% test, with k-fold cross-validation strongly recommended because small validation estimates are noisy; for medium datasets (1,000–100,000), a common starting point is 70–80% / 10–15% / 10–15%; for large datasets (over ~100,000), even 98% / 1% / 1% leaves plenty of validation and test samples. For imbalanced data, use stratified splits so every part keeps the original class proportions.
(1) Three Roles: Train fits, validation tunes, test judges; each set answers a different question.
(2) Size-Dependent Ratios: Small data needs more training share plus cross-validation; big data can spare 1–2% for evaluation.
(3) Stratification: Preserve class ratios in every split when classes are imbalanced.

Figure 1: Split ratios by dataset size: the smaller the data, the larger the training share (and the more you need cross-validation); big data can evaluate on 1–2%.
Mathematical Formulation:
$D = D_{\mathrm{train}} \cup D_{\mathrm{val}} \cup D_{\mathrm{test}}$
$D_{\mathrm{train}} \cap D_{\mathrm{val}} = D_{\mathrm{train}} \cap D_{\mathrm{test}} = D_{\mathrm{val}} \cap D_{\mathrm{test}} = \emptyset$
Where:
$D$ is the full dataset, partitioned into three disjoint subsets: no sample may appear in two roles.
$D_{\mathrm{train}}$ is the training set used to fit the model parameters.
$D_{\mathrm{val}}$ is the validation set used to tune hyperparameters and trigger early stopping.
$D_{\mathrm{test}}$ is the test set, used exactly once for the final unbiased estimate; reusing it for tuning silently turns it into validation data.
Login to view more content
March 28, 2025
ML0017 Data Augmentation
What are the common data augmentation techniques?
Answer
Data augmentation increases the diversity and effective size of a training set by creating modified versions of the existing data, especially valuable in computer vision and NLP, where collecting and labeling new data is expensive. In computer vision, common techniques are geometric transformations (rotate, flip, crop, scale), color adjustments (brightness, contrast, saturation, color jitter), and noise injection (random noise or blur). In NLP: synonym replacement, back translation (translate to another language and back), and random insertion or deletion of words. For tabular data: SMOTE-style synthetic sample generation and small random noise on numeric features. The benefits: better robustness and generalization, less overfitting (the model cannot memorize a moving target), relief for class imbalance, and lower data-collection cost.
(1) Core Idea: Apply label-preserving transformations so one sample teaches many variations of the same concept.
(2) By Data Type: Geometric/color/noise for images, synonym/back-translation/edit for text, SMOTE/noise for tables.
(3) Benefits: Robustness, regularization against overfitting, imbalance relief, and cost savings, all from data you already have.

Figure 1: One sample, five augmented views. The label never changes (only the appearance does), so the model learns invariance instead of memorizing pixels.
Mathematical Formulation:
$\tilde{x} = T(x; \phi), \quad \tilde{y} = y$
$\min_\theta \; \mathbb{E}_{(x,y)} \, \mathbb{E}_{\phi} \, \ell\big(f(T(x;\phi);\theta), y\big)$
Where:
$T(x;\phi)$ is the augmentation transform applied to sample $x$ with random parameters $\phi$ (e.g., rotation angle, crop offset).
$\tilde{x}$ is the augmented sample and $\tilde{y}$ its label, unchanged because the transforms are label-preserving.
$f(\cdot;\theta)$ is the model with parameters $\theta$ , and $\ell$ is the training loss.
$\mathbb{E}_{\phi}$ is the expectation over random augmentations: training minimizes loss over infinitely many variants, which is what regularizes the model.
Login to view more content
March 28, 2025
ML0010 Epoch Selection
What are effective strategies for selecting the appropriate number of training epochs in machine learning?
Answer
An epoch is one complete pass through the entire training dataset; choosing the right number of epochs means striking a balance between undertraining and overfitting. The effective strategies are: monitor validation metrics: if validation loss plateaus or starts increasing, further training adds nothing; implement early stopping: halt automatically when performance stops improving and keep the best weights; experiment: begin with a moderate range such as 10–100 epochs and adjust from the training/validation curves; and assess model and data complexity: complex models or datasets may need more epochs to capture the underlying patterns, while simpler problems converge quickly. Related definitions: an iteration is a single parameter update over one batch, so with 1,000 training samples and batch size 100, one epoch consists of 10 iterations.
(1) What An Epoch Is: One full pass over the training set; $N/B$ iterations per epoch for $N$ samples and batch size $B$ .
(2) Selection Strategy: Watch validation loss and let early stopping pick the epoch for you.
(3) Practical Range: Start around 10–100 epochs; scale up with task complexity, down for simple ones.

Figure 1: The validation curve answers “how many epochs”: stop in the sweet-spot zone: before it, the model is undertrained; after it, you are overfitting.
Mathematical Formulation:
$I = \frac{N}{B}$
$t^{*} = \arg\min_t \; \mathcal{L}_{\mathrm{val}}(t)$
Where:
$I$ is the number of iterations (parameter updates) in one epoch.
$N$ is the number of training samples and $B$ is the batch size.
$t^{*}$ is the ideal number of epochs, the one early stopping approximates.
$\mathcal{L}_{\mathrm{val}}(t)$ is the validation loss after $t$ epochs; its minimum marks the sweet spot in Figure 1.
Login to view more content
March 16, 2025
ML0009 Batch Size Selection
What are the best strategies for selecting the appropriate batch size?
Answer
Selecting an appropriate batch size is a crucial hyperparameter choice that trades training efficiency against optimization behavior. The practical strategy: start with a moderate value (16, 32, or 64) and adjust based on memory, gradient stability, and validation performance. Three factors drive the choice: memory constraints (larger batches need more GPU memory), dataset size (large datasets can sustain large batches; small datasets often benefit from the extra variability of small batches), and learning rate interaction (large batches usually allow (or require) a proportionally higher learning rate). Large batches train faster per epoch with stabler gradient estimates and higher memory cost, but can converge to sharp minima that generalize worse; small batches update more often with noisier gradients, which can explore flatter minima and generalize better at lower memory cost.
(1) Trade-Off Summary: Large batch = fast and stable but possibly sharp minima; small batch = slow and noisy but often better generalization.
(2) Practical Starting Points: Try 16/32/64 first, then scale with memory and dataset size.
(3) Interaction With LR: Scale the learning rate roughly proportionally when you change the batch size.

Figure 1: Same loss surface, two batch sizes: the small-batch path is noisy but wanders into a wider basin, while the large-batch path is smooth but direct.
Mathematical Formulation:
$\hat{g} = \frac{1}{B}\sum_{i=1}^{B}\nabla_\theta \mathcal{L}_i$
$\mathrm{Var}[\hat{g}] \propto \frac{\sigma^2}{B}$
Where:
$\hat{g}$ is the gradient estimate computed from one mini-batch.
$B$ is the batch size, and $i\in\{1,\ldots,B\}$ indexes the samples in the batch.
$\nabla_\theta \mathcal{L}_i$ is the per-sample loss gradient.
$\sigma^2$ is the per-sample gradient variance; doubling $B$ halves the gradient noise, which is why large batches give stable estimates and small batches give noisy ones.
Login to view more content
March 16, 2025
ML0008 Learning Rate Selection
What are the best practices for selecting an optimal learning rate?
Answer
Selecting an appropriate learning rate is one of the most important choices in training a neural network: it largely determines how quickly and how well the model learns. Four practices cover most situations. First, grid or random search over a range (e.g., 0.0001, 0.001, 0.01) while watching training performance, to narrow down an effective value. Second, use adaptive optimizers such as Adam, RMSProp, or Adagrad, which adjust the effective rate per parameter from gradient history and need less manual tuning. Third, apply learning rate schedules (step decay, exponential decay, or cosine annealing) that shrink the rate as training approaches convergence. Fourth, monitor the training loss: if it stops decreasing or oscillates, adjust the rate. Too high a rate overshoots the optimum and oscillates or diverges; too low a rate converges slowly or stalls; the right rate converges efficiently to a good solution.
(1) Why It Matters: The learning rate scales every update step; it is the single hyperparameter that most often decides whether training works at all.
(2) Four Practices: Search a range, prefer adaptive optimizers, decay the rate over time, and watch the loss curve.
(3) Diagnose From The Curve: Oscillating or rising loss means too high; an almost flat, slowly creeping loss means too low.

Figure 1: Reading the loss curve: too low creeps down slowly, a good rate drops fast and plateaus, too high oscillates and can diverge.
Mathematical Formulation:
$\theta_{t+1} = \theta_t - \eta_t \, \nabla_\theta \mathcal{L}(\theta_t)$
$\eta_t = \eta_0 \, \gamma^{\lfloor t / s \rfloor} \quad \text{(step decay)}$
Where:
$\theta_t$ denotes the model parameters at step $t$ .
$\eta_t$ is the learning rate at step $t$ , now time-dependent because of the schedule.
$\eta_0$ is the initial learning rate, $\gamma \in (0,1)$ is the decay factor, and $s$ is the number of steps between decays.
$\nabla_\theta \mathcal{L}(\theta_t)$ is the gradient of the training loss $\mathcal{L}$ , scaled by $\eta_t$ at every update.

Figure 2: Three standard schedules. All shrink $\eta_t$ over training so the model takes large steps early and fine steps near convergence.
Login to view more content
March 16, 2025
ML0007 Dropout
What is dropout in neural network training?
Answer
Dropout is a regularization technique used during neural network training to prevent overfitting. At each training step, a fraction of neurons (and their connections) are randomly “dropped out”, meaning their activations are set to zero. This forces the network to learn more robust features, because it cannot rely on any single neuron; instead it learns distributed representations, effectively training an ensemble of many smaller sub-networks that share weights. At inference time every neuron is active, so the model uses all learned features without randomness. To bridge the train/inference gap, inverted dropout scales the active neurons by $1/(1-p)$ during training so no adjustment is needed at inference; the alternative standard dropout scales the weights by $(1-p)$ at inference instead.
(1) Mechanism: Randomly zero a fraction $p$ of activations per training step; the dropped set changes every step.
(2) Why It Works: Prevents co-adaptation of neurons and approximates training an exponentially large ensemble of sub-networks.
(3) Training vs Inference: Inference uses the full network; activations are rescaled (during training with inverted dropout, or at inference with standard dropout) so expected magnitudes match.

Figure 1: Left: the full network. Right: one training step with dropout: grayed, crossed-out neurons are zeroed, forcing the remaining sub-network to carry the prediction.
Mathematical Formulation:
$\tilde{h}_i = \frac{m_i}{1-p} \cdot h_i$
$m_i \sim \mathrm{Bernoulli}(1-p)$
Where:
$\tilde{h}_i$ is the scaled activation of neuron $i$ actually passed to the next layer during training.
$h_i$ is the original activation of neuron $i$ , and $i$ indexes the neurons in a layer.
$m_i$ is the binary dropout mask: 1 keeps the neuron, 0 drops it.
$p$ is the drop probability; $1-p$ is the keep probability, and dividing by it is the inverted-dropout scaling that keeps $\mathbb{E}[\tilde{h}_i] = h_i$ .

Figure 2: Choosing the drop probability $p$ : too little dropout leaves overfitting, too much underfits; the test-error minimum is the sweet spot (typically $p$ around 0.2–0.5).
Login to view more content
March 16, 2025

ML0002 Machine Learning Type

What is the difference between supervised learning and unsupervised learning?

Answer

Supervised learning trains on labeled data (every example comes with a target output) and learns a mapping that predicts the label of new, unseen inputs. Unsupervised learning works on unlabeled data and instead discovers hidden structure in the inputs themselves, such as clusters or low-dimensional representations. The practical difference is the supervision signal: supervised models are corrected against known answers during training, while unsupervised models optimize an intrinsic objective like reconstruction error or cluster compactness. Supervised learning covers classification and regression; unsupervised learning covers clustering and dimensionality reduction.

(1) Data Requirement: Supervised needs input–label pairs $(x_i, y_i)$ ; unsupervised needs only inputs $x_i$ , which are far cheaper to collect.
(2) Learning Objective: Supervised minimizes prediction error against the labels; unsupervised optimizes structure objectives such as cluster compactness or reconstruction loss.
(3) Typical Tasks: Classification and regression for supervised; clustering, dimensionality reduction, and density estimation for unsupervised.

Supervised learning pipeline with labels versus unsupervised pipeline without labels

Figure 1: The supervision signal is the dividing line: supervised training compares predictions with labels, unsupervised training exploits structure in the inputs alone.

Mathematical Formulation:
$\hat{y} = f(x;\theta)$
$\min_\theta \frac{1}{N}\sum_{i=1}^{N}\ell(f(x_i;\theta), y_i)$
$\min_{C_1,\ldots,C_K}\sum_{k=1}^{K}\sum_{x_i \in C_k}\lVert x_i - \mu_k \rVert^2$

Where:

$f(x;\theta)$ is the supervised model with parameters $\theta$ , mapping input $x$ to a predicted label $\hat{y}$ .
$\ell(\cdot)$ is the per-example supervised loss (e.g., cross-entropy or squared error), and $i\in\{1,\ldots,N\}$ indexes the $N$ labeled examples.
$C_k$ is the $k$ -th cluster of a $K$ -means objective and $\mu_k$ its centroid; there are no labels $y_i$ anywhere in the unsupervised objective.

Labeled data with a decision boundary versus unlabeled data grouped into clusters

Figure 2: Left: labeled points let a supervised model fit a decision boundary. Right: without labels, an unsupervised model can only group points by similarity.

Aspect	Supervised Learning	Unsupervised Learning
Training data	Labeled pairs $(x_i, y_i)$	Unlabeled inputs $x_i$
Goal	Predict outputs for new inputs	Discover hidden patterns or structure
Typical tasks	Classification, regression	Clustering, dimensionality reduction
Feedback signal	Direct error against known labels	Intrinsic objective (compactness, reconstruction)

February 25, 2025

ML0001 Loss Curve Plot
The following training loss curves were produced under four different experiment settings. Which curve most likely corresponds to a correct training setup, and what does each of the remaining curves indicate?

Figure 1: Training loss curves recorded under four experiment settings (A–D). Only one reflects healthy training.
Answer
Curve A most likely corresponds to a correct training setup: the loss drops quickly in early epochs and then flattens as the model converges, which is the signature of steady learning with a well-tuned learning rate. Curve B, where the loss increases monotonically, indicates divergence: typically a learning rate far too large or a sign error in the loss or gradient. Curve C stays flat, meaning the weights are barely updating: a near-zero learning rate, broken gradient flow, or frozen parameters. Curve D oscillates sharply, the classic symptom of a learning rate too high for stable descent, so the optimizer keeps overshooting the minimum.
(1) Healthy Curve: A smooth, rapid decrease that levels off toward a floor, showing that gradients flow and the step size is well tuned.
(2) Failure Signatures: Rising loss means divergence, flat loss means no learning, and violent oscillation means unstable steps.
(3) First Knobs To Turn: Check the learning rate first, then gradient flow (vanishing or exploding), then the loss wiring itself.
Mathematical Formulation:
$\theta_{t+1} = \theta_t - \eta \, \nabla_\theta \mathcal{L}(\theta_t)$
$\mathcal{L}_t \to \mathcal{L}_{\min} \quad \text{as} \quad t \to T \text{ for a healthy run}$
Where:
$\theta_t$ denotes the model parameters at epoch $t$ , and $T$ is the final epoch.
$\eta$ is the learning rate, the single hyperparameter behind curves B and D in most real failures.
$\nabla_\theta \mathcal{L}(\theta_t)$ is the gradient of the training loss $\mathcal{L}$ ; if it vanishes or is disconnected, the curve goes flat as in C.
$\mathcal{L}_{\min}$ is the approximate floor the loss converges to, above zero in practice because of label noise and mini-batch variance.
Login to view more content
February 23, 2025