Category: Easy

ML0053 Hinge Loss for SVM
Explain the Hinge Loss function used in SVM.
Answer
The hinge loss is the key element of Support Vector Machines: it penalizes both misclassified points and correctly classified points that lie inside the margin. Points classified correctly with margin to spare get zero loss; the loss grows linearly as a point moves toward or across the decision boundary. This structure is exactly what pushes the SVM to maximize the margin, promoting robust, generalizable boundaries.
(1) Zero Loss Zone: When $y \cdot f(\mathbf{x}) \geq 1$ , the point is correctly classified and outside (or exactly on) the margin, the loss is 0.
(2) Linear Penalty Zone: When $y \cdot f(\mathbf{x})$ falls below 1, i.e., inside the margin or misclassified, the loss grows linearly with the violation.
(3) Convex But Not Smooth: The kink at $y \cdot f(\mathbf{x}) = 1$ makes the function non-differentiable there, so optimization uses subgradients instead of plain gradients.

Figure 1: Hinge loss vs the margin $y \cdot f(\mathbf{x})$ : flat at zero once the point is beyond the margin (right of the dashed line at 1), ramping up linearly inside the margin and for misclassified points (left of it). The kink at 1 is where subgradients take over.
Mathematical Formulation:
$\text{Hinge Loss} = \max\big(0,\; 1 - y \cdot f(\mathbf{x})\big)$
Where:
$y \in \{-1, +1\}$ is the true label.
$f(\mathbf{x})$ is the raw model output (the signed score, before any threshold).
The product $y \cdot f(\mathbf{x})$ is the (functional) margin: positive means correctly classified, ≥ 1 means correct with margin.
Login to view more content
June 11, 2025
ML0051 Linear SVM
Can you explain the key concepts behind a Linear Support Vector Machine?
Answer
A Linear Support Vector Machine (Linear SVM) is a classifier that finds the optimal straight line (hyperplane) separating two classes by maximizing the margin between them. It relies on a few critical points (the support vectors) and offers strong generalization, especially on linearly separable data.
(1) Hyperplane: The decision boundary that separates data points of different classes.
(2) Margin: The distance between the hyperplane and the nearest data point of each class; the SVM maximizes it.
(3) Support Vectors: The points lying closest to the hyperplane; they alone define it: moving any other point changes nothing.
(4) Objective: Maximize the margin while minimizing classification error (hard margin forbids error; soft margin prices it in).

Figure 1: Hard-margin SVM: the solid hyperplane $w^T x + b = 0$ sits midway between the dashed margins $w^T x + b = \pm 1$ , and only the circled support vectors touch the margin: they alone determine the boundary.
Mathematical Formulation:
$f(\mathbf{x}) = \mathbf{w}^\top \mathbf{x} + b$
$\hat{y} = \mathrm{sign}(\mathbf{w}^\top \mathbf{x} + b) = \mathrm{sign}(f(\mathbf{x}))$
$\min_{\mathbf{w}, b} \; \frac{1}{2} \|\mathbf{w}\|^2$
$\text{subject to } y_i(\mathbf{w}^\top \mathbf{x}_i + b) \geq 1 \quad \text{for all } i$
Where:
$\mathbf{x}$ is the input feature vector, $\mathbf{w}$ the weight vector, $b$ the bias; $\hat{y}$ is the predicted label.
$\mathrm{sign}(\cdot)$ returns +1 if its argument is ≥ 0 and −1 otherwise; $y_i \in \{-1, +1\}$ is the true label of point $\mathbf{x}_i$ .
Minimizing $\frac{1}{2}\|\mathbf{w}\|^2$ under those constraints is the hard-margin objective: since the margin width is $2 / \|\mathbf{w}\|$ , small $\|\mathbf{w}\|$ means a wide margin.
Login to view more content
June 11, 2025

ML0049 Logistic Regression II

Please compare Logistic Regression and Neural Networks.

Answer

Logistic regression is a straightforward, linear model suited to linearly separable data, offering good interpretability and fast, simple training. Neural networks are powerful non-linear models capable of capturing intricate patterns in large datasets, often at the expense of interpretability and higher computational demands.

(1) Expressiveness: Logistic regression draws one hyperplane; neural networks compose hidden layers into arbitrarily curved boundaries.
(2) Workflow: Logistic regression often needs manual feature engineering; neural networks learn feature representations automatically.
(3) Cost: Logistic regression trains in seconds with low overfitting risk when features are well chosen; networks need more data, compute, and regularization.

Logistic regression drawn as a neural network with an input layer feeding one sigmoid output neuron, with no hidden layer

Figure 1: Logistic regression is a neural network: an input layer plus a single output neuron with a sigmoid activation and no hidden layer, the simplest possible network, for binary classification.

Mathematical Formulation:
$\hat{y} = \sigma(\mathbf{w}^{\top} \mathbf{x} + b)$
$\hat{y} = g\big(W_L \, f(\cdots f(W_1 \mathbf{x} + b_1) \cdots) + b_L\big)$

Where:

The first line is logistic regression: one linear score through a sigmoid $\sigma$ , equivalent to a 0-hidden-layer network.
The second line is a general neural network: compositions of weight matrices $W_l$ , biases $b_l$ , and non-linearities $f$ , with output map $g$ .
Setting the number of hidden layers to zero and $g = \sigma$ recovers logistic regression exactly.

Feature	Logistic Regression	Neural Networks (NN)
Model Type	Linear	Non-linear, multi-layer
Architecture	Single-layer (no hidden layers)	Multi-layer (can be deep)
Non-linearity	Can’t model non-linear relationships directly	Can capture complex non-linear patterns
Feature Engineering	Often needed manually	Learns feature representations automatically
Interpretability	High	Low (acts like a black box)
Training Time	Fast, simple to train	Slower, needs more compute & data
Overfitting Risk	Low (if features are well-chosen)	Higher (requires regularization)
Common Use Cases	Simple classification problems (e.g., churn prediction)	Complex tasks (e.g., image, text, speech recognition)

June 10, 2025

ML0048 Logistic Regression

Can you explain logistic regression and how it contrasts with linear regression?

Answer

Logistic regression maps inputs to a probability space for classification, while linear regression estimates continuous outcomes through a direct linear relationship. Logistic regression estimates the probability that a binary outcome ( $y = 1$ ) occurs given an input vector $\mathbf{x}$ , by passing a linear score through the sigmoid function; the output lies between 0 and 1 and is interpreted as the probability of class membership. Note that standard logistic regression is still fundamentally a linear model at its core: it cannot model non-linear relationships directly.

(1) Purpose: Linear regression predicts a continuous output (price, height); logistic regression predicts a probability for classification (spam or not).
(2) Output Range: Linear regression emits any real number (can be negative or above 1); logistic regression squashes to (0, 1) with the sigmoid.
(3) Assumption: Linear regression assumes a linear input-output relationship; logistic regression assumes a linear relationship between inputs and the log-odds of the positive class.

Left panel linear regression line fitting continuous data, right panel sigmoid curve fitting binary zero one data with a 0.5 decision boundary

Figure 1: The contrast in one picture: linear regression fits a straight line through continuous targets (left); logistic regression fits an S-shaped sigmoid to 0/1 labels (right), with the 0.5 probability level acting as the decision boundary.

Mathematical Formulation:
$\Pr(y = 1 \mid \mathbf{x}) = \frac{1}{1 + e^{-(\mathbf{w}^{\top} \mathbf{x} + b)}}$

Where:

$\mathbf{x}$ is the input feature vector.
$\mathbf{w}$ is the weight vector and $b$ the bias; the linear score $\mathbf{w}^{\top}\mathbf{x} + b$ is the log-odds of the positive class.
The sigmoid maps the score to (0, 1); classifying at threshold 0.5 gives a linear decision boundary.

Feature	Logistic Regression	Linear Regression
Type of Problem	Classification	Regression
Dependent Variable	Categorical (e.g., Yes/No, 0/1, True/False)	Continuous (e.g., Price, Temperature, Age)
Output	Probability (0 to 1)	Real number (any value)
Core Function	Sigmoid function (for binary)	Identity function
Loss Function	Log-loss (Cross-entropy)	Mean Squared Error (MSE)

June 9, 2025

ML0047 Parameters
What are the differences between parameters and hyperparameters?
Answer
Parameters are the values a model learns from its training data, while hyperparameters are settings defined by the user that guide the training process and model architecture. Parameters represent the model’s internal knowledge and decision rules; hyperparameters govern how the model learns, influencing its architecture, training dynamics, and ultimately its ability to generalize to unseen data: choosing them well is crucial for building an effective model.
(1) Parameters: Internal variables learned from data (e.g., weights and biases); adjusted during training by the optimization algorithm; capture the model’s learned patterns and information.
(2) Hyperparameters: External configurations set before training (e.g., learning rate, batch size, number of layers); remain fixed during training and are not updated by the learning process; influence how the model learns and its overall structure.
(3) Who Updates Whom: The optimizer updates parameters by minimizing the loss; the practitioner (or a search procedure) updates hyperparameters by watching validation performance.

Figure 1: The two knobs of a model: parameters (weights, biases) live inside the training loop and are updated every step by the optimizer; hyperparameters (learning rate, batch size, depth, …) are set before training and shape how that loop behaves.
Mathematical Formulation:
$\theta_{t+1} = \theta_t - \eta \, \nabla_\theta \mathcal{L}(\theta_t)$
$\lambda^* = \arg\min_{\lambda} \; \mathcal{L}_{\text{val}}\big(\hat{\theta}(\lambda)\big)$
Where:
$\theta$ denotes the parameters (weights and biases), updated every training step $t$ by gradient descent.
$\eta$ is the learning rate, itself a hyperparameter, set before training and never learned by the optimizer.
$\lambda$ stands for hyperparameters generally; $\hat{\theta}(\lambda)$ is the fully trained model under that setting, and the outer minimization over validation loss is hyperparameter search.
Login to view more content
May 29, 2025
ML0046 Forward Propagation
Please explain the process of Forward Propagation.
Answer
Forward propagation is the process by which a neural network takes an input and generates a prediction: the input data is passed systematically through each layer of the network. At every neuron, a weighted sum of the previous layer’s outputs (plus a bias) is computed, then a non-linear activation is applied; this repeats layer by layer until the output layer emits the final prediction. It is essentially the prediction phase of the network: information flows in one direction, from input to output, using the learned weights and biases.
(1) Input Layer: The network receives the raw input data.
(2) Layer-Wise Processing: Each neuron computes a linear combination (weighted sum plus bias), then applies a non-linear activation (ReLU, sigmoid, tanh) to introduce non-linearity.
(3) Propagation Through Layers: The output of one layer becomes the input to the next, progressing through all hidden layers.
(4) Output Generation: The final layer applies a task-appropriate function (softmax for classification, a linear function for regression) to produce the prediction.

Figure 1: One direction only: each layer transforms activations into pre-activations ( $z = Wx + b$ ) and back into activations ( $a = f(z)$ ) until the output layer produces the prediction. No gradient information flows here; that is backpropagation’s job.
Mathematical Formulation:
$z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$
$a^{(l)} = f\big(z^{(l)}\big)$
$\hat{y} = a^{(L)}$
Where:
$a^{(l-1)}$ is the previous layer’s activation vector ( $a^{(0)} = x$ , the input).
$W^{(l)}, b^{(l)}$ are layer $l$ ‘s learned weights and biases; $z^{(l)}$ is the pre-activation.
$f(\cdot)$ is the activation function; $L$ is the number of layers and $\hat{y}$ the prediction.

Figure 2: A worked example with concrete numbers: inputs $x = (0.5, 0.1, 0.4)$ flow through fixed weights and biases; each hidden node shows its computed $z$ and sigmoid activation $a$ , ending in the scalar prediction. Every number on the diagram follows from the two formulas above.
Login to view more content
May 27, 2025
ML0045 Multi-Layer Perceptron
What is a Multi-Layer Perceptron (MLP)? How does it overcome Perceptron limitations?
Answer
A Multi-Layer Perceptron (MLP) is a feedforward neural network with one or more hidden layers between the input and output layers. Its hidden layers use non-linear activation functions (ReLU, sigmoid, or tanh) to model complex relationships, and it is trained with backpropagation, which adjusts the weights to minimize errors. MLPs are used for classification, regression, and function approximation.
(1) Hidden Layers + Non-Linearity: Each hidden unit applies a non-linear activation to its weighted sum, letting the network compose many linear maps into a curved function.
(2) Overcomes The Perceptron: Unlike a single-layer perceptron (restricted to linearly separable problems), an MLP learns non-linear decision boundaries and handles problems such as XOR.
(3) Universal Approximation: With enough neurons and layers, an MLP can approximate any continuous function, making it a powerful general-purpose model.

Figure 1: An MLP for 3-class classification: 4 inputs, two hidden layers (6 and 4 units) with non-linear activations, and 3 softmax-style outputs. Every layer is fully connected to the next.
Mathematical Formulation:
$h_l = f\big(W_l\, h_{l-1} + b_l\big)$
$\hat{y} = g\big(W_L\, h_{L-1} + b_L\big)$
Where:
$h_l$ is the activation vector of layer $l$ ( $h_0 = x$ is the input).
$W_l, b_l$ are the weight matrix and bias of layer $l$ , learned by backpropagation.
$f(\cdot)$ is the hidden non-linearity (ReLU/sigmoid/tanh); $g(\cdot)$ is the output map (softmax for classification, identity for regression).

Figure 2: The payoff: on XOR, the perceptron’s single line misclassifies half the positives, while the MLP’s hidden units carve a curved region that separates the classes perfectly.
Login to view more content
May 26, 2025
ML0044 Perceptron
Describe the Perceptron and its limitations.
Answer
The perceptron is a simple linear classifier that computes a weighted sum of input features, adds a bias, and applies a step function to produce a binary decision. It works well only for data that is linearly separable, where a straight line (or hyperplane in higher dimensions) can separate the classes.
(1) Linear Score: The perceptron combines inputs linearly as $w^T x + b$ ; geometry-wise this defines one hyperplane.
(2) Step Activation: A threshold turns the score into a hard 0/1 output. Note this non-linearity at the output still leaves the decision boundary linear.
(3) Limitations: It cannot solve non-linearly-separable problems like XOR, a single layer cannot model complex patterns, and it outputs bare binary values with no confidence or probability.

Figure 1: Perceptron structure: inputs $x_1, x_2, x_3$ and a constant 1 (for the bias) feed through weights into a single unit whose step activation emits the binary output.
Mathematical Formulation:
$y = f(w^T x + b)$
$f(z) = 1 \text{ if } z \geq 0;\quad f(z) = 0 \text{ otherwise}$
Where:
$y$ is the predicted output (0 or 1).
$w$ is the weight vector, $x$ the input vector, $b$ the bias term.
$f(\cdot)$ is the step activation; $z = w^T x + b$ is the linear pre-activation whose sign alone decides the class.

Figure 2: What one hyperplane can and cannot do: the left data is linearly separable and a perceptron solves it; the right XOR pattern defeats every single straight line, the classic motivation for hidden layers.
Login to view more content
May 26, 2025
ML0043 Feature Scaling
Walk me through the rationale behind Feature Scaling in machine learning.
Answer
Feature scaling is a fundamental data preprocessing step that normalizes or standardizes the range of numerical features, so all features contribute equally to the model. It leads to faster convergence, improved accuracy, and better overall performance, especially for algorithms sensitive to feature magnitudes or based on distance calculations (e.g., SVM, KNN), where an unscaled large-range feature would overpower the others.
(1) Definition: Normalize or standardize input features so they sit on a similar scale.
(2) Why Needed: Many ML models are sensitive to feature magnitude; scaling prevents dominant features from overwhelming the rest purely because of their units.
(3) Two Common Methods: Min-max scaling maps features to a fixed range (usually [0, 1]); standardization (z-score) centers features to mean 0 and standard deviation 1.

Figure 1: The same 100 samples under the two scalings: the original features live on incompatible scales (Feature 1 in [0, 100], Feature 2 around 1000); min-max compresses both axes into [0, 1]; standardization centers the cloud at the origin with unit spread. The shape of the point cloud is preserved: only the units change.
Mathematical Formulation:
$X_{\text{normalized}} = \frac{X - X_{\text{min}}}{X_{\text{max}} - X_{\text{min}}}$
$X_{\text{standardized}} = \frac{X - \mu}{\sigma}$
Where:
$X$ is the original feature value.
$X_{\text{min}}$ and $X_{\text{max}}$ are the feature’s minimum and maximum in the training data.
$\mu$ and $\sigma$ are the feature’s mean and standard deviation in the training data.
Login to view more content
May 25, 2025
ML0042 Early Stopping
What is Early Stopping? How is it implemented?
Answer
Early stopping is a regularization technique that halts training when the model’s performance on a validation set stops improving, thus avoiding overfitting. It monitors a metric such as validation loss or validation accuracy and stops after a defined number of stagnant epochs (the patience). This ensures efficient training and better generalization.
(1) Split Data: Reserve a validation set separate from the training set.
(2) Evaluate Each Epoch: After every training epoch, measure performance on the validation set.
(3) Track Improvement: If performance improves, save the model and reset the patience counter; if not, increment the counter; when it reaches the patience, stop training.
(4) Restore Best Weights: After stopping, reload the weights from the epoch that yielded the best validation performance, not the final epoch.

Figure 1: Early stopping in action: training loss falls monotonically, but validation loss bottoms out at epoch 60 (ideal stop) and then rises as the model overfits. With patience 10, training actually halts at epoch 70 and the weights from epoch 60 are restored.
Mathematical Formulation:
$t^* = \arg\min_{t} \; \mathcal{L}_{\text{val}}\big(\theta_t\big)$
$\text{stop at } t^* + p \text{ if no epoch in } (t^*,\, t^* + p] \text{ beats } \mathcal{L}_{\text{val}}(\theta_{t^*})$
Where:
$\theta_t$ are the model weights after epoch $t$ ; $\mathcal{L}_{\text{val}}$ is the validation loss.
$t^*$ is the epoch with the best validation loss, the checkpoint whose weights are restored at the end.
$p$ is the patience: how many consecutive non-improving epochs are tolerated before stopping.
Login to view more content
May 25, 2025