Tag: NN

DL0068 Computational Graphs

What is a computational graph, and why is it useful in deep learning frameworks?

Answer

A computational graph is a directed acyclic graph whose nodes are operations or stored values and whose edges are the tensors flowing between them. Deep learning frameworks materialize this graph while running the forward pass: PyTorch records it dynamically on every iteration (define-by-run), while early TensorFlow built the whole graph statically before execution. Its central use is reverse-mode automatic differentiation: every node carries a known local derivative, so the chain rule becomes a mechanical backward sweep that produces every parameter’s gradient in a single pass, whose cost is roughly 2-3 times the forward pass and does not grow with the number of parameters being differentiated. The same data structure also drives memory planning such as activation checkpointing, whole-graph compiler optimizations like kernel fusion in XLA or torch.compile, device placement, and deployment export to formats such as ONNX.

(1) Structure and Evaluation: nodes are ops or values and edges are tensors; the forward pass evaluates nodes in topological order and caches every intermediate, because the backward sweep needs those values to compute local derivatives.
(2) One Sweep, All Gradients: each node multiplies its incoming adjoint $\bar{v} = \partial L / \partial v$ by its local Jacobian and passes the result to its inputs, summing contributions at fan-out; a single reverse traversal yields gradients for all parameters, which is why backprop scales to billion-parameter models.
(3) Dynamic vs Static Graphs: PyTorch’s define-by-run graph follows arbitrary Python control flow and debugs like ordinary code; static graphs (TensorFlow 1, exported ONNX) trade that flexibility for ahead-of-time whole-graph optimization and portable deployment.

Forward computational graph of L = (wx + b - y)^2: input boxes x=2, w=3 feed a multiply node producing u=6, then an add node with b=1 producing z=7, a subtract node with y=5 producing e=2, and a square node producing L=4

Figure 1: Forward pass of $L = (wx + b - y)^2$ with $x=2,\ w=3,\ b=1,\ y=5$ : each op consumes its input tensors, emits one value, and caches it for the backward sweep.

The backward sweep traverses the same graph in reverse. Starting from the seed $\bar{L} = 1$ , each node applies the chain rule locally: the square node returns $2e = 4$ , the subtraction forwards $+4$ to $z$ and $-4$ to $y$ , the addition copies its adjoint to both $u$ and $b$ , and the multiplication swaps operands, delivering $\bar{w} = 4 \cdot x = 8$ and $\bar{x} = 4 \cdot w = 12$ . No node needs global knowledge of the loss; correctness of the whole sweep follows from composing local derivatives edge by edge. The figure shows $\bar{x}$ and $\bar{y}$ to make the mechanics uniform, but a framework skips those branches when the input and target carry requires_grad=False, so only $\bar{w}$ and $\bar{b}$ are actually materialized. When a value feeds several nodes (fan-out), its adjoint is the sum of the contributions along each outgoing edge, which is exactly the multivariate chain rule.

Backward pass over the same graph with arrows reversed: adjoint labels on every edge showing e-bar=4, z-bar=4, y-bar=-4, u-bar=4, b-bar=4, w-bar=4 times 2=8, x-bar=4 times 3=12, seeded by L-bar=1

Figure 2: The same graph in reverse: each node multiplies the incoming adjoint by its local derivative, so one sweep accumulates $\bar{w} = 8$ and $\bar{b} = 4$ at the leaves.

The graph pays for itself well beyond gradients. Since it records exactly which intermediates feed the loss, frameworks can plan memory: reverse mode must retain cached activations, so training a depth- $N$ network costs $O(N)$ activation memory, and gradient checkpointing cuts this to $O(\sqrt{N})$ by storing only boundary activations and recomputing each segment during the backward pass, at the price of roughly one extra forward evaluation. A static snapshot of the same graph lets compilers fuse ops, pick kernel layouts, and place tensors across devices, and exporters freeze it into a portable artifact. The trade-off is flexibility: a define-by-run graph is rebuilt every iteration, so data-dependent if-statements and loops simply work, while a static graph must capture control flow symbolically, which is why TensorFlow 2 defaulted to eager execution and recovers graph benefits selectively through tf.function tracing.

Mathematical Formulation:
$L = (w\,x + b - y)^2$
$\bar{v}_i = \sum_{j \,\in\, \mathrm{children}(i)} \bar{v}_j \, \frac{\partial v_j}{\partial v_i}$

Where:

$L$ is the scalar loss; $w, x, b, y$ are the weight, input, bias, and target scalars of the running example.
$v_i$ is the output value of node $i$ , and $\bar{v}_i = \partial L / \partial v_i$ is its adjoint, the gradient of the loss with respect to that node’s output.
$\mathrm{children}(i)$ are the nodes that consume $v_i$ ; each child contributes its incoming adjoint times the local partial $\partial v_j / \partial v_i$ , and the results sum, which is the multivariate chain rule.

Aspect	Dynamic Graph (PyTorch Eager)	Static or Compiled (TF1, ONNX, torch.compile)
Graph Built	Every forward pass, fresh per iteration	Once, ahead of execution or by tracing
Control Flow	Native Python if/for; graph follows the data	Symbolic capture; data-dependent branches are hard
Debugging	Standard Python debugger on eager values	Graph-level tooling; values not materialized eagerly
Optimization	Op-by-op dispatch, limited cross-op rewrites	Whole-graph fusion, layout, device placement
Deployment	Needs an explicit export step	Graph is already a portable artifact

August 12, 2026

DL0051 Sparsity in NN
Explain the concept of “Sparsity” in neural networks.
Answer
Sparsity means that many of a network’s weights or activations are exactly zero. Sparse models are smaller to store, cheaper to run, and often more interpretable. It arises naturally (ReLU zeroes negative activations) or is induced deliberately through L1 regularization, pruning, or structured sparsity patterns designed for hardware acceleration.
(1) Weight Sparsity: Pruning or L1 drives unimportant weights to zero, so the model keeps accuracy with a fraction of its parameters.
(2) Activation Sparsity: ReLU produces zeros for all negative pre-activations, so only a subset of neurons “fire” per input.
(3) Hardware Payoff: Only structured sparsity (e.g., NVIDIA’s 2:4 pattern: 2 zeros in every 4 values) maps to real speedups; scattered zeros alone rarely accelerate dense GPU kernels.
Mathematical Formulation:
$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_i |w_i|$
$\mathrm{ReLU}(x) = \max(0, x)$
Where:
$w_i$ is the $i$ -th model weight; the L1 penalty pushes small weights to exact zeros, with $\lambda$ controlling sparsity strength.
$x$ is a neuron’s pre-activation input; ReLU’s hard zero is the simplest source of activation sparsity.

Figure 1: L1-induced weight sparsity: without L1 the weights form a broad bell; with L1 they collapse into a spike at exactly zero.
Hardware Support (NVIDIA 2:4): Since Ampere (A100) and continuing in Hopper/Blackwell, Sparse Tensor Cores require every block of 4 values to contain at least 2 zeros (50% structured sparsity); they then skip the zero multiplies, doubling effective GEMM throughput and roughly halving the stored weights plus memory bandwidth.
Login to view more content
December 30, 2025
ML0066 Model Capacity
Without activation functions, how does the model capacity of a 2-layer neural network compare to a 20-layer network?
Answer
Without activation functions, a neural network, regardless of depth, collapses to a single affine transformation (strictly linear if biases are omitted), so the 2-layer and 20-layer networks have the same representational capacity provided no hidden layer is narrower than $\min(d_0, d_L)$ : both can only express affine mappings. Extra depth adds parameters but zero expressiveness: the collapsed map has at most $d_0 d_L + d_L$ effective degrees of freedom, and its rank is capped by the narrowest layer (the bottleneck) and by the input/output dimensions, never raised by depth or by parameter count. Depth alone provides no additional power to capture non-linear relationships; it changes only the optimization dynamics and implicit bias, never the representable function class.
(1) Layers Collapse: A composition of linear maps is itself linear: stacking 20 of them adds nothing a single layer cannot express.
(2) Only Bottlenecks Limit Capacity: Only a bottleneck layer (narrower than both input and output) lowers the achievable rank; otherwise a wide 2-layer and a wide 20-layer network express exactly the same affine family. Extra parameters from width or depth are redundant knobs, not more capacity.
(3) Neither Fits Non-Linear Data: Both are restricted to affine functions of the input (a hyperplane decision boundary for classification), so without non-linearities the extra depth buys no expressiveness for its compute.
Mathematical Formulation:
$y = W_{\text{eff}}\, x + b_{\text{eff}}$
$\text{Params in layer } i = d_{i-1} d_i + d_i$
Where:
$W_{\text{eff}} = W_L W_{L-1} \cdots W_1$ is the effective weight matrix, the product of all layers’ matrices, itself just one matrix; $b_{\text{eff}}$ the effective bias.
$d_0$ and $d_L$ are the network’s input and output dimensions, while $d_{i-1}$ and $d_i$ are the input and output widths of layer $i$ ; the per-layer parameter count is weights plus biases.
$\mathrm{rank}(W_{\text{eff}}) \leq \min(d_0, d_1, \cdots, d_L)$ : even with every hidden layer arbitrarily wide, the rank is still capped at $\min(d_0, d_L)$ , and any hidden bottleneck only lowers it further, the only sense in which architecture limits capacity here.

Figure 1: The collapse in practice: on sine-wave data, a network without activations (blue) fits nothing but a straight line regardless of depth, while the same architecture with ReLU activations (orange) tracks the true curve (black dots) almost perfectly.
Login to view more content
August 16, 2025

ML0049 Logistic Regression II

Please compare Logistic Regression and Neural Networks.

Answer

Logistic regression is a straightforward, linear model suited to linearly separable data, offering good interpretability and fast, simple training. Neural networks are powerful non-linear models capable of capturing intricate patterns in large datasets, often at the expense of interpretability and higher computational demands.

(1) Expressiveness: Logistic regression draws one hyperplane; neural networks compose hidden layers into arbitrarily curved boundaries.
(2) Workflow: Logistic regression often needs manual feature engineering; neural networks learn feature representations automatically.
(3) Cost: Logistic regression trains in seconds with low overfitting risk when features are well chosen; networks need more data, compute, and regularization.

Logistic regression drawn as a neural network with an input layer feeding one sigmoid output neuron, with no hidden layer

Figure 1: Logistic regression is a neural network: an input layer plus a single output neuron with a sigmoid activation and no hidden layer, the simplest possible network, for binary classification.

Mathematical Formulation:
$\hat{y} = \sigma(\mathbf{w}^{\top} \mathbf{x} + b)$
$\hat{y} = g\big(W_L \, f(\cdots f(W_1 \mathbf{x} + b_1) \cdots) + b_L\big)$

Where:

The first line is logistic regression: one linear score through a sigmoid $\sigma$ , equivalent to a 0-hidden-layer network.
The second line is a general neural network: compositions of weight matrices $W_l$ , biases $b_l$ , and non-linearities $f$ , with output map $g$ .
Setting the number of hidden layers to zero and $g = \sigma$ recovers logistic regression exactly.

Feature	Logistic Regression	Neural Networks (NN)
Model Type	Linear	Non-linear, multi-layer
Architecture	Single-layer (no hidden layers)	Multi-layer (can be deep)
Non-linearity	Can’t model non-linear relationships directly	Can capture complex non-linear patterns
Feature Engineering	Often needed manually	Learns feature representations automatically
Interpretability	High	Low (acts like a black box)
Training Time	Fast, simple to train	Slower, needs more compute & data
Overfitting Risk	Low (if features are well-chosen)	Higher (requires regularization)
Common Use Cases	Simple classification problems (e.g., churn prediction)	Complex tasks (e.g., image, text, speech recognition)

June 10, 2025

DL0019 Go Deep
How does increasing network depth impact the learning process?
Answer
Increasing network depth enhances feature learning and model capacity, but brings training instability, higher computational cost, and design complexity. Deeper stacks learn hierarchical representations (edges become textures, shapes, then objects) and can express certain complex functions far more efficiently than shallow networks, yet without modern techniques like residual connections and normalization, very deep networks are difficult or impossible to train.
Benefits of Depth:
(1) Improved Feature Hierarchy: Each layer composes the previous layer’s features, building increasingly abstract, high-level representations.
(2) Increased Model Capacity: More layers let the network model more complex functions and patterns.
(3) Exponential Efficiency for Complex Functions: Deep networks can represent some functions with vastly fewer units. For example, the $n$ -input parity function needs roughly $2^{n-1}$ neurons in a single hidden layer but only about $n$ small layers in a deep network.
Challenges of Depth:
(1) Vanishing/Exploding Gradients: Gradients shrink or grow multiplicatively through layers; without skip connections a 100-layer network may fail to train because gradients vanish before reaching early layers.
(2) Increased Computational Cost: Deeper networks need significantly more compute and training time.
(3) Higher Data Requirements: More parameters mean more overfitting risk unless the dataset grows correspondingly.

Figure 1: A 6-layer network tracks the target’s high-frequency detail that a single-hidden-layer network smooths over. Depth buys compositional expressiveness.
Where Depth Hurts Training: Each layer multiplies gradients by its Jacobian, so gradient magnitude behaves like a product of many factors; it collapses exponentially when those factors are consistently small.

Figure 2: In a plain deep network gradient norms decay exponentially toward the vanishing floor; skip connections keep gradients healthy across all 50 layers.
Mathematical Formulation:
$\frac{\partial \mathcal{L}}{\partial x_1} = \prod_{l=1}^{L} J_l \cdot \nabla_{x_L}\mathcal{L}$
$\lVert g_1 \rVert \approx \lVert \nabla_{x_L}\mathcal{L} \rVert \cdot \prod_{l=1}^{L} \sigma_l$
Where:
$J_l$ is the Jacobian of layer $l$ ; gradients reaching the first layer are a product of $L$ Jacobians.
$\sigma_l$ is the dominant singular value of $J_l$ ; if most $\sigma_l$ are below 1, the product, and hence $\lVert g_1 \rVert$ , decays exponentially in $L$ .
Residual blocks replace each factor with $I + J_{F_l}$ , keeping the identity path alive so gradient norms stay near 1 even at 50+ layers.
Login to view more content
May 30, 2025
ML0046 Forward Propagation
Please explain the process of Forward Propagation.
Answer
Forward propagation is the process by which a neural network takes an input and generates a prediction: the input data is passed systematically through each layer of the network. At every neuron, a weighted sum of the previous layer’s outputs (plus a bias) is computed, then a non-linear activation is applied; this repeats layer by layer until the output layer emits the final prediction. It is essentially the prediction phase of the network: information flows in one direction, from input to output, using the learned weights and biases.
(1) Input Layer: The network receives the raw input data.
(2) Layer-Wise Processing: Each neuron computes a linear combination (weighted sum plus bias), then applies a non-linear activation (ReLU, sigmoid, tanh) to introduce non-linearity.
(3) Propagation Through Layers: The output of one layer becomes the input to the next, progressing through all hidden layers.
(4) Output Generation: The final layer applies a task-appropriate function (softmax for classification, a linear function for regression) to produce the prediction.

Figure 1: One direction only: each layer transforms activations into pre-activations ( $z = Wx + b$ ) and back into activations ( $a = f(z)$ ) until the output layer produces the prediction. No gradient information flows here; that is backpropagation’s job.
Mathematical Formulation:
$z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$
$a^{(l)} = f\big(z^{(l)}\big)$
$\hat{y} = a^{(L)}$
Where:
$a^{(l-1)}$ is the previous layer’s activation vector ( $a^{(0)} = x$ , the input).
$W^{(l)}, b^{(l)}$ are layer $l$ ‘s learned weights and biases; $z^{(l)}$ is the pre-activation.
$f(\cdot)$ is the activation function; $L$ is the number of layers and $\hat{y}$ the prediction.

Figure 2: A worked example with concrete numbers: inputs $x = (0.5, 0.1, 0.4)$ flow through fixed weights and biases; each hidden node shows its computed $z$ and sigmoid activation $a$ , ending in the scalar prediction. Every number on the diagram follows from the two formulas above.
Login to view more content
May 27, 2025
ML0045 Multi-Layer Perceptron
What is a Multi-Layer Perceptron (MLP)? How does it overcome Perceptron limitations?
Answer
A Multi-Layer Perceptron (MLP) is a feedforward neural network with one or more hidden layers between the input and output layers. Its hidden layers use non-linear activation functions (ReLU, sigmoid, or tanh) to model complex relationships, and it is trained with backpropagation, which adjusts the weights to minimize errors. MLPs are used for classification, regression, and function approximation.
(1) Hidden Layers + Non-Linearity: Each hidden unit applies a non-linear activation to its weighted sum, letting the network compose many linear maps into a curved function.
(2) Overcomes The Perceptron: Unlike a single-layer perceptron (restricted to linearly separable problems), an MLP learns non-linear decision boundaries and handles problems such as XOR.
(3) Universal Approximation: With enough neurons and layers, an MLP can approximate any continuous function, making it a powerful general-purpose model.

Figure 1: An MLP for 3-class classification: 4 inputs, two hidden layers (6 and 4 units) with non-linear activations, and 3 softmax-style outputs. Every layer is fully connected to the next.
Mathematical Formulation:
$h_l = f\big(W_l\, h_{l-1} + b_l\big)$
$\hat{y} = g\big(W_L\, h_{L-1} + b_L\big)$
Where:
$h_l$ is the activation vector of layer $l$ ( $h_0 = x$ is the input).
$W_l, b_l$ are the weight matrix and bias of layer $l$ , learned by backpropagation.
$f(\cdot)$ is the hidden non-linearity (ReLU/sigmoid/tanh); $g(\cdot)$ is the output map (softmax for classification, identity for regression).

Figure 2: The payoff: on XOR, the perceptron’s single line misclassifies half the positives, while the MLP’s hidden units carve a curved region that separates the classes perfectly.
Login to view more content
May 26, 2025
ML0044 Perceptron
Describe the Perceptron and its limitations.
Answer
The perceptron is a simple linear classifier that computes a weighted sum of input features, adds a bias, and applies a step function to produce a binary decision. It works well only for data that is linearly separable, where a straight line (or hyperplane in higher dimensions) can separate the classes.
(1) Linear Score: The perceptron combines inputs linearly as $w^T x + b$ ; geometry-wise this defines one hyperplane.
(2) Step Activation: A threshold turns the score into a hard 0/1 output. Note this non-linearity at the output still leaves the decision boundary linear.
(3) Limitations: It cannot solve non-linearly-separable problems like XOR, a single layer cannot model complex patterns, and it outputs bare binary values with no confidence or probability.

Figure 1: Perceptron structure: inputs $x_1, x_2, x_3$ and a constant 1 (for the bias) feed through weights into a single unit whose step activation emits the binary output.
Mathematical Formulation:
$y = f(w^T x + b)$
$f(z) = 1 \text{ if } z \geq 0;\quad f(z) = 0 \text{ otherwise}$
Where:
$y$ is the predicted output (0 or 1).
$w$ is the weight vector, $x$ the input vector, $b$ the bias term.
$f(\cdot)$ is the step activation; $z = w^T x + b$ is the linear pre-activation whose sign alone decides the class.

Figure 2: What one hyperplane can and cannot do: the left data is linearly separable and a perceptron solves it; the right XOR pattern defeats every single straight line, the classic motivation for hidden layers.
Login to view more content
May 26, 2025
ML0041 Concept of NN
Please explain the concept of a Neural Network.
Answer
A neural network (NN) is a machine learning model composed of layers of interconnected neurons. It learns patterns in data by adjusting weights through training, enabling tasks like classification, regression, and more. Neural networks are inspired by biology (they are computer systems modeled after the human brain’s network of neurons), and they excel at identifying complex, non-linear patterns, which makes them suitable for image recognition, natural language processing, and data classification.
(1) Layered Structure: A neural network consists of an input layer, one or more hidden layers, and an output layer; data flows from input to output through the hidden layers.
(2) Neurons And Activation: Each neuron computes a weighted sum of its inputs, adds a bias, and applies an activation function. Weights and biases are learnable parameters adjusted during training, and activation functions (e.g., ReLU, sigmoid) introduce the non-linearity that lets the network model complex relationships.
(3) Learning Process: The network learns by adjusting weights and biases through training algorithms such as backpropagation, minimizing the error between its predictions and the actual results.

Figure 1: A feedforward neural network: every neuron in one layer connects to every neuron in the next. Each connection carries a learned weight; each hidden and output neuron adds a bias and applies an activation function.
Mathematical Formulation:
$a_j = f\Big(\sum_{i} w_{ji}\, x_i + b_j\Big)$
Where:
$x_i$ are the inputs to the neuron (raw features for the input layer, previous-layer activations otherwise).
$w_{ji}$ is the weight on the connection from input $i$ to neuron $j$ ; $b_j$ is the neuron’s bias; both are learned during training.
$f(\cdot)$ is the activation function (ReLU, sigmoid, tanh, …); $a_j$ is the neuron’s output passed to the next layer.
Login to view more content
May 24, 2025
ML0037 Bias in NN
Why is bias used in neural networks?
Answer
The bias term in a neuron ( $z = Wx + b$ ) is the learnable offset that shifts the activation function’s threshold. Without it, every neuron’s pre-activation would be a strictly linear (origin-passing) function of its inputs: a neuron could only fire proportionally to its input, and every decision boundary or fitted function would be forced through the origin. The bias removes that constraint: it lets a neuron activate even when its weighted input sums to zero, and lets decision boundaries sit anywhere in input space, not just through the origin. This adds crucial flexibility for approximating real-world functions, compensates for systematic offsets in the data, and plays a role loosely analogous to the firing threshold of a biological neuron: the bias sets how much input stimulation is needed before the neuron becomes active. In short, the bias is to a neuron what the intercept is to linear regression: a small parameter with an outsized effect on representational power.
(1) Shifts the Threshold: The bias moves the activation curve left/right, so a neuron can fire (or stay off) at any input level.
(2) Escapes the Origin: Without bias, boundaries and fitted functions are forced through (0,0); with it they can sit anywhere.
(3) Flexibility: One extra learnable parameter per neuron that absorbs systematic offsets and improves approximation.

Figure 1: Why bias matters in one picture: the data’s trend clearly does not pass through the origin. The no-bias model (orange) is constrained through (0,0) and misfits everywhere; the model with a bias term (blue) shifts the line up and fits the trend.
Mathematical Formulation:
$z = \sum_{i} w_i x_i + b = Wx + b$
$a = f(z)$
$z > 0 \Leftrightarrow \sum_i w_i x_i > -b$
Where:
$x_i$ are the inputs, $w_i$ the weights, $b$ the bias, $z$ the pre-activation, $a$ the output.
$-b$ acts as the effective threshold: the third line says the unit crosses its threshold exactly when the weighted input exceeds $-b$ . With $b > 0$ the neuron activates more easily (even at zero input); with a negative bias it requires stronger input to fire.
Login to view more content
May 17, 2025