Interview for Machine Learning

Tag: RNN

DL0062 LSTM
What is an LSTM (Long Short-Term Memory) network? How does it address vanishing gradients?
Answer
A Long Short-Term Memory network is a gated recurrent neural network designed to carry useful information across many sequence steps. Each cell maintains a cell state and uses forget, input, and output gates to control what is retained, written, and exposed. Its additive cell-state update creates a shorter gradient path than repeatedly multiplying through a vanilla RNN’s nonlinear state transition, so gradients can remain useful when forget gates stay near one. LSTMs mitigate vanishing gradients rather than eliminating them: saturated gates, long products of forget factors, and poor optimization can still weaken learning.

Figure 1: One LSTM timestep: gated reads and writes surround an additive cell-state highway, while the output gate produces the hidden state.
(1) Two Recurrent States: The cell state $c_t$ is the long-term memory path, while the hidden state $h_t$ is the exposed representation used by the next step and downstream layers.
(2) Gated Update: The forget gate scales old memory, the input gate controls a candidate update, and the output gate selects how much of the updated memory becomes visible.
(3) Gradient Preservation: The derivative along the direct cell-state path contains products of forget gates instead of repeated full recurrent Jacobians; values near one preserve gradient flow.
Mathematical Formulation:
$f_t=\sigma\!\left(W_f[x_t,h_{t-1}]+b_f\right)$
$i_t=\sigma\!\left(W_i[x_t,h_{t-1}]+b_i\right)$
$\tilde c_t=\tanh\!\left(W_c[x_t,h_{t-1}]+b_c\right)$
$c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t$
$o_t=\sigma\!\left(W_o[x_t,h_{t-1}]+b_o\right)$
$h_t=o_t\odot\tanh(c_t)$
Where:
$t$ is the timestep; $x_t$ is the current input, and $h_{t-1},c_{t-1}$ are the previous hidden and cell states.
$f_t,i_t,o_t\in(0,1)$ are element-wise forget, input, and output gates; $\tilde c_t$ is the candidate cell update.
$c_t$ is the updated long-term cell state and $h_t$ is the exposed hidden state.
$W_f,W_i,W_c,W_o$ and $b_f,b_i,b_c,b_o$ are learned affine parameters; $[x_t,h_{t-1}]$ denotes concatenation.
$\sigma$ is sigmoid, $\tanh$ is hyperbolic tangent, and $\odot$ is element-wise multiplication; the direct derivative includes $\partial c_t/\partial c_{t-1}=f_t$ .

Figure 2: Why LSTMs mitigate vanishing gradients: a vanilla RNN repeatedly multiplies full nonlinear Jacobians, whereas the LSTM provides a gated direct memory path.
Login to view more content
July 20, 2026
DL0032 Transformer VS RNN
What makes Transformers more parallel-friendly than RNNs?
Answer
The fundamental difference lies in their architecture: RNNs sequentially process data, with each step depending on the output of the previous one. Transformers, on the other hand, utilize attention to examine all parts of the sequence simultaneously, enabling parallel processing. This parallelizability is a key reason for the Transformer’s superior performance on many tasks and its dominance in modern natural language processing.
(1) No Temporal Dependency: Transformers process all input tokens simultaneously, unlike RNNs, which depend on previous hidden states.
(2) Self-Attention is Fully Parallelizable: Attention scores are computed for all positions in a single pass.
(3) Optimized for GPUs: Matrix multiplications in Transformers leverage GPU cores better than the sequential loops in RNNs.
The figure below demonstrates the architectures of RNNs and Transformers.
Login to view more content
August 10, 2025