LSTM Archives - Interview for Machine Learning

What is an LSTM (Long Short-Term Memory) network? How does it address vanishing gradients?

Answer

A Long Short-Term Memory network is a gated recurrent neural network designed to carry useful information across many sequence steps. Each cell maintains a cell state and uses forget, input, and output gates to control what is retained, written, and exposed. Its additive cell-state update creates a shorter gradient path than repeatedly multiplying through a vanilla RNN’s nonlinear state transition, so gradients can remain useful when forget gates stay near one. LSTMs mitigate vanishing gradients rather than eliminating them: saturated gates, long products of forget factors, and poor optimization can still weaken learning.

LSTM cell architecture showing forget, input, candidate, and output gates around the cell-state path.

Figure 1: One LSTM timestep: gated reads and writes surround an additive cell-state highway, while the output gate produces the hidden state.

(1) Two Recurrent States: The cell state $c_t$ is the long-term memory path, while the hidden state $h_t$ is the exposed representation used by the next step and downstream layers.
(2) Gated Update: The forget gate scales old memory, the input gate controls a candidate update, and the output gate selects how much of the updated memory becomes visible.
(3) Gradient Preservation: The derivative along the direct cell-state path contains products of forget gates instead of repeated full recurrent Jacobians; values near one preserve gradient flow.

Mathematical Formulation:
$f_t=\sigma\!\left(W_f[x_t,h_{t-1}]+b_f\right)$
$i_t=\sigma\!\left(W_i[x_t,h_{t-1}]+b_i\right)$
$\tilde c_t=\tanh\!\left(W_c[x_t,h_{t-1}]+b_c\right)$
$c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t$
$o_t=\sigma\!\left(W_o[x_t,h_{t-1}]+b_o\right)$
$h_t=o_t\odot\tanh(c_t)$

Where:

$t$ is the timestep; $x_t$ is the current input, and $h_{t-1},c_{t-1}$ are the previous hidden and cell states.
$f_t,i_t,o_t\in(0,1)$ are element-wise forget, input, and output gates; $\tilde c_t$ is the candidate cell update.
$c_t$ is the updated long-term cell state and $h_t$ is the exposed hidden state.
$W_f,W_i,W_c,W_o$ and $b_f,b_i,b_c,b_o$ are learned affine parameters; $[x_t,h_{t-1}]$ denotes concatenation.
$\sigma$ is sigmoid, $\tanh$ is hyperbolic tangent, and $\odot$ is element-wise multiplication; the direct derivative includes $\partial c_t/\partial c_{t-1}=f_t$ .

Comparison of gradient propagation through a vanilla recurrent network and an LSTM cell-state path.

Figure 2: Why LSTMs mitigate vanishing gradients: a vanilla RNN repeatedly multiplies full nonlinear Jacobians, whereas the LSTM provides a gated direct memory path.

Tag: LSTM

DL0062 LSTM