DL0062 LSTM

A Long Short-Term Memory network is a gated recurrent neural network designed to carry useful information across many sequence steps. Each cell maintains a cell state and uses forget, input, and output gates to control what is retained, written, and exposed. Its additive cell-state update creates a shorter gradient path than repeatedly multiplying through a vanilla RNN’s nonlinear state transition, so gradients can remain useful when forget gates stay near one. LSTMs mitigate vanishing gradients rather than eliminating them: saturated gates, long products of forget factors, and poor optimization can still weaken learning.

Figure 1: One LSTM timestep: gated reads and writes surround an additive cell-state highway, while the output gate produces the hidden state.

(1) Two Recurrent States: The cell state is the long-term memory path, while the hidden state is the exposed representation used by the next step and downstream layers.
(2) Gated Update: The forget gate scales old memory, the input gate controls a candidate update, and the output gate selects how much of the updated memory becomes visible.
(3) Gradient Preservation: The derivative along the direct cell-state path contains products of forget gates instead of repeated full recurrent Jacobians; values near one preserve gradient flow.

Mathematical Formulation:

Where:

• is the timestep; is the current input, and are the previous hidden and cell states.
• are element-wise forget, input, and output gates; is the candidate cell update.
• is the updated long-term cell state and is the exposed hidden state.
• and are learned affine parameters; denotes concatenation.
• is sigmoid, is hyperbolic tangent, and is element-wise multiplication; the direct derivative includes .

Figure 2: Why LSTMs mitigate vanishing gradients: a vanilla RNN repeatedly multiplies full nonlinear Jacobians, whereas the LSTM provides a gated direct memory path.