Tag: LSTM

DL0074 GRU vs LSTM

How does a GRU differ from an LSTM, and when would you prefer one over the other?

Answer

Both are gated recurrent cells that replace the vanilla RNN’s repeated multiplication by an additive state update, which is what keeps gradients alive across long sequences. The LSTM carries two state tensors (the cell state $c_t$ and the hidden state $h_t$ ) and three gates: forget, input, and output. The GRU is a leaner reparameterization that keeps one state: it merges forget and input into a single update gate $z_t$ whose two branches are tied as $z_t$ and $1 - z_t$ , deletes the output gate so the full state is exposed to the next layer, and adds a reset gate $r_t$ that suppresses the previous state inside the candidate. The practical consequence is three weight blocks instead of four, so roughly 25% fewer recurrent parameters and matmuls per step, plus one less tensor to carry and initialize. Accuracy is usually close (Chung et al. 2014; Greff et al. 2017), so the choice is driven by data size, latency, and whether the task needs the LSTM’s unbounded memory.

(1) Gate Inventory: LSTM has forget, input, and output gates; the GRU has update and reset only, because tying the write and erase decisions into $z_t$ and $1 - z_t$ removes one gate and dropping the output gate removes another.
(2) One State vs Two: the LSTM’s protected cell state $c_t$ is never squashed on the recurrent path and is only revealed through the output gate, while the GRU’s single $h_t$ is both the memory and the emitted representation.
(3) Parameter and Compute Ratio: with input width $D$ and hidden width $H$ , the GRU costs exactly $3/4$ of the LSTM’s recurrent parameters and per-step FLOPs, which matters most when the recurrent stack dominates the model.
(4) Expressivity Gap: at finite precision, the LSTM’s additive, unsquashed cell state can implement a genuine counter, whereas the GRU’s convex-combination update keeps the state inside $[-1,1]$ and can only approximate counting (Weiss et al. 2018).
(5) How to Choose: prefer the GRU for small or medium datasets, short-to-medium sequences, and tight latency or memory budgets; prefer the LSTM for long sequences, large corpora, and tasks that need precise long-horizon bookkeeping such as language modeling.

Two side-by-side cell diagrams: the LSTM panel shows a cell-state track with multiply and add operators fed by forget and input gates plus an output gate producing h_t; the GRU panel shows a single hidden-state track with an update gate feeding both branches and a reset gate feeding the candidate

Figure 1: The LSTM keeps a separate cell-state track written by forget and input gates and read out through an output gate; the GRU folds everything onto one hidden-state track, where the update gate supplies both mixing weights and the reset gate only conditions the candidate.

Mathematical Formulation:
$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$
$h_t = o_t \odot \tanh(c_t)$
$\tilde{h}_t = \tanh(W_h x_t + r_t \odot U_h h_{t-1} + b_h)$
$h_t = (1 - z_t) \odot \tilde{h}_t + z_t \odot h_{t-1}$
$P_{\mathrm{LSTM}} = 4(HD + H^2 + H)$
$P_{\mathrm{GRU}} = 3(HD + H^2 + H)$

Where:

$h_t$ is the hidden state emitted at step $t$ and $c_t$ the LSTM’s internal cell state; both start from zero at $t = 0$ .
$x_t$ is the input at step $t$ , with $D$ its width and $H$ the hidden width.
$f_t, i_t, o_t$ are the LSTM forget, input, and output gates and $\tilde{c}_t$ its candidate, each of the form $\sigma(Wx_t + Uh_{t-1} + b)$ (with $\tanh$ for the candidate).
$z_t$ and $r_t$ are the GRU update and reset gates, both sigmoid, and $\tilde{h}_t$ is its candidate state.
$\odot$ is the elementwise product, $\sigma$ the logistic sigmoid mapping to $(0,1)$ , and $W, U, b$ the input, recurrent, and bias parameters of one gate block.
$P$ counts recurrent-cell parameters only, so the ratio $P_{\mathrm{GRU}}/P_{\mathrm{LSTM}} = 3/4$ holds for any $D$ and $H$ .

Grouped bar chart on a log scale of recurrent parameter counts for hidden widths 128, 256, 512 and 1024 with input width equal to hidden width: LSTM bars are 0.13M, 0.53M, 2.10M and 8.39M while GRU bars are 0.10M, 0.39M, 1.57M and 6.29M

Figure 2: Recurrent parameters per layer with $D = H$ : three gate blocks instead of four give the GRU exactly 0.75x the LSTM count at every width, so at $H = 512$ one layer holds 1.57M against 2.10M.

Property	GRU	LSTM
Gates	2: update, reset	3: forget, input, output
State carried	Hidden state only	Hidden state plus cell state
Weight blocks	3	4
Params at D = H = 512	1.57M	2.10M
Memory exposure	Whole state is exposed downstream	Output gate filters what leaves the cell
Unbounded counting	No: state stays bounded by the convex update	Yes: additive, unsquashed cell state
Where it tends to win	Small data, short sequences, tight latency	Long sequences, large corpora, language modeling

August 14, 2026

DL0062 LSTM
What is an LSTM (Long Short-Term Memory) network? How does it address vanishing gradients?
Answer
A Long Short-Term Memory network is a gated recurrent neural network designed to carry useful information across many sequence steps. Each cell maintains a cell state and uses forget, input, and output gates to control what is retained, written, and exposed. Its additive cell-state update creates a shorter gradient path than repeatedly multiplying through a vanilla RNN’s nonlinear state transition, so gradients can remain useful when forget gates stay near one. LSTMs mitigate vanishing gradients rather than eliminating them: saturated gates, long products of forget factors, and poor optimization can still weaken learning.
(1) Two Recurrent States: The cell state $c_t$ is the long-term memory path, while the hidden state $h_t$ is the exposed representation used by the next step and downstream layers.
(2) Gated Update: The forget gate scales old memory, the input gate controls a candidate update, and the output gate selects how much of the updated memory becomes visible.
(3) Gradient Preservation: The derivative along the direct cell-state path contains products of forget gates instead of repeated full recurrent Jacobians; values near one preserve gradient flow.

Figure 1: One LSTM timestep: gated reads and writes surround an additive cell-state highway, while the output gate produces the hidden state.
Mathematical Formulation:
$f_t=\sigma\!\left(W_f[x_t,h_{t-1}]+b_f\right)$
$i_t=\sigma\!\left(W_i[x_t,h_{t-1}]+b_i\right)$
$\tilde c_t=\tanh\!\left(W_c[x_t,h_{t-1}]+b_c\right)$
$c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t$
$o_t=\sigma\!\left(W_o[x_t,h_{t-1}]+b_o\right)$
$h_t=o_t\odot\tanh(c_t)$
Where:
$t$ is the timestep; $x_t$ is the current input, and $h_{t-1},c_{t-1}$ are the previous hidden and cell states.
$f_t,i_t,o_t\in(0,1)$ are element-wise forget, input, and output gates; $\tilde c_t$ is the candidate cell update.
$c_t$ is the updated long-term cell state and $h_t$ is the exposed hidden state.
$W_f,W_i,W_c,W_o$ and $b_f,b_i,b_c,b_o$ are learned affine parameters; $[x_t,h_{t-1}]$ denotes concatenation.
$\sigma$ is sigmoid, $\tanh$ is hyperbolic tangent, and $\odot$ is element-wise multiplication; the direct derivative includes $\partial c_t/\partial c_{t-1}=f_t$ .

Figure 2: Why LSTMs mitigate vanishing gradients: a vanilla RNN repeatedly multiplies full nonlinear Jacobians, whereas the LSTM provides a gated direct memory path.
Login to view more content
July 20, 2026

Tag: LSTM

DL0074 GRU vs LSTM

DL0062 LSTM