DL0032 Transformer VS RNN

Written by

What makes Transformers more parallel-friendly than RNNs?

Answer

The fundamental difference is dependency structure: an RNN computes each hidden state from the previous one, $h_t = f(h_{t-1}, x_t)$ , so step $t$ cannot start before step $t-1$ finishes. A Transformer replaces recurrence with self-attention, which scores every pair of positions simultaneously, so all tokens are processed in one parallel pass. This turns sequential loops into dense matrix multiplications that saturate modern GPUs.

(1) No Temporal Dependency: Transformers process all input tokens at once; there is no hidden-state chain forcing order.
(2) Fully Parallelizable Attention: All $n^2$ attention scores are computed in a single matrix product $QK^\top$ , and the FFN applies to all positions simultaneously.
(3) Optimized for GPUs: Large GEMM kernels keep thousands of GPU cores busy, unlike the RNN’s long chain of small dependent steps.

Mathematical Formulation:
$h_t = f(h_{t-1},\, x_t)$
$S = QK^\top$

Where:

$h_t$ is the RNN hidden state at step $t$ ; it cannot be computed before $h_{t-1}$ exists.
$S$ is the full $n \times n$ attention score matrix, produced by one parallel GEMM from the query and key matrices $Q, K$ .

Side-by-side diagram of an unrolled RNN processing inputs one cell at a time through a hidden-state chain versus a Transformer block receiving all tokens at once in parallel.

Figure 1: RNN: a serial chain where each step waits for the previous hidden state; Transformer: the whole sequence enters the block simultaneously.

Training-Time Consequence: With a sequence of length $n$ , an RNN needs $n$ sequential steps no matter how much hardware you have: latency grows linearly with sequence length. A Transformer’s forward pass is a constant number of parallel matrix operations; the cost grows in FLOPs, not in wall-clock dependency depth.

Computation graph comparison showing the RNN as a vertical chain of dependent cell evaluations versus the Transformer as three wide parallel layers of matrix operations.

Figure 2: Dependency depth: RNN needs $n$ ordered steps; the Transformer needs only $O(1)$ sequential layers, each internally parallel.

Aspect	RNN	Transformer
Token dependency	Sequential: $h_t$ needs $h_{t-1}$	None: all tokens at once
Training steps for n tokens	$n$ ordered steps	One parallel pass
Core operation	Many small vector updates	Large batched GEMMs
Long-range signal path	Through $n$ hidden states (vanishing gradients)	One attention hop, $O(1)$ path length

Why It Matters: Parallelism is why Transformers can train on web-scale corpora (a workload that would take an RNN impractically long to finish), and why attention became the default sequence model in modern NLP.

RNN Transformer

DL0032 Transformer VS RNN

Comments

Leave a Reply Cancel reply

More posts

MSD0064 Notification Ranking System

MSD0063 Image Copyright Detection

MSD0062 Ads Ranking System

MSD0061 Nearby Place Recommendation