DL0174 Wide vs Deep LLM Architectures

A decoder layer costs about parameters in the attention projections and about in the MLP, so a fixed budget pins the product and the only remaining freedom is the aspect ratio . Cutting depth by 4x buys only 2x width, which is why the two axes are so asymmetric in practice. Depth buys serial computation: each layer is one more composition step, and multi-hop reasoning and compositional generalization improve with layers in a way that width cannot replicate. Width buys parallel capacity and hardware efficiency: larger GEMMs, higher arithmetic intensity, clean tensor-parallel sharding, and a smaller KV cache per token because the cache grows linearly in . Empirically the pretraining loss is remarkably insensitive to shape across the middle of the range, so the decision is usually settled by systems constraints rather than by quality. Depth is the axis that costs you latency and training stability, and width is the axis that costs you head redundancy, embedding budget, and eventual depth starvation.

(1) The Budget Is Quadratic In Width: the iso-parameter constraint is , so width is an expensive axis and depth is a cheap one. Doubling forces you to delete three quarters of the layers.
(2) Depth Is Serial Expressivity: a layer is one composition step, and tasks requiring chained inferences need enough layers to host them. Depth-efficiency saturates, though, with the useful depth for a given width growing only logarithmically in .
(3) Width Is Capacity And Throughput: knowledge storage tracks total parameters rather than shape, but wide layers give larger matrix multiplies, better MFU, and more heads per layer for parallel feature lookup.
(4) Depth Destabilizes Training: residual variance accumulates over branches, so deep stacks need output-projection init, Pre-LN or DeepNet-style residual scaling, and they show more loss spikes and attention rank collapse.
(5) Depth Is A Latency Multiplier: autoregressive decode pays a fixed per-layer serial cost (kernel launches plus two tensor-parallel all-reduces), so adds directly to time per token even though total FLOPs are unchanged.
(6) Small Budgets Flip The Answer: the embedding matrix costs , which at a 128k vocabulary swamps a sub-billion model, so on-device work favors deep and thin shapes.

Figure 1: Every shape on the curve costs the same parameters. The curve is a straight line in log-log space with slope , which is the whole trade-off in one picture: width is quadratically expensive, so moving from 64 layers to 16 layers pays for only a 2x wider residual stream. Real production shapes cluster in the middle band, roughly between 60 and 160.

The stability side is where depth stops being free. In a Pre-LN residual stack the output is a sum of branch contributions, so without correction the variance entering the final norm grows roughly linearly in depth, gradients through the early layers get squeezed, and the effective learning rate per layer drifts as you add layers. The standard fixes are scaling the residual output projections by at init, DeepNet-style residual weighting for very deep stacks, and QK-norm to stop attention logits from drifting. Deep stacks also suffer token-uniformity (rank) collapse, where repeated pure attention drives representations toward a rank-one subspace and the MLPs plus residual path are what hold it off. Width has the opposite profile. Hyperparameter transfer across width is well understood through muP, so you can tune a small proxy model and reuse the learning rate, whereas transfer across depth needed its own theory. The failure mode of extreme width is not divergence but waste: many heads become redundant, the per-layer computation is over-provisioned relative to the number of serial steps available, and the model has no place to put a long chain of dependent inferences.

Mathematical Formulation:

Where:

• is the non-embedding parameter count, from of attention projections plus of MLP weights per layer (a SwiGLU MLP with hidden width matches the same count).
• is depth and the residual width; a fixed budget holds constant, so 4x fewer layers buys 2x width.
• is the aspect ratio, the single knob this question is really about. GPT-3 175B sits at .
• is KV-cache bytes per token, with key-value heads, head dimension , and bytes per element. Under GQA with a fixed head count it is proportional to alone.
• is per-token decode latency, the weight bytes read per step, the achievable HBM bandwidth, and the fixed serial cost per layer from kernel launches and collective synchronization.
• Because is set by the budget and not by the shape, the term is identical across shapes and only the term differs.

Iso-Parameter Shapes At About 7.2B Non-Embedding Parameters:

These three shapes are interchangeable on the training-cost sheet and nearly interchangeable on validation loss, yet they behave very differently in a serving stack. With GQA at 8 key-value heads and in fp16, each layer stores 4 KiB per token, so the 64-layer shape carries 4x the KV cache of the 16-layer shape and supports a proportionally smaller batch at the same context length. On the latency side, a plausible tensor-parallel-8 deployment has roughly microseconds per layer once you count two all-reduces plus launch overhead, which turns depth into 2.56 ms of pure serial cost at 64 layers against 0.64 ms at 16 layers, on top of an identical weight-read term.

Figure 2: Same parameters, same FLOPs, 2.2x difference in single-stream decode latency. The green weight-read term is fixed by the budget, so all of the spread comes from the serial term. The right panel shows the second penalty of depth: KV cache is linear in , so the deep shape gives up 4x in memory per sequence and therefore in achievable batch size.