DL0081 Tokenizer Vocabulary Size Effects

Vocabulary size is one of the few hyperparameters that moves quality, memory, and throughput in opposite directions at the same time, so there is a genuine optimum rather than a “bigger is better” rule. On the cost side, the embedding and unembedding tables grow linearly in (up to parameters when untied), the output projection adds FLOPs per token, and the training-time logits tensor of shape in fp32 becomes one of the largest activations in the whole model. On the benefit side, a larger vocabulary compresses text into fewer tokens, which shortens sequences, shrinks the KV cache, cuts the attention term, and reduces the number of autoregressive decode steps needed to emit the same text. The catch is that compression improves only logarithmically in while cost grows linearly, and each added token gets a smaller slice of the training signal, so rare embeddings end up undertrained. Empirically the optimum grows with model size: 32k was reasonable for GPT-2-era models, Meta’s Llama 3 moved to 128,256 tokens, and Google’s Gemma 2 uses 256,128, while byte-level and byte-patch models sit at the opposite extreme.

(1) Memory Cost Is Linear: at and , untied tables hold about 1.05B parameters, roughly 13% of an 8B model, and 2.1 GB of bf16 weights before optimizer state.
(2) Compression Gain Is Logarithmic: going 32k → 128k buys only about 11% fewer tokens per document in English, while the embedding tables quadruple.
(3) The Output Softmax Dominates Small Models: the projection is about 7% of forward FLOPs for an 8B model but roughly a third for a 1B model with .
(4) Quality Is Non-Monotonic: too small and sequences are long and multilingual fertility explodes; too large and rare rows receive too few gradient updates, producing dead or glitch tokens.
(5) Throughput Has Two Units: tokens per second falls slightly as grows, but bytes (or words) per second usually rises because fewer steps are needed for the same text.

Figure 1: Illustrative English compression: quadrupling the vocabulary from 32k to 128k lifts bytes per token from 3.9 to 4.4, which shortens sequences by only about 11%, while the embedding tables grow 4x.

The most under-appreciated cost is not the weights but the logits activation. Cross-entropy is normally computed in fp32, so a single 8192-token sequence against a 128k vocabulary materializes bytes, about 4.2 GB, and a microbatch of four pushes past 16 GB before the backward pass. This is why large-vocabulary training runs adopt chunked or fused cross-entropy kernels that never hold the full logits matrix, and why teams sometimes cap purely for activation-memory reasons. At inference the picture flips: the vocabulary contributes a fixed GEMM per decode step, which is cheap relative to the KV-cache reads in a large model, and the shorter sequence means fewer steps overall, so end-to-end latency for generating a fixed passage typically improves. A separate trap is evaluation: validation loss is not comparable across tokenizers, because the per-token loss is defined over a different unit of text, so comparisons must be normalized to bits per byte or bits per character.

Mathematical Formulation:

Where:

• is the parameter count of the input embedding plus output unembedding; the factor 2 drops to 1 when the two are tied.
• is the vocabulary size and the model width; is the non-vocabulary (attention plus MLP) parameter count.
• is the raw text size in bytes, the compression ratio in bytes per token, and the resulting token count; and are corpus- and language-dependent fit constants.
• is approximate training FLOPs under the rule, which is why a larger both adds parameters and removes tokens.
• is the logits activation in bytes, with microbatch and precision (4 for fp32); is the compute-optimal vocabulary, whose exponent near 0.83 means it grows more slowly than the rest of the model.

Figure 2: Illustrative shape of the trade-off at fixed compute: loss measured in bits per byte is U-shaped in , and the minimum shifts right as non-embedding capacity grows, because a bigger body can afford to spend parameters on the vocabulary.