DL0080 Tokenization in LLMs

Tokenization is the reversible mapping between raw text and the sequence of integer ids a language model actually consumes, drawn from a fixed vocabulary of size . Modern LLMs use subword vocabularies that sit between characters (short vocab, very long sequences) and words (huge vocab, unavoidable out-of-vocabulary gaps), and the dominant recipe is byte-level BPE: count adjacent symbol pairs over a training corpus, greedily merge the most frequent pair, and repeat until the vocabulary reaches its target size. Because the base alphabet is the 256 possible bytes rather than Unicode characters, any input is representable and there is no UNK token, at the cost of spending several tokens per character on scripts the merge table never learned. The tokenizer is frozen before pretraining and is effectively part of the architecture: it fixes the embedding matrix and output projection ( parameters), it decides how many model passes a sentence costs, and it determines whether the model sees 1024 as one symbol or three. Most surprising LLM behaviors around arithmetic, non-English cost, and trailing whitespace trace back to this layer rather than to the transformer itself.

(1) Subword Is A Middle Ground: character-level models keep the vocabulary tiny but inflate sequence length, and attention is in that length; word-level vocabularies need hundreds of thousands of entries and still miss rare words and typos.
(2) How BPE Is Trained And Applied: training produces an ordered merge list; encoding replays those merges greedily on each pretokenized chunk, so segmentation is deterministic and not a search over the best split.
(3) Pretokenization Comes First: in the GPT-style recipe a regex splits text on whitespace, punctuation, and digit groups before BPE runs, which is why the leading space belongs to the token (" the" and "the" are different ids) and why, with standard pretokenization, merges do not cross word boundaries; that constraint comes from the pretokenizer rather than from BPE itself, and implementations that skip or relax it (see SuperBPE below) do learn multi-word tokens.
(4) Vocabulary Size Is A Compute Knob: raising lowers fertility (tokens per word) so each sentence needs fewer forward passes, but it grows the embedding and softmax cost, which is why the field drifted from 32k to 128k vocabularies (Llama 2 → Llama 3) as models got larger.
(5) The Failure Modes Users Notice: digit grouping degrading arithmetic, non-English prompts costing two to three times more tokens for the same content, glitch tokens whose embeddings were barely trained, and special chat-template ids that must never be injectable from user text.

Figure 1: Illustrative fertility gap: with a merge table dominated by English text, the same content in Hindi consumes roughly more tokens, so it costs more per request and reaches the context limit sooner.

Two implementation details matter more in production than the choice of merge algorithm. First, digit handling: tokenizers that merge arbitrary digit runs give the model inconsistent units for numbers, so 2024, 202, and 24 share no stable structure and column-wise arithmetic has to be learned per token; Llama 3 and several recent tokenizers instead force digits into groups of at most three, and some research tokenizers split every digit. Second, special tokens such as beginning-of-text and end-of-turn markers are inserted out of band by the chat template, not produced by BPE over user text; if a serving layer lets a user string encode into those ids, the model’s turn structure can be rewritten from inside a prompt. Related traps include unnormalized Unicode (NFC versus NFKC changes ids for accented text), tokenizer and checkpoint version skew, which silently shifts every embedding lookup, and untrained vocabulary rows that surface as glitch tokens the model cannot repeat back.

Mathematical Formulation:

Where:

• is the pair chosen at each BPE training step, and is its frequency as adjacent symbols in the corpus; the merge is appended to an ordered list that encoding later replays.
• is fertility: tokens produced for whitespace words, measured per language and per corpus, so it is a property of the tokenizer and the text, not of the model.
• counts the input embedding plus the untied output projection for vocabulary and hidden width ; tying the two halves this term but couples input and output geometry.
• is the compute per word of text, where is the per-token cost of everything except the vocabulary layers; the two -dependent effects pull in opposite directions, which is why an interior optimum in exists and grows with model size.

Figure 2: Illustrative trade-off for a 1B-scale model with : shorter sequences from a larger vocabulary stop paying for themselves once the output-layer cost rivals the transformer body, and the minimum shifts right as and depth grow.