DL0039 Transformer Weight Tying

Weight sharing (weight tying) reuses the same parameters in multiple places instead of learning separate matrices. The most common form ties the input embedding matrix with the output projection that produces vocabulary logits: one matrix maps token IDs into -dimensional space at the bottom and maps hidden states back to logits at the top. A stronger form (ALBERT) shares weights across all Transformer layers, trading capacity for extreme parameter efficiency.

(1) Embedding–Output Tying: The output logits are computed with the transpose of the input embedding, , saving an entire matrix.
(2) Consistency Benefit: Input and output spaces share one geometry: tokens that embed close together also get similar output preferences, which acts as a useful regularizer.
(3) Layer Sharing (ALBERT): One block’s weights are reused for every layer: depth comes from recurrence through the same block, not from new parameters.

Mathematical Formulation:

Where:

• is the shared embedding matrix, used to look up inputs and, transposed, to score outputs.
• is the final hidden state; is the logit of vocabulary token ; is vocabulary size.

Figure 1: One matrix, two jobs: embeds input tokens at the bottom and, transposed, projects hidden states to logits at the top.

How Much It Saves: The untied output projection is a full matrix; for a 50k vocabulary at that is ~51M parameters, often the single largest matrix in a small model. Tying removes it entirely while typically costing little or no accuracy.

Figure 2: Tying deletes the output projection matrix, one of the largest single blocks of parameters in vocabulary-heavy models.

Vocabulary Mismatch Caveat: Tying assumes input and output share a vocabulary. When source and target vocabularies differ (multilingual translation), only the decoder input embedding and output projection can be tied; the encoder embedding stays separate.