DL0088 PEFT: Parameter-Efficient Fine-Tuning

All three families freeze the pretrained weights and train a tiny set of new parameters, but they differ in where those parameters live, and that single structural choice decides inference cost, serving flexibility, and achievable quality. LoRA adds a low-rank update in parallel to existing weight matrices, so after training the update folds back into and costs exactly zero extra latency. Serial adapters insert a small bottleneck MLP inside the block’s forward path, which keeps task modules cleanly composable but adds depth, so every decoded token pays for them. Prefix and prompt tuning touch no weight matrix at all: prefix tuning prepends trainable key/value vectors at every layer, prompt tuning prepends trainable vectors only at the input embeddings, which is the cheapest option in parameter count but the least stable and it consumes context length. In practice LoRA is the default (paired with 4-bit quantization as QLoRA when GPU memory is the binding constraint), adapters win when modular composition matters more than latency, and prefix/prompt tuning wins when the base weights must stay byte-identical or when you want per-request conditioning.

(1) Where The Parameters Live: LoRA sits parallel to chosen projections, adapters sit serially between sublayers, prefix tuning sits in the KV cache rather than in any weight tensor.
(2) Mergeability Decides Latency: only LoRA is exactly mergeable into ; adapters add two matmuls and a nonlinearity per sublayer that cannot be folded away, which is a 10-25% decode overhead at batch 1 on a 7B model.
(3) Capacity Is A Knob, Except For Prompts: rank and bottleneck width scale capacity smoothly, while prompt tuning’s capacity saturates and only becomes competitive with full fine-tuning above roughly 10B parameters.
(4) Multi-Tenant Serving: unmerged LoRA and prefix tuning both support heterogeneous batching (per-request low-rank gather, or per-request KV prefix) from one frozen base copy; merged LoRA needs a separate weight copy per task and adapters need a weight swap per request.

Figure 1: The three injection sites in one block. LoRA is parallel and therefore mergeable, the adapter is serial and therefore always in the critical path (real adapter recipes place one after attention and one after the FFN), and prefix tuning changes only K and V, leaving every weight untouched.

Two consequences are worth stating precisely because interviewers probe them. First, the merge is algebraically exact, not an approximation: is just another matrix of the same shape, so a merged LoRA checkpoint is indistinguishable from a normally fine-tuned one at serving time, and the price is that you can no longer swap tasks without unmerging. Second, PEFT saves optimizer and gradient memory, not activation memory: with Adam you drop from roughly 12-16 bytes per trainable parameter of state to almost none, which is why a 7B model fine-tunes on a single 24GB card, but activations still scale with batch size and sequence length, and QLoRA’s 4-bit NF4 base plus paged optimizers is what pushes a 65B model onto one 48GB GPU. Prefix tuning has a third quirk: because the prefix occupies KV slots, a 20-token prefix at every layer permanently shortens the usable context and its quality is non-monotonic in prefix length, which is why the original work needed a reparameterization MLP to train stably at all.

Mathematical Formulation:

Where:

• is the sublayer output and the input; is the frozen pretrained matrix of shape .
• and are the LoRA factors with rank ; is Gaussian-initialized and is zero-initialized so training starts from the pretrained function.
• is a scaling constant, and the ratio keeps the effective update magnitude roughly stable when you sweep ; is the merged weight used at inference.
• and form the adapter bottleneck of width , with nonlinearity ; the residual term plus near-zero initialization makes the adapter an approximate identity at step 0.
• are the per-layer, per-head prefix key and value vectors of length ; are the ordinary projected keys and values, so attention runs over positions for a sequence of length .

Figure 2: Illustrative budgets for a 7B decoder with and 32 layers. Parameter counts span three orders of magnitude, yet the latency picture is unrelated to them: the smallest method is not the fastest, and merged LoRA is free.