DL0131 DoRA: Weight-Decomposed LoRA

DoRA reparameterizes every pretrained weight matrix into a magnitude vector and a directional matrix, and then lets LoRA update only the direction. Concretely it writes , where is the column-wise L2 norm, the trainable vector is initialized to , and the low-rank product is added to the direction before renormalization. The motivation comes from a weight decomposition analysis: when full fine-tuning updates a layer, the magnitude change and the direction change are negatively correlated (a large rotation with only slight rescaling, or the reverse), while plain LoRA shows a positive, roughly proportional relation, so it can essentially only push both quantities in the same direction. Giving magnitude its own parameters removes that coupling, and the renormalization makes the gradient reaching the directional component orthogonal to the current direction, which is the classic weight-normalization conditioning effect: the low-rank branch can rotate the weight without simultaneously inflating its scale. In practice this buys a learning pattern that mirrors full fine-tuning, noticeably better accuracy at low rank (the DoRA paper reports about a 3 to 4 point average gain over LoRA on the eight commonsense-reasoning benchmarks with LLaMA-7B, and matching LoRA at half the rank in its ablations), and zero inference overhead, because collapses back into one merged matrix.

(1) Two Degrees Of Freedom Instead Of One: LoRA has a single additive update ; DoRA splits the same layer into a scalar-per-column scale and a normalized direction, so scale and rotation are optimized by separate tensors.
(2) Exact Identity At Initialization: with and the adapted layer reproduces bit-for-bit, so training starts from the pretrained function with no loss spike.
(3) Gradient Is Projected, Not Just Scaled: differentiating through the column norm inserts the projector , which removes the radial component of the directional gradient and bounds its effective step.
(4) Learning Pattern Matches Full Fine-Tuning: measured over training, DoRA reproduces the negative versus slope of full fine-tuning, whereas LoRA’s slope is positive.
(5) Cheapest Where LoRA Hurts Most: the extra magnitude vector costs parameters per layer, roughly 6% on top of a rank-8 adapter for a projection, and pays off most at to .
(6) Training Overhead, Not Serving Overhead: the column norm of must be recomputed each step, which raises training memory; detaching that norm from the backward graph recovers roughly a quarter of the overhead with negligible accuracy change, and inference is unaffected after merging.

Figure 1: A DoRA layer. Only the magnitude vector and the LoRA factors receive gradients; the frozen direction is perturbed by , renormalized per column, and rescaled by . Because the final expression is a single matrix, the adapter still merges away at inference.

The stability argument is easier to see from what LoRA cannot express. In LoRA the only knob is the additive term, so any attempt to rescale a column also rotates it, and any attempt to rotate it also changes its norm; the two effects are welded together by a single rank- product, which is why its measured points fall on a line with positive slope. Full fine-tuning has free parameters and shows the opposite pattern, with subtle magnitude changes accompanying substantial directional change. DoRA recovers that behavior with extra parameters, and the practical consequences are the ones interviewers care about: lower sensitivity to the learning rate and to the LoRA scaling factor , because a badly scaled low-rank branch is renormalized away instead of blowing up the weight norm, and much flatter accuracy-versus-rank curves, because a rank-4 direction plus a full-rank magnitude still spans useful updates.

Figure 2: Measured magnitude change against direction change per adapted module. Full fine-tuning and DoRA trace a negative slope, meaning scale and rotation are traded against each other, while LoRA‘s positive slope shows the two are coupled by its single additive term.

Mathematical Formulation:

Where:

• is the adapted weight actually used in the forward pass, and is the frozen pretrained weight it starts from.
• is the trainable magnitude vector and the unnormalized direction, with and the initialization that makes at step 0.
• (initialized to zero) and are the LoRA factors with rank .
• indexes output columns, indexes training checkpoints, and takes the L2 norm of each column independently, returning a row vector.
• is that column norm, and is the orthogonal projector that strips the radial part of the gradient, so the directional update is norm-preserving to first order.
• and are the magnitude and direction change metrics whose correlation is plotted in Figure 2, with because it is one minus a cosine similarity.