Fine-tune by adding, not rewriting

figure 1 · the whole idea

Fine-tune by adding, not rewriting

Fine-tuning a large model the normal way updates every one of its weights. Do that for five tasks and you have five full copies of a very large model to store, version, and serve. As models grew into the hundreds of billions of parameters, that stopped being practical.

LoRA's move is to leave the pretrained weights completely alone — frozen — and learn a small, separate correction to sit beside them. The big model already knows almost everything; a new task only needs a nudge, so why rewrite the whole thing to express it?

click any figure to zoom · scroll to magnify · drag to pan

This is the third answer in the making models smaller arc, and the sideways one. Distillation trains a new small model; the lottery ticket finds a small one hiding inside; LoRA keeps the big model whole and shrinks the fine-tune instead. All three chase the same question — how little is actually necessary?

figure 2 · low rank

The update is low-rank

Here is the observation everything rests on. When you fine-tune, the change to a weight matrix — call it ΔW — turns out to have very low intrinsic rank. In plain terms: the fine-tune moves the weights in only a handful of directions, not in all of them. And any low-rank matrix can be written as a tall-thin matrix times a wide-short one.

So instead of learning the full ΔW, LoRA learns those two small factors, B and A, and takes their product as the update. A full matrix is d×d weights; the two factors are only 2·d·r, and the rank r is tiny — often 1, 2, or 8. That single substitution is the whole method.

figure 3 · the forward pass

How a layer runs

In the model, nothing dramatic happens. The input runs through the frozen weight matrix as always, and also through the little pair B·A, and the two results are simply added.

The frozen path carries what the model already knew; the trained path carries the task-specific correction. One nice detail: B is initialized to zero, so at the very start the correction is nothing at all — training begins exactly at the pretrained model and only departs as far as the task demands.

figure 4 · the parameter count

The saving, by hand

Take one attention weight matrix in a large model, with dimension 4096, and a LoRA rank of 8. Count the trainable weights both ways.

Two hundred and fifty-six times fewer, for that one matrix. Across an entire 175-billion-parameter model the paper reports on the order of 10,000× fewer trainable parameters — a 350 GB checkpoint becomes a 35 MB adapter — with no measurable loss in quality versus full fine-tuning.

figure 5 · merge and swap

No inference tax, and a shelf of specialists

There's a second advantage that's easy to miss. Because B·A has the same shape as W, once training is done you can just add them together into a single matrix W′. At inference there's one matrix where there was one before — zero extra latency. Adapter methods that insert new layers cannot say the same.

Or don't merge, and keep the adapters separate: one frozen base loaded on the GPU, and a shelf of tiny per-task adapters you swap in by loading a few megabytes. That's what makes running many specialists on shared hardware affordable.

the equations · worked

The forward pass, and what it costs to train

applied · our reading

The arc's practical endpoint

For sovereign, domain-specific systems this is the most directly useful paper in the whole making models smaller arc. Train and freeze one base model on-premise. Then, for each task — tax, customs, payroll — fine-tune a small adapter, version it like a piece of code, and swap it in place. Nothing large ever leaves the box, and a new specialist costs megabytes, not a model.

the caveat we would flag LoRA reshapes directions the base already has — it adapts existing knowledge cheaply, but it can't install capacity the base fundamentally lacks. If a domain needs knowledge the base never saw, an adapter won't conjure it. And the rank r is a genuine knob: set it too low and a hard task quietly underfits, with no error to warn you.

next in the arc

Where this goes

That completes making models smaller — distillation, the lottery ticket, and LoRA, three routes to the same end. The next arc, data & scaling laws, steps back to ask the prior question: given a fixed compute budget, how big should the model be and how much data should it see? That's Chinchilla.