Teaching a small model to think like a big one

figure 1 · the whole idea

The wrong answers are the lesson

You have a big, accurate model and you wish it were small — cheaper to run, able to live on a phone or inside a firewall. The obvious plan is to train a small model on the same labelled data. It never works as well: the small model doesn't have the capacity to rediscover everything the big one learned.

Distillation's move is unexpected. Don't train the small model on the labels — train it on the big model's full answer. When the big model looks at a handwritten 2, it doesn't just say "2". It says: almost certainly a 2, a little bit like a 3, a faint touch like a 7, nothing like a 4. That shape — which wrong answers it finds tempting — is knowledge the one-hot label throws away.

click any figure to zoom · scroll to magnify · drag to pan

The paper's own example: shown a photo of a BMW, the big model gives a tiny probability to "garbage truck" — but many times more than it gives to "carrot". Those minuscule numbers encode how the model sees the world. Hinton calls it the dark knowledge: most of what a trained network knows is hiding in the probabilities it assigns to answers that are wrong.

figure 2 · temperature

Temperature turns up the whispers

There's a catch. The interesting probabilities — the "a bit like a 3" — are often tiny, so tiny the student barely feels them during training. The fix is a single knob called temperature, applied inside the softmax that turns the model's raw scores into probabilities.

At temperature 1 you get the model's normal, confident answer. Turn the temperature up and the distribution softens: the tall bar shrinks, the small bars rise, and the hidden structure becomes loud enough to learn from. Turn it up too far and everything blurs toward equal — so the useful setting is warm, not boiling.

Crucially, temperature never changes which answer is most likely. It only changes how much the student is told about the runners-up. The teacher and the student are heated to the same temperature during training; the student is cooled back to 1 when it's time to actually use it.

figure 3 · the training signal

Two signals, one student

If you only copied the teacher, the student would faithfully reproduce the teacher's mistakes and have no anchor to the truth. So the student learns from two losses at once.

• The distillation loss pushes the student's softened output to match the teacher's softened output — copy the whole shape, dark knowledge and all.
• The student loss pushes the student's ordinary output to match the real, ground-truth label — stay honest.

The two are added in a weighted sum. The one fiddly detail, the T² factor: softening at temperature T shrinks the distillation gradients by roughly 1/T², so you multiply that loss back up by T² to keep the two signals comparable in strength. That's the entire training recipe.

figure 4 · the toy example

The same scores, two temperatures

Softmax with temperature is just exp(zᵢ / T) for each score, divided by the sum so it adds to one. Raising T shrinks every score before the exponential, which flattens the result. Watch what that does to one digit.

Same four scores throughout. At T = 1 the "2" takes 64% and the "7" is 0.03 — so small its gradient is negligible and the student effectively never hears about it. At T = 4 the "7" rises to 0.17: now it's a real target the student is forced to fit. Nothing about the teacher changed — you just turned up the part of its answer worth copying.

the payoff At high temperature, matching the teacher's softened probabilities is almost the same as matching its raw scores directly. That's why it works: the student isn't just told the answer, it's told how far the teacher leaned toward every alternative — the full geometry of the teacher's judgment, not a single point.

figure 5 · results

It works — and it leaks knowledge you never handed over

On MNIST, a small network trained from scratch makes 146 test errors. The big network makes 67. Distilled from the big one, the same small network drops to 74 — closing almost the entire gap without gaining a single parameter.

The striking part is the second panel. The authors built a transfer set that contained no examples of the digit 3 at all, and distilled a student on it. The student still recognised 3s at test time — 98.6% correct — because the soft targets for other digits (a 5 that leans toward 3, an 8 that does too) had quietly encoded what a 3 looks like.

Think about what that means. The knowledge of a class the student was never shown transferred through the similarity structure of the other classes. The soft targets carry the teacher's whole map of how things resemble each other — far more than any set of labels could.

the equations · worked

Soften, blend, and the gradient underneath

applied · our reading

A specialist you can own

This 2015 paper is the root of every "small model, big-model quality" result since — and it's directly load-bearing for the work we do. A frontier model has judgment about tax law that a small model can't learn from scratch. Distillation is the bridge: let the big model answer your own questions, and train a small, on-premise student on the full shape of those answers.

The student ends up small enough to run inside your boundary, cheap enough to serve at scale, and carrying much of the teacher's judgment — the precondition for a sovereign specialist that never phones home.

the caveat we would flag Distillation copies the teacher's judgment including its mistakes, and a confident-but-wrong soft target is a very persuasive teacher. In a regulated domain, distil on curated, verified questions and keep the hard-label loss anchored to ground truth — so the teacher's errors can't launder themselves into the student as fact.

next in the arc

Where this goes

Distillation copies a model into a smaller one. The next two entries in making models smaller come at the same goal from other directions: the Lottery Ticket Hypothesis finds the small network already hiding inside the big one, and LoRA fine-tunes by adding a thin adapter instead of moving the weights at all. All three are answers to the same question: how little is actually necessary?