Most big models are undertrained

figure 1 · the whole idea

Most of the giants were trained wrong

For a few years the recipe for a better language model was simple: make it bigger. GPT-3 had 175 billion parameters; Gopher had 280 billion. This paper asked whether all those parameters were actually being used well — and found that they weren't. The giants had been fed far too little data for their size.

To prove it, DeepMind took the same compute budget that trained the 280B Gopher and instead trained a 70B model — a quarter the size — on more than four times the data. They called it Chinchilla, and it beat Gopher across the board.

click any figure to zoom · scroll to magnify · drag to pan

Note the exact claim, because it's precise: not "smaller is better" in general, but "for a fixed compute budget, the field had been choosing the wrong point on the size-versus-data trade-off." Spend the same money differently and you get a better model.

figure 2 · the trade-off

A fixed budget is a pie you have to split

The reason a trade-off exists at all is a piece of arithmetic. Training compute is roughly 6 × parameters × tokens. So for a fixed budget, the product of model size and data is fixed: every parameter you add is a token you don't get to train on, and vice versa.

That means there's a sweet spot. Too small, and the model can't absorb what the data has to teach. Too big, and it's starved — it never sees enough tokens to fill its capacity. The whole paper is a careful search for where that peak sits.

figure 3 · isoFLOP curves

One budget at a time

The method is more convincing than any single model. For a given compute budget, they trained many models of different sizes and recorded each one's final loss. Plot loss against model size and you get a U — too small underfits, too big starves, and the bottom of the U is the best size for that budget.

Do it for several budgets and the bottoms of the U's line up. Joining them gives a clean rule: as you scale compute, the optimal model size and the optimal amount of data grow together, at roughly the same rate. Three independent approaches in the paper all agree on it.

figure 4 · ~20 tokens per parameter

About twenty tokens per parameter

Distilled to a rule of thumb: a compute-optimal model should see roughly 20 training tokens for every parameter. A 70B model wants about 1.4 trillion tokens — which is exactly what Chinchilla got. Now apply the same rule to the models of the day.

GPT-3 and Gopher wanted trillions of tokens and got 300 billion — undertrained by more than ten times. The field had been pouring its budget into parameters and leaving the models hungry. Model and data are meant to grow in lockstep, and the ratio had been badly backwards.

figure 5 · the result

Right-sizing pays twice

Chinchilla didn't just edge out Gopher — it beat it across language modelling, reading, reasoning, and the MMLU knowledge test, at a quarter of the size.

And there's a bonus that keeps paying. Because the compute-optimal model is smaller, it's also cheaper to run — about a quarter the inference cost, every single time you serve it, forever. Better on the day it ships and cheaper on every day after: the case for right-sizing is hard to argue with.

the equations · worked

The loss law, then the optimum under a budget

applied · our reading

The compass, and its fine print

For a domain model the lesson is to resist the reflex to reach for more parameters. Balance size against data; treat ~20 tokens per parameter as a starting point. But two pieces of fine print matter as much as the rule.

the fine print First, Chinchilla optimizes training compute only — it says nothing about the cost of serving the model afterwards. If you'll run a specialist millions of times, the optimum shifts smaller still, which is exactly what the next entry works out. Second, recall phi-1: better data can move this whole curve, so 20:1 is a compass, not a law.

next in the arc

Where this goes

Chinchilla found the compute-optimal balance for training. Beyond Chinchilla asks the obvious next question: once you also pay to serve the model — on every query, for its whole lifetime — the sums change, and the answer is to train an even smaller model on even more data.