knowledge base · entry 11
Once you pay to serve it, train it smaller
A read-through of the paper that corrected Chinchilla by adding the cost everyone forgot: not training the model, but serving it for the rest of its life.
every figure zooms — click it, scroll to magnify, drag to pan
part one
The idea, before any jargon
figure 1 · the whole idea
You train the model once — then serve it forever
Chinchilla answered a precise question: for a fixed training budget, what's the best model? But a model you actually deploy isn't only trained — it's run, over and over, for its whole life. Every query costs a little compute, and those littles add up without bound.
This paper adds that second term to the equation. Total cost is training plus inference-over-the-lifetime — and for anything popular, the second dwarfs the first. Optimize the whole bill instead of just the training slice, and the "optimal" model changes.
click any figure to zoom · scroll to magnify · drag to pan
It's a small conceptual addition with a large consequence. Nothing about Chinchilla was wrong; it just answered a different question than the one a deployer actually faces.
figure 2 · the shift
Count inference, and the sweet spot slides smaller
Recall Chinchilla's size-versus-quality hill. Its peak — the compute-optimal point — was found counting training cost alone. Add the cost of serving, and the point you actually want moves: toward a smaller model, which is cheaper to run, trained on more data to make up the quality.
And it's not a fixed shift. The more queries you expect to serve over the model's life, the heavier the inference term, and the further left the optimum slides. Your expected usage is now an input to how big the model should be.
part two · the mechanism
How you buy quality more cheaply
figure 3 · overtraining
Two roads to the same quality
The move that makes this work is "overtraining": pushing a small model far past Chinchilla's data-optimal point. You can reach a given quality target two ways — a big model trained the "right" amount, or a small model trained much longer. They end up equally good.
But they are not equally cheap to run. The small model costs less on every query, and you pay the extra training exactly once. If you'll serve the model a lot, that's a trade worth making — spend more up front to spend less on every call forever after.
part three · the number
When it actually pays
figure 4 · the crossover
It's a crossover
Total cost is training-once plus per-query-cost times number-of-queries. Plot total cost against how much you'll serve the model, for a big model versus a small overtrained one, and you get two straight lines that cross.
Below the crossover — a model you'll run rarely — Chinchilla's answer is right, because training cost dominates. Above it — the regime any widely-used model lives in — the smaller, longer-trained model is cheaper for its entire life. The paper works out where that crossover sits.
part four · the evidence
This already happened
figure 5 · the result
Why the models you use are "overtrained"
The clearest proof is the models themselves. Chinchilla said ~20 tokens per parameter. The models people actually run are trained far past that, and each generation pushes further.
Llama-2 7B saw about 290 tokens per parameter; Llama-3 8B, on the order of a thousand. By Chinchilla's training-only rule these are wildly "overtrained" — and that's exactly the point. They're served constantly, so a smaller model earns its extra training back on every query. This paper is the theory those choices were reaching for.
the math · in the paper's notation
One extra term, one changed optimum
The whole correction is a second term in the cost and a swap of what's being minimized. Both equations decoded, with a serving scenario pushed through them by hand.
the equations · worked
The lifetime bill, and the new question to minimize
§2 · total lifetime compute
Ctotal = 6 · N · Dtr + 2 · N · Dinf
read aloud: the training term you know from Chinchilla, plus two FLOPs per parameter for every token the model will ever serve. Inference costs 2 rather than 6 because there is no backward pass — but Dinf grows without bound for as long as the model is in production, and that is the term everyone had been leaving off the invoice.
- Nparameters — note it multiplies both terms, which is why shrinking it pays twice
- Dtrtraining tokens — paid once
- Dinflifetime inference tokens — your expected serving volume, now an input to model sizing
§2 · the flipped optimization
minN, Dtr Ctotal(N, Dtr, Dinf) subject to L(N, Dtr) = ℓ
read aloud: fix the quality you need, then find the cheapest lifetime way to buy it. Chinchilla fixed the budget and maximized quality; this paper fixes quality — using Chinchilla's own fitted L(N, D) from the previous entry — and minimizes the whole bill. As Dinf grows, the solution slides to smaller N and larger Dtr: the same quality, carried by a cheaper-to-run model trained past 20:1.
- ℓthe target loss — the quality bar your product actually needs
- L(N, D)Chinchilla's parametric loss law, reused as the quality constraint
part five · what we take from it
Sizing a specialist you'll actually run
applied · our reading
Let expected usage set the size
This closes the budgeting picture. Chinchilla sizes the training run; this paper corrects it for the serving that follows. For a domain specialist the decision is simple to state: if you'll answer millions of queries, train it small and long; if it's a rarely-used or one-off model, Chinchilla's sizing still holds.
the caveat we would flag Overtraining is not a free lunch — it hits diminishing returns and eventually runs out of fresh, high-quality data, and low-quality repetition can actively hurt. The compass is "size the model for your expected serving volume", not "always go smaller". Paired with Chinchilla and phi-1, you have the three levers: how big, how much data, and how good the data is.
next in the arc
Where this goes
That closes data & scaling laws. The final arc, frontiers, turns from how to train the standard recipe well to what might replace it: reasoning learned from reward (DeepSeek-R1), and sequence models that drop attention altogether (Mamba).
source
Sardana & Frankle — MosaicML / Databricks (2023). Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. arXiv:2401.00448. The tokens-per-parameter figures are from the named models; the cost curves and the crossover are our own illustration of the paper's argument.
← back to the knowledge base · ⚙ the illustrated decoder · ▶ run the forward pass
attention.sh — a division of Jinacode Systems. Transformers,
fine-tuning, and sovereign LLMs.
we're hiring the curious — hello@attention.sh