Better data beats a bigger model

figure 1 · the whole idea

A 1.3B model that codes like a 15B one

Everyone had been scaling the same way: more parameters, more data scraped from the web, more compute. This paper asked a heretical question — what if the bottleneck isn't how much data but how good it is? So they built a small model, phi-1, and fed it a tiny, carefully curated diet of "textbook-quality" code instead of the usual web dump.

It out-coded models an order of magnitude larger. On HumanEval — the standard "can it write a correct function from a description" benchmark — phi-1 at 1.3B parameters beat StarCoder at 15.5B, despite seeing a tiny fraction of the data.

click any figure to zoom · scroll to magnify · drag to pan

This is the counterweight to the scaling-laws story in the rest of this arc. Chinchilla tells you the optimal ratio of model size to data volume; phi-1 tells you that a good enough teacher can move the curve itself. Both are true, and they pull in useful tension.

figure 2 · the raw material

What "textbook quality" means

The whole paper turns on one intuition, so it's worth making concrete. Most code on the internet runs but teaches nothing: no context, cryptic names, no explanation of why. A textbook example is the opposite — self-contained, clearly named, documented, with a worked instance. Same concept, wildly different value to a learner.

A model learns the patterns in its data. Feed it landfill and it learns to imitate landfill; feed it a curriculum and it learns the concepts. The bet is that a training set should read like a textbook, and everything else in the method is just how you assemble one at scale.

figure 3 · the data recipe

Filter, then fabricate

First, filtering. They took a big corpus of public code and trained a small classifier to score each file for "educational value" — does this read like it belongs in a textbook? — then kept only the top slice. That alone throws away the vast majority of the data.

Second, synthesis. A larger model (GPT-3.5) was prompted to write new textbook-style material: explanations, and a set of clean worked exercises. Because it's generated to be instructive, it's clean by construction.

The combined training set — filtered code, synthetic textbooks, synthetic exercises — comes to roughly 7 billion tokens. Next to the trillion-token corpora everyone else was using, it's a rounding error. The cleverness is entirely in the classifier and the generator; the training run itself is completely ordinary.

figure 4 · the efficiency

Smaller, and by two orders of magnitude less data

It's worth seeing the sizes side by side, because the gap is not subtle. phi-1 is smaller on every axis that usually predicts performance, and better on the one that matters.

Around 150× fewer training tokens, about 12× fewer parameters — and a higher score. When the inputs shrink on every axis and the output goes up, the cause isn't scale. It's what the tokens contain.

figure 5 · the jump

A small, clean fine-tune reorganized what it could do

The most striking result is almost an afterthought in the method. Before the final step, phi-1-base — already trained on textbooks and filtered code — is good, not great. Then they fine-tune it on CodeExercises, a small set of clean worked problems, and the score nearly doubles.

What makes it more than a data point: the fine-tuned model gained abilities the exercises never demonstrated — using libraries and patterns absent from that small set. A modest, high-quality fine-tune didn't just add facts; it seems to have reorganized the capabilities already latent in the base model. Quality, it turns out, compounds.

the equations · worked

How the score is scored, and what the run cost

applied · our reading

The most actionable lesson in the arc

For a specialist system in a regulated domain, this is the paper that changes what you do on Monday. The instinct is to scrape more source material — more circulars, more rulings, more forms. phi-1 says the opposite: a smaller set of curated, worked examples — the domain rendered as a textbook — can beat a far larger pile of raw documents.

the caveat we would flag The textbooks here were authored by a larger model, so the method quietly assumes a capable teacher — and a synthetic curriculum can inherit that teacher's mistakes, or leak the very benchmark you mean to test on. In a domain where a wrong answer is a compliance event, the curation has to be verified by a human who knows the law, not just generated and trusted.

next in the arc

Where this goes

phi-1 argues that data quality can move the curve. The next two entries in data & scaling laws pin down the curve itself: Chinchilla derives the compute-optimal ratio of model size to training tokens, and Beyond Chinchilla corrects it once you count the cost of serving the model, not just training it.