The three-step template every assistant follows

figure 1 · the misalignment gap

Predicting text is not following instructions

GPT-3 was trained to do exactly one thing: continue text the way the internet would. Ask it “Explain the moon landing to a six-year-old” and a plausible internet continuation is… more requests like yours, or an essay written for adults. The capability to answer is in there; the objective never asked for it. The paper names this the gap between the training objective and the user’s intent — the model is not on your side yet.

click any figure to zoom · scroll to magnify · drag to pan

Everything that follows is machinery for closing that gap with human judgment as the training signal — first by imitation, then by comparison.

figure 2 · the template

SFT, then a reward model, then RL

The pipeline runs in three stages, each with its own modest dataset written or judged by a team of about 40 trained labelers. Step one: supervised fine-tuning on ~13,000 demonstrations — labelers write the answer they wish the model gave (entry 14 in industrial form). Step two: train a reward model on ~33,000 prompts’ worth of ranked comparisons. Step three: let the model generate freely on ~31,000 fresh prompts and optimize it with reinforcement learning against the reward model’s score.

Notice the quantities: tens of thousands, not millions. The scarce, expensive ingredient is calibrated human judgment — the entire post-training run cost about 60 petaflop/s-days against GPT-3’s 3,640 for pretraining, under 2%.

figure 3 · the reward model

A model that learns to score answers

Step two is the conceptual heart. Writing perfect answers is slow and hard; ranking answers is fast and reliable. So labelers are shown between K = 4 and K = 9 of the model’s own responses to one prompt and simply order them best to worst. A separate 6B model — the reward model — is then trained so that its scalar score agrees with every pairwise preference implied by the ranking: loss = −log σ(r(x, yw) − r(x, yl)).

That scalar is the crucial artifact: it converts “which of these did a human like more?” into a signal a training loop can maximize — soft, rich feedback of exactly the kind entry 06 argued teaches fastest.

figure 4 · worked by hand

Why rank nine answers instead of two

Here is the labeling economics, worked out. A ranking of K answers implies K(K−1)/2 pairwise comparisons. Rank 9 responses in one sitting and you have banked 36 comparisons for roughly the effort of reading nine answers once — versus asking 36 separate “A or B?” questions. All 36 go into one training batch, which also stops the reward model from overfitting to any single prompt’s quirks.

figure 5 · the RL loop

Chasing the reward — on a leash

Step three: the model generates an answer, the reward model scores it, and PPO nudges the weights toward higher scores. Left alone, this loop finds the reward model’s blind spots — so a per-token KL penalty tethers the policy to the SFT model: objective = r(x,y) − β·KL(policy ‖ SFT). Drift far from how the SFT model speaks and the penalty eats the reward. One more refinement — mixing pretraining gradients back in (PPO-ptx) — matters enough that we give it the next figure.

Hold onto the KL leash — the term β·KL(π ‖ πref) is the exact quantity entry 16 starts from when it makes this whole loop collapse into a single supervised loss.

results · human preference

1.3B, preferred over a model 100× its size

The result that made the paper famous: labelers preferred the 1.3B InstructGPT’s answers to those of the raw 175B GPT-3. At matched size the margin is a rout — the 175B InstructGPT wins 85 ± 3% of head-to-heads against 175B GPT-3, and 71 ± 4% even when GPT-3 is helped with a carefully engineered few-shot prompt.

The reading we keep: post-training is worth orders of magnitude of scale on the axis users actually feel. The cheapest capability upgrade in the building is usually alignment, not parameters — the post-training echo of entry 10.

figure 7 · the alignment tax

What alignment quietly costs — and the fix

Aligning on human preference regressed public NLP benchmarks — SQuAD, DROP, HellaSwag, translation. The paper names this the alignment tax: optimize hard for one behavior and you quietly pay elsewhere. Their fix, PPO-ptx, mixes pretraining gradients into the RL updates — anchoring the model to its original distribution while the reward pulls on style — and recovers the regressions almost entirely.

the deeper caveat aligned to whom? The preferences belong to roughly 40 contractors, screened for agreement (72.6 ± 1.5% inter-annotator), mostly US and Southeast Asia based. The paper is unusually honest that this is alignment to a standard, not the standard — and InstructGPT still follows harmful instructions and still confidently invents facts. The template ships with its blind spots visible.

the equations · worked

Learn a score, then chase it on a leash

applied · our reading

The template, three stages later

Every post-training pipeline since — including the ones later in this arc — is a remix of these three steps. What we carry into our own work:

• Collect comparisons, not just corrections. The pipeline’s scalable ingredient is ranked outputs — cheap for a domain expert to produce (read nine answers, order them) and each session banks dozens of training pairs. Expert review queues should emit rankings as a by-product.
• Never optimize a learned reward without a leash. The KL penalty is not a hyperparameter nicety; it is what stops the model from strip-mining the reward model’s blind spots. Any “optimize what the judge likes” loop — RLHF or LLM-as-judge distillation — needs the same tether.
• Always run the evals you did not train on. The alignment tax was only visible because they measured SQuAD and DROP after preference tuning. A domain fine-tune that only reports domain metrics is hiding the invoice.

Next in the arc: the pipeline’s expensive middle — the reward model and the RL loop — gets mathematically deleted. Entry 16: DPO.