Your language model is secretly a reward model

figure 1 · two routes to the same place

Delete the middleman

RLHF’s route to “prefer the better answer” is long: fit a reward model to human comparisons, then run PPO — with a value network and a frozen reference copy along for the ride — to chase that reward. Four models in memory, a sampling loop, and PPO’s famous instability. DPO’s observation: the destination of that whole journey can be written down directly as a loss on the preference pairs themselves.

click any figure to zoom · scroll to magnify · drag to pan

Nothing about the goal changes — it is the same KL-leashed reward maximization as entry 15. What changes is that the reward model and the RL loop turn out to be scaffolding you can fold into the loss and throw away.

figure 2 · the change of variables

Solve for the policy, then read it backwards

The KL-constrained reward objective has a known closed-form optimum: π*(y|x) ∝ πref(y|x)·exp(r(x,y)/β) — boost the reference model’s probabilities wherever reward is high. DPO’s move is to read that equation backwards: solve it for the reward, and every reward function becomes expressible through the policy it would produce: r(x,y) = β·log π(y|x)/πref(y|x) + β·log Z(x). The awkward term is Z(x), a sum over every possible answer — uncomputable. And here is the gift: human preferences under the Bradley–Terry model depend only on reward differences, and in a difference, Z(x) cancels.

That cancellation is the entire paper. Preferences compare two answers to the same prompt, so the one intractable object never needs to exist. What remains is written purely in probabilities the language model already computes.

figure 3 · the loss

One line, four probabilities

Substitute the reparameterized reward into the Bradley–Terry likelihood and out falls the DPO loss — a binary cross-entropy over preference pairs. For each triple (prompt, chosen answer y_w, rejected answer y_l), gather four numbers: the policy’s and the reference model’s log-probabilities of both answers. Push the chosen margin up, the rejected margin down, through a sigmoid, scaled by β.

Training is: freeze a copy of your SFT model as the reference, run both models over a batch of pairs, compute the loss, backpropagate. No sampling during training, no reward model, no value network — a preference stage that behaves like ordinary fine-tuning.

figure 4 · worked by hand

The loss on one toy pair

To feel the mechanics, run one pair through by hand with β = 0.5, in per-token-averaged log-probabilities. The model slightly prefers the rejected answer — so the implicit reward margin comes out negative, the sigmoid weight comes out large, and this pair teaches a strong lesson.

Run the same arithmetic on a pair the model already orders correctly and the weight shrinks toward zero — DPO spends its gradient where the model is wrong, which is precisely what keeps it from over-optimizing pairs it has already learned.

figure 5 · the gradient

A gradient that knows how wrong it is

Differentiate the loss and the structure is legible: increase the likelihood of the chosen answer, decrease the rejected one — with the whole update scaled by σ(r̂l − r̂w), the model’s own estimate of how badly its implicit reward mis-orders the pair. Confidently-correct pairs contribute almost nothing; confidently-wrong pairs dominate the batch.

results · summarization

Matches PPO at its best — and doesn’t fall over

On Reddit TL;DR summarization, judged by GPT-4 against human-written references, DPO reaches about a 61% win rate at temperature 0 — above PPO’s 57% at its best temperature. The sharper finding is robustness: sweep the sampling temperature and PPO’s win rate collapses toward the base model’s, while DPO’s barely moves. On Anthropic helpful-harmless dialogue, DPO is the only computationally cheap method that improves on the dataset’s own preferred completions — roughly matching a 128-sample best-of-n that costs two orders of magnitude more compute at inference.

figure 7 · the DPO family

Pick by the data you can actually collect

DPO’s simplicity made it a foundation to iterate on, and the descendants are best told apart by one question: what feedback can you actually gather? Paired comparisons → DPO (or IPO if it overfits). Only independent thumbs-up/thumbs-down → KTO. Want to skip the separate SFT stage → ORPO. Memory too tight for a reference copy → SimPO. All of them are one-line trainer swaps in today’s tooling.

the finding that outranks the choice across this whole family, the quality of the preference pairs dominates the choice of loss. A better dataset beats a fancier objective almost every time — which is entry 14’s lesson, again, one stage later.

the equations · worked

Solve, substitute, differentiate

applied · our reading

Preference tuning for teams without an RL bench

• Default to SFT → DPO. It is stable, cheap, debuggable with ordinary fine-tuning intuition, and it removes the classic excuse — “we don’t have RLHF infrastructure” — for shipping an unaligned domain model. Add complexity only when you can name the specific gain.
• Design the review queue to emit pairs. Every expert correction of a model draft is a (rejected, chosen) pair with the prompt attached. Log them from day one and the preference dataset builds itself as a by-product of normal review work — the richest cheap signal a domain team has.
• Mind what β is holding back. The reference model is the only thing standing between you and reward hacking without a reward model — DPO can still over-shoot on narrow data. Keep the KL anchor honest: reasonable β, fresh SFT reference, and an eval you did not train on (entry 15’s tax lesson applies unchanged).

Next in the arc: the other half of affordability. Preference tuning became one loss; entry 17 makes the hardware fit on one GPU.