The recipe, published — every stage, every dataset

figure 1 · the transparency gap

The recipe everyone used and nobody published

By 2024, every strong model was post-trained hard — and the labs said almost nothing about how. Open models shipped weights but not the data or the decisions; the actual craft — what to mix, in what order, with which checks — was folklore. Tulu 3’s stated purpose is to close that gap: a family of models at the open frontier, with the training data, recipes, code, infrastructure, and evaluation suite all released.

click any figure to zoom · scroll to magnify · drag to pan

figure 2 · the assembly line

Curate → SFT → DPO → RLVR

The pipeline is the arc in one diagram. Stage 0: skill-targeted data curation with aggressive decontamination (entry 14’s discipline, industrialized). Stage 1: supervised fine-tuning on the curated mixture (entry 15’s step one). Stage 2: preference tuning — but with length-normalized DPO, not PPO (entry 16’s shortcut, hardened). Stage 3: the new move — reinforcement learning with verifiable rewards on tasks a program can grade.

Each stage starts from the previous stage’s checkpoint, and each is evaluated before the next begins. Nothing advances on vibes — the eval suite is fixed up front, with an unseen split held back to catch overfitting to the development benchmarks.

figure 3 · the data engine

Prompts by the skill, not by the pile

The data work is organized backwards from target skills — knowledge, reasoning, math, coding, instruction following, safety — rather than forwards from whatever corpora were lying around. Real conversations (WildChat, OpenAssistant) are mixed with persona-driven synthetic prompts: hundreds of thousands of generated instructions, each written “as” a different persona to force diversity — the Self-Instruct idea grown up. Every source is then decontaminated against the entire evaluation suite before training.

the quiet centerpiece decontamination gets its own released tooling. If eval prompts leak into training data, every number downstream is fiction — the paper treats this as an engineering discipline, not a footnote, and so should any team that fine-tunes.

figure 4 · the preference stage

DPO at industrial scale — with one fix

The preference stage is DPO, chosen over PPO-based RLHF after ablation — cheaper, stabler, and as good. Two upgrades matter. The pairs are substantially on-policy: the SFT model’s own completions are compared against other models’ outputs, so the training signal criticizes the very model being tuned. And the loss is length-normalized — dividing the log-ratios by response length — because vanilla DPO quietly learns that longer answers win, the verbosity bias every judge-trained system inherits.

figure 5 · RLVR

When the grader is a program, skip the reward model

The final stage is the bridge to entry 12. For skills where correctness is checkable — grade-school math (GSM8K), competition math (MATH), precise instruction following (IFEval) — you do not need a learned reward model at all. Generate an answer, run the checker, reward only verified correctness, and optimize. No reward-model blind spots to strip-mine, no judge to sweet-talk: the model is paid only when it is actually right.

Weeks after this report, DeepSeek-R1 showed the same verifiable-reward idea, pushed with enough scale and patience, produces emergent chain-of-thought reasoning. Tulu 3 is the modest, reproducible version of the bet R1 made famous.

figure 6 · worked by hand

The data budget, stage by stage

The shape of the spending repays attention: roughly a million carefully mixed prompts at SFT, a few hundred thousand preference pairs at DPO, and tens of thousands of verifiable prompts at RLVR — a pyramid that narrows exactly as the signal per example gets richer. Note what it echoes: imitation is the bulk, preference is the refinement, verified reward is the final polish.

results · the scoreboard

Past the instruct models it started from

Both Tulu 3 models start from Llama 3.1 base — the same base Meta’s own instruct models use. On the averaged evaluation suite, the open recipe finishes ahead of Llama 3.1 Instruct, Qwen 2.5 Instruct and Mistral’s instruct models at matched size — and at 70B, ahead of the closed GPT-4o-mini and Claude 3.5 Haiku. The recipe, not the base, is the difference.

An 8B variant shows the same pattern at small scale, and the released artifacts — the SFT mixture, the preference mixtures, the RLVR sets, the decontamination tools, the eval harness — mean the claim is checkable, which is the entire point.

the equations · worked

Normalize the length, verify the reward

applied · our reading

The arc, folded into a checklist

• Run the pyramid, not a stage. The wins are cumulative: curation, then SFT, then preferences, then — only where a verifier exists — RLVR. A domain team can stop after DPO and keep most of the value; what it must not do is skip the curation and eval discipline the stages sit on.
• Fix the evals before touching the data. Tulu 3 froze its suite — with an unseen split — before training, then decontaminated against it. That ordering (evals → decontamination → data → training) is the transferable method, independent of scale.
• In verifiable domains, verified reward is cheap alignment. Tax computations, citation formats, statutory deadlines — a regulated domain is full of program-checkable answers. RLVR-style “reward only what the checker passes” is directly available to a team our size, no reward model required.

where the arc goes next the frontier keeps moving — reasoning-first RL (entry 12), distillation from reasoning models back into small ones (entry 06’s revenge), and the open frontier models’ own tech reports. The recipe is public now; the craft is in the running of it.