knowledge base · entry 12
Reasoning that trained itself
A read-through of the paper that grew a reasoning model from a plain one using only reinforcement learning — no worked solutions, just a signal for whether the answer was right.
every figure zooms — click it, scroll to magnify, drag to pan
part one
The idea, before any jargon
figure 1 · the whole idea
Reasoning that trained itself
The usual way to make a model reason is to show it: collect thousands of worked, step-by-step solutions and fine-tune on them. DeepSeek's first model, R1-Zero, was shown none. It started from a plain base model and was only ever told, after each attempt, whether its final answer was right.
From that single bit of feedback, it taught itself to reason — writing long chains of thought, checking its own steps, and backtracking. Reasoning wasn't copied from examples; it emerged because thinking carefully earned the reward.
click any figure to zoom · scroll to magnify · drag to pan
This is why the paper opens the frontiers arc. Everything before it makes the standard recipe cheaper or better; this one changes what the training signal is — from imitating humans to being rewarded for getting a checkable answer right.
figure 2 · the reward
The reward is just: is the answer right?
The trick that makes pure RL work here is the reward. It isn't a learned "judge" model scoring how good an answer seems — those can be gamed. It's a plain, rule-based check: extract the final answer, compare it to the known solution, and give a reward only if it matches. Plus a small reward for using the required format.
Because the reward is a verifiable fact — a number that matches, code that passes its tests — the model can't earn it by merely sounding convincing. That's the strength, and also the boundary: it works cleanly only where an answer can be checked. Math, code, and logic fit; open-ended judgment does not.
part two · the mechanism
How the signal becomes learning
figure 3 · GRPO
Compare several tries, keep what beats the batch
Turning "right / wrong" into a weight update is where the algorithm, GRPO, comes in. For each question it samples a whole group of answers, scores them, and computes the group's average. Answers above average get pushed toward; answers below get pushed away.
The elegant part is what's missing: no separate "critic" network to estimate how good a state is, as older RL methods need. Just sample a handful of answers and let them compete against their own batch. It's cheaper to run, and it's all the signal the policy needs.
part three · the surprise
Behaviour nobody wrote in
figure 4 · the aha moment
The "aha moment"
The result that made this paper famous isn't a benchmark number — it's a behaviour. Watched over the course of RL training, the model's answers spontaneously get longer: it learns, without being told, that spending more tokens thinking leads to more right answers.
And it starts doing something no one programmed: catching its own mistakes mid-solution — "wait, let me re-check that" — trying another approach, and verifying the result. These are strategies a human problem-solver uses, and they appeared here purely because they paid off. Capability emerging from incentive rather than imitation is the whole point.
part four · the result
From raw reasoning to a usable model
figure 5 · Zero to R1
A light polish on top of pure RL
R1-Zero reasoned brilliantly but read badly — mixing languages, hard to follow. So the full DeepSeek-R1 adds a light supervised "cold start" for readability, then more RL, then a final polish. The reasoning came from RL; the supervision only made it legible.
The payoff: on competition math like AIME, R1 matches OpenAI's o1 — the strongest closed reasoning model of the moment — and it shipped with open weights. Reasoning stopped being a proprietary secret and became something you could download.
the math · in the paper's notation
GRPO in two equations
Section 2.2.1 of the paper is where most readers stall. It is two equations: standardize the group's rewards, then take a clipped, leashed step. Both read aloud, with the advantage worked on a toy group.
the equations · worked
Standardize the group, then step carefully
§2.2.1 · the group-relative advantage
Ai = ri − mean( {r1, …, rG} )std( {r1, …, rG} )
read aloud: an answer's advantage is how far its reward sits above the group's average, in units of the group's spread. "Better than your batch-mates" is the entire learning signal — no learned critic estimates it.
- Gthe group size — how many answers are sampled per question
- rianswer i's reward: the rule-based accuracy check plus the format check from figure 2
- Aithe advantage — positive means "push the model toward this answer", negative "away"
§2.2.1 · the GRPO objective
𝒥(θ) = E [ 1G Σi=1G min( ρiAi, clip(ρi, 1−ε, 1+ε) Ai ) − β·DKL( πθ ‖ πref ) ]
read aloud: average the advantage-weighted probability ratios over the group — but clip the ratio so no single update moves the policy too far — and subtract a penalty for drifting from the reference model. It is PPO's clipped step with the critic network deleted and the group average in its place.
- ρithe probability ratio πθ(oi|q) / πθold(oi|q) — how much more likely the new policy makes answer i than the policy that sampled it
- clip, εthe trust region: ratios beyond 1±ε are flattened, so one lucky answer can't yank the weights
- βthe KL leash — the same tether InstructGPT introduced, keeping the policy near πref
- πrefthe frozen reference policy the drift is measured against
part five · what we take from it
A reasoner you can own
applied · our reading
Reasoning you can own — and distil down
For a sovereign, domain-specific system, two things here matter. First, R1 is open, and its reasoning distils into small dense models — the trick from entry 06 — so a small on-prem model can inherit the chains of thought. Second, the RL recipe itself is usable wherever your domain has checkable answers: a tax calculation, a rule-compliance test, a reconciliation that either balances or doesn't.
the caveat we would flag The whole method rests on a verifiable reward. It fits tasks with ground truth and fits "is this advice sound?" far less — there's no key to check open-ended judgment against. And the long chains of thought that make it strong cost real inference tokens on every query, which is exactly the lifetime-serving cost the last two entries were about. Reasoning is powerful, but you pay for it by the token.
next in the arc
Where this goes
R1 changed the training signal. The last entry, Mamba, changes the architecture — a sequence model that drops attention's quadratic cost for a compact recurrent state, and the most serious challenger yet to the Transformer's dominance.
source
DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. The AIME/benchmark numbers and the R1-Zero→R1 pipeline are the paper's; the reward flow, the GRPO sketch, and the self-correction snippet are our own illustration.
← back to the knowledge base · ⚙ the illustrated decoder · ▶ run the forward pass
attention.sh — a division of Jinacode Systems. Transformers,
fine-tuning, and sovereign LLMs.
we're hiring the curious — hello@attention.sh