A model that grades itself — and a judge that knows better

figure 1 · the whole idea

The checker is the model itself

Entry 03 built a cascade with a separately trained scorer sitting on the cheap model's answers. AutoMix asks a sharper question: what if the cheap model can check its own work? After it answers, you simply prompt the same small model again — “look at your answer and the context; is it right?” No second model to train, no scorer to maintain.

There is an obvious objection, and the whole paper is the answer to it: a model that was unsure enough to get the question wrong is hardly a trustworthy judge of its own wrongness. So AutoMix adds a second layer — a meta-verifier — whose entire job is to decide how much to believe that shaky self-assessment.

click any figure to zoom · scroll to magnify · drag to pan

Two moving parts, then: a self-verification that is cheap but noisy, and a meta-verifier that turns that noise into a sound decision. The rest of this page is those two, one at a time.

figure 2 · self-verification

Checking is easier than answering

The self-check is posed as an entailment question, and this framing is what makes it work. Instead of “is your answer good?” — which invites a model to flatter itself — it asks: given this context, is this answer entailed by it? That is a narrower, more grounded question, and in retrieval-style tasks it has a happy property: verifying an answer against a passage is genuinely easier than producing it.

So a model too weak to answer every question reliably can still, quite often, notice when its own answer contradicts the context — and it does so few-shot, with no extra training. The output is a probability v that the answer is correct.

figure 3 · the catch

But the self-check lies

“Quite often” is not “reliably.” Plot the self-verification score against whether the answer was actually correct, and the two clouds overlap badly: confident- sounding wrong answers score high, and perfectly good answers sometimes score low.

This kills the obvious design — pick a threshold on v and escalate below it. Any threshold you choose accepts a pile of junk on one side and needlessly escalates good answers on the other. The fix is not a cleverer threshold. It is to stop treating v as the truth and start treating it as noisy evidence.

figure 4 · the hidden state

Simple, Complex — and Unsolvable

Here is the observation that makes the meta-verifier possible. Every question is secretly one of three kinds. Simple: the cheap model already got it right — keep the answer. Complex: the cheap model failed but the big one would succeed — escalate, and it is worth every penny. Unsolvable: neither model can answer it — and here escalating is pure waste, because the expensive model will fail too.

A plain cascade cannot tell Complex from Unsolvable — both look like “the cheap answer seems shaky,” so it pays for the big model on both. Recognising that a question is hopeless rather than merely hard, and pocketing the money, is a saving no upfront router and no simple threshold can capture.

the reframing the question is never “is this hard?” It is “would escalating actually help?” — which is a question about the hidden kind, not the surface difficulty. That is exactly what the meta-verifier estimates.

figure 5 · deciding under noise

From a noisy score to a belief to an action

The meta-verifier never acts on v directly. It treats the true kind of the question (Simple / Complex / Unsolvable) as a hidden state, treats the self-verification score as a noisy observation of it, and from that forms a belief — a probability over the three kinds. Then it picks the action (accept, or route up) that maximises expected reward, where reward trades answer quality against cost.

The payoff of this machinery is visible in the two low readings above. To a threshold they are identical — both well under 0.5 — yet the right actions are opposite: one is a Complex question worth escalating, the other Unsolvable and best left alone. Only the belief, informed by how often each kind produces such a score, can separate them. The policy is solved once on a validation set; at serving time it is a lookup.

figure 6 · the metric

IBC — how much accuracy each unit of cost buys

How do you score a mixer? The two endpoints are fixed: the cheap model (low cost, low accuracy) and the dear one (high, high). The honest baseline is the straight line between them — “flip a coin, sometimes call the big one” — whose slope is the trivial rate of benefit per cost. A real method has to beat that slope.

IBC — incremental benefit per cost — measures exactly this: the accuracy gained per extra unit of cost, relative to the cheap model. A mixer whose curve bows above the straight line is buying accuracy more efficiently than random escalation. AutoMix lifts IBC by up to 89% over the strongest baseline.

figure 7 · the results

Across five datasets, and a third rung

The win holds across five context-grounded QA benchmarks — NarrativeQA, QASPER, CNLI, QuALITY, COQA — routing between LLaMa2-13B and LLaMa2-70B, a roughly 50× cost gap. The gains are largest where the cheap model can most reliably self-diagnose; CNLI is the high-water mark at +89%.

The same meta-verifier extends cleanly to three rungs — LLaMa2-13B → LLaMa2-70B → GPT-4 — choosing among them by asking, at each step, whether the belief says the next model up would actually help. It is the noise-aware meta-verifier, not the self-check alone, that turns a lying signal into a dependable stop-or-climb decision.

applied · our reading

Verifiers in a domain system

Our regulated-domain systems are mostly retrieval-grounded — answers drawn from a specific tax ruling, statute or filing. That is precisely the setting where AutoMix's ideas pay, because checking an answer against the retrieved source is the natural operation. Three things we carry across:

• Harvest self-verification — it is nearly free. Even a model too weak to always answer can often flag its own errors against the retrieved context. That signal costs one extra prompt, and it is the cheapest verifier you will ever build.
• Treat the verifier's score as evidence, not verdict. The meta-verifier idea generalises past POMDPs: never threshold a raw confidence: combine it with what you know about your traffic. A calibrated belief beats a hard cutoff every time the signal is noisy — which is always.
• Learn to recognise “hopeless.” The Unsolvable state is the most under-used lever in production. Escalating a question no model can answer burns the expensive budget for nothing; a system that can abstain instead saves money and flags the query for a human — which, in a regulated domain, is the correct outcome anyway.

the caveat we would flag the meta-verifier's policy is only as good as the validation data it is calibrated on, and its reward bakes in a cost-versus-quality exchange rate. In high-stakes work, weight that reward to fear a missed Complex escalation far more than a wasted Unsolvable one — being cheap on a question that needed the strong model is the expensive mistake.