The model that decides which model answers

figure 1 · the whole idea

Not a better model — a better choice of model

Suppose you have two assistants. One is brilliant and charges a fortune per question. The other is decent and costs almost nothing. Most businesses pick one and live with the consequences: either you overpay for every trivial question, or you save money and get let down on the hard ones.

There is an obvious third option, and it is the entire paper: put someone at the door whose only job is to look at each question and decide which assistant gets it. That doorkeeper is called a router, and it is itself a very small, very cheap model.

click any figure to zoom · scroll to magnify · drag to pan

The comparison people reach for is hospital triage: the nurse at the door is not treating anyone. They are making one fast, cheap judgement about who needs the expensive specialist — and getting that judgement roughly right is worth enormous amounts of money.

figure 2 · why anyone bothers

Quality is bought at a steep discount, then a terrible one

Here is the fact that makes routing worth doing at all. If you slide from “everything to the cheap model” toward “everything to the expensive model”, your cost rises in a straight line — every question you upgrade costs the same extra amount. But quality does not rise in a straight line. It leaps early and then flattens out.

That flattening is the opportunity. Somewhere on that dial there is a setting where you have captured nearly all of the quality and are paying nowhere near all of the cost. The paper's job is to find it — and, more importantly, to make sure the questions you do upgrade are the right ones.

the catch

The router has to guess blind

Here is what makes this hard rather than obvious. To know for certain that the cheap model can handle a question, you would have to let it try and then check the answer — at which point you have already spent the money, and checking the answer is often harder than producing it.

So the router never sees an answer. It sees only the question, and must predict something about a future it will never observe: would paying more have actually bought a better answer here? Everything that follows — the training data, the four model designs, the three evaluation metrics — exists to make that one blind guess reliable.

figure 3 · section 3.1

Where the training signal comes from

Nobody has a dataset labelled “this question is hard”. So the authors borrow one that already exists in disguise.

On Chatbot Arena, people ask a question, get two anonymous answers from two different models, and vote for the better one. That vote is exactly the label a router needs — not “is this hard?” but the far more useful “did the stronger model actually win this particular question?” Sometimes it did. Sometimes both answers were fine and the vote was a coin flip. That second case is an easy question, defined in the only way that matters commercially.

Two practical wrinkles, both visible above. First, the arena has hundreds of models, so any specific pairing is rare — the authors group models into ten tiers by Elo rating and treat “any tier 0–1 model” as strong and “any tier 2 model” as weak. That turns 80,000 scattered battles into 65,000 usable strong-vs-weak comparisons.

Second, 65,000 is still thin. So they manufacture more: golden labels from exam-style questions where the right answer is known (got it right = won), and judge labels where GPT-4 is asked to grade both answers itself across 120,000 prompts, for about $700. As we will see in part four, this augmentation is not a footnote — it is most of the paper's performance.

figure 4 · section 3.2

Four ways to guess

With a training set in hand, the authors try four quite different designs. They all take a question in and put a probability out — how likely is the expensive model to win here? — but they get there by different instincts.

1. 01

   similarity-weighted ranking

   “What happened last time someone asked something like this?” Find the most similar questions in the training data and let their recorded winners vote, weighted by how similar each one is. No real training — it looks things up.

2. 02

   matrix factorization

   “What is each model good at, and what does this question need?” Learn a skill profile per model and a demand profile per question, then check the overlap. This one gets its own section below — it is the most interesting of the four, and the best performer.

3. 03

   a BERT classifier

   “Just read the question and decide.” Take a small off-the-shelf text encoder and fine-tune it as a straight yes/no classifier. The dullest option, and a perfectly respectable one.

4. 04

   a Llama-3-8B classifier

   “Ask a small LLM how hard this looks.” Fine-tune an 8-billion-parameter model to rate difficulty on a 1–5 scale and threshold it. The most capable reader of the four, and by some distance the most expensive to run.

figure 5 · the toy example

Skill × demand

Think of how film recommendations work. The system does not store “Tapan likes Heat”. It learns a handful of hidden dimensions — gritty, funny, long, loud — scores every film on them and every viewer on them, and predicts a match by multiplying the two together.

Matrix factorization does exactly this, with models in place of viewers and questions in place of films. Training invents some number of hidden dimensions; below we pretend there are three and that they happen to mean coding, creative writing and factual recall. In reality nobody names them and nobody chooses them — they emerge from which model won which questions.

Watch the figure loop and notice what changed between the two questions: nothing about the models. Same table, same numbers. The coding question lands on the one dimension where GPT-4's lead is enormous, so the gap is wide and the router pays up. The trivia question lands on the dimension where the two models are nearly level, so the gap collapses and the router keeps the money.

This is why a single “GPT-4 is better than Mixtral” number could never work. It would give the same answer for both questions. The multiplication is what makes the comparison question-specific — a model only gets credit on the dimensions this particular question actually asks about.

It is also the reason for the transfer result at the end of this page. The machinery that reads a question and works out what it demands is not about any particular model. Swap GPT-4 for Claude and you need a new skill row; the hard-won part, understanding questions, still applies.

figure 6 · the first metric

PGR — how much of the gap did you recover?

Start with the problem PGR solves. Suppose your router scores 75 out of 100 on some benchmark. Is that good? You cannot possibly say, because you do not know what the two models it is choosing between would have scored.

So PGR throws away the absolute number and measures position instead. Put the cheap model's score at one end of a ruler and the expensive model's at the other. PGR is simply how far along that ruler the router landed, as a percentage.

In our toy numbers the cheap model scores 60, GPT-4 scores 90, so there are 30 points on the table. A router scoring 75 has taken 15 of those 30 — PGR = 50%. A router scoring 90 has taken all of them — PGR = 100%.

the trap PGR says nothing whatsoever about cost. You can score PGR = 100% by sending every single question to GPT-4 — perfect quality, zero savings, completely pointless. PGR only becomes meaningful once you pair it with a budget, which is exactly what the next metric does.

figure 7 · the second metric

CPT — and what did that quality cost?

CPT stands for call-performance threshold, and it is the number a person actually deploying this thing cares about. It turns the question around. Instead of “how good is my router at this budget?”, it asks:

I need to recover 50% of the quality gap. What is the smallest share of questions I have to send to the expensive model to get there?

Written CPT(50%), and the answer is a percentage of calls. Lower is better — it is a price tag, not a score.

The dashed diagonal is the honest baseline: a router that flips a coin. Send 50% of questions to GPT-4 at random and you recover roughly 50% of the gap — that is what “no skill” looks like, and any real router has to beat it.

The trained router's curve bulges above that diagonal, and CPT reads off the horizontal distance between them. On MT Bench the router hits 50% of the gap at 13.4% of calls, where random needs 49.0%. That ratio — 49.0 ÷ 13.4 ≈ 3.66 — is where the paper's headline “3.66× cost reduction” comes from. It is not a modelling improvement. It is the same models, called more selectively.

figure 8 · the third metric

APGR — one number for the whole curve

CPT is precise but narrow: it describes a single budget. If you are comparing four router designs and do not yet know what budget you will run at, you want one number that summarises the entire curve.

APGR — average PGR — does the dullest possible thing. Sample the curve at ten evenly spaced budgets, read off the PGR at each, average them. That is the whole metric.

Geometrically it is the shaded area under the curve, and that makes it easy to read at a glance. A coin-flip router draws a straight diagonal, whose area is exactly half the box — APGR = 0.50, always. Anything meaningfully above 0.5 is a router that has learned something; the best in the paper reach 0.80.

how they fit together PGR grades one run. CPT prices one quality target. APGR ranks routers when you have not yet chosen a target. Same curve, read three ways — and the 0.50 baseline is the sanity check running through all three.

results · section 4.1

The numbers

Strong model: gpt-4-1106-preview. Weak model: Mixtral-8x7B. Higher APGR is better; lower CPT is better.

Three things worth taking from that table.

The gains are real but uneven. MT Bench is a mix of easy and hard prompts, so there is a lot of easy work to hand off cheaply and the router shines. MMLU is uniformly exam-hard — there is simply less cheap work available, and every router struggles to beat random by much. A router's headroom is a property of your traffic, not of the router.

Augmentation did the heavy lifting. Trained on arena battles alone, the routers are mediocre. Add the golden and judge-labelled data and they jump. The interesting contribution here is arguably the data recipe, not the model architectures.

No architecture dominates. Matrix factorization wins on MT Bench, the Llama-based classifier wins on GSM8K. Given matrix factorization trains in minutes on a single 8GB GPU and the causal LLM needs an 8B model in the serving path, the cheap option is doing remarkably well.

figure 9 · section 4.3

The result that matters most

Everything above could be an expensive curiosity if the router had to be retrained every time a model changed — which, in this industry, is roughly monthly.

So the authors ran the obvious test. Take the router trained on GPT-4 versus Mixtral, change nothing about it, and use it to route between Claude 3 Opus and Llama 3 8B — two models absent from its training data entirely.

APGR falls from 0.80 to 0.70. It degrades, but it does not collapse — and 0.70 is still far above the 0.50 you would get from coin flips. The router never learned who GPT-4 is. It learned which questions are hard, and difficulty turns out to be a property of the question rather than of whoever is answering it.

Practically: you can train a router once and keep swapping the models underneath it as better and cheaper ones arrive.

applied · our reading

Routing in a domain system

We build specialised LLM systems for regulated domains — tax, law, finance — where the traffic looks nothing like Chatbot Arena. A GST assistant gets a long tail of “what is the rate on this item?” alongside a thin, expensive head of genuine interpretation questions. That distribution is much closer to MT Bench than to MMLU, which is the shape where routing pays best.

Three things we would carry across:

• The label is the contribution. “Did paying more actually help here?” is a question you can answer from your own logs, cheaply, without anybody annotating difficulty. Any system with a strong and a weak tier is already generating this data.
• Start with matrix factorization. Minutes to train, negligible to serve, and it beat an 8B classifier on the most realistic benchmark in the paper.
• Measure with CPT, report with APGR. APGR is for choosing between designs. CPT is what you put in front of whoever signs off the budget, because it is denominated in money.

the caveat we would flag in a regulated domain the cost of a wrong answer is not symmetric. Being cheap on a question that turns out to need care is not a lost 3.66× — it is a compliance incident. The threshold you deploy at should be well to the conservative side of the one that optimises the curve.