One ruler for every router — and a humbling result

figure 1 · why it exists

Four papers, four rulers

The four entries before this one each reported a win — and each in its own currency. RouteLLM speaks in APGR, Hybrid LLM in a quality gap, FrugalGPT in cost-cut percentages, AutoMix in IBC. Each ran on its own models, its own prompts, its own baselines. So a perfectly reasonable question — which of these routers is actually best? — had no answer. There was nothing to lay them side by side on.

RouterBench is that missing thing: not another router, but a shared testbed and a single honest metric. One set of models, one set of prompts, one plane, one number.

click any figure to zoom · scroll to magnify · drag to pan

This is the capstone of the routing arc on these pages. Everything from entries 01–04 — upfront routers and reactive cascades alike — can be dropped onto the plane this paper defines and compared on equal terms. That comparison is where the humbling result lives.

figure 2 · the testbed

Every model's answer, already run and priced

The practical heart of the paper is a dataset, not an algorithm. The authors ran every model on every prompt in advance and recorded, for each pair, both the quality and the cost. That is 405,467 outcomes across 11 models, 8 datasets, and 64 tasks — commonsense, knowledge, maths, code, conversation, and retrieval.

With the grid frozen, evaluating a router costs nothing. A router is just a rule for picking one cell per column; to score it you look up the cells it chose and sum the cost and quality. No inference, no API bill, no variance from re-running models. Two teams testing two routers are, at last, measuring the same thing.

figure 3 · the stage

Eleven models, placed by cost and quality

Put cost on one axis and quality on the other, and every model becomes a point. The cheap open models sit low and to the left; GPT-4 sits high and far to the right. The prize is the upper-left — top quality without the top bill — and the telling fact is that no single model owns it. That empty corner is the entire reason to route.

figure 4 · the achievable region

You can mix routers — so you compare curves, not points

Here is the methodological move that makes fair comparison possible. If router A gives you one cost-quality point and router B another, you can send a fraction of traffic to each and land anywhere on the line between them. Mixtures fill in a whole region, and the top edge of everything achievable is a non-decreasing convex hull — as you spend more, quality never drops.

So a router is not scored as a single point but as the best hull it can trace across every budget. And the hull of the raw models themselves — mixing them with no cleverness at all — is a baseline with a name: the zero router. Any learned router has to clear that hull to have earned its keep.

figure 5 · oracle and random

Between knowing nothing and knowing everything

Two reference routers frame the whole plane. At the bottom, the random router — pick a model by coin flip — traces the straight line of no skill. At the top, the oracle sends each query to the single best model for it (the cheapest, if several tie). The oracle is unreachable by construction: it would need to know each answer's quality before paying to generate it. But it marks the ceiling.

Every real router lives in the band between these two. And the size of the gap between the oracle and the best practical router is, quietly, the most important number in the paper — it is how much quality is still being left unbought.

figure 6 · AIQ

Average improvement in quality

To rank routers you want a single number, and — echoing RouteLLM's APGR — the honest one is the area under the router's cost-quality curve, normalised by the cost range. Average the quality it delivers across every budget. That is AIQ, average improvement in quality: higher means more quality per unit of spend, without having to fix a budget first.

It is the same shape of idea as the metrics in the earlier entries — reduce a whole trade-off curve to one comparable scalar — now applied uniformly to everyone, on the same grid.

results · the reality check

Today's routers barely beat the trivial one

Measured this way, the finding is bracing. The predictive routers — KNN and MLP models that learn to pick per query — scarcely clear the zero router, the baseline that does no learning at all and just mixes the raw models. The one design that pulls clearly ahead is the cascade with a low-error verifier — the machinery of entries 03 and 04. And every method trails the oracle by a wide margin.

Two things fall out of that table. The ceiling is high and mostly unclaimed — the oracle gap is enormous, so there is real headroom for better routers. And, at least on this benchmark today, reactive beats predictive: seeing a real answer before deciding (the cascade) is worth more than guessing well upfront.

applied · our reading

Measuring routers in a domain system

This is the entry we reach for before we build, not after. Its gift is a discipline for not fooling ourselves about whether a router is worth shipping. Three things we carry across:

• Build your own RouterBench first. Precompute a grid of your candidate models' cost and quality over a representative slice of your traffic, then evaluate routers offline by lookup. It is cheap, it is repeatable, and it is the only way to compare options honestly before spending on inference.
• Beat the zero router, not “always-GPT-4.” The humbling result is the lesson: a router that looks good against the strong model alone may not beat a no-learning mix of your models. Make the convex hull your baseline, or you will ship a router that earns nothing.
• Treat the oracle gap as an opportunity budget. It quantifies how much a better router could still capture. When it is large — as it usually is — the case for investing in the verifier or the labels is quantified, not hand-waved.

the caveat we would flag RouterBench's cost axis is provider list prices at one moment. In a sovereign, self-hosted deployment the cost curve is your own GPU economics, and it reshapes the whole plane — a “dear” API model may be cheaper than an open one you host badly. Rebuild the plane with your real costs before trusting any ranking, including this paper's.