The router that knows when cheap is good enough

figure 1 · the whole idea

Not a smaller model — a wiser choice between two

The setup is the one from entry 01: a large model that answers beautifully and bills accordingly, a small one that is cheap and usually fine, and a tiny router at the door that reads each question and picks which model gets it. Most questions never need the expensive option; the router's whole job is to notice which ones do.

The twist in this paper is in one word: quality-aware. The router does not try to answer the unanswerable question “is this hard?” It asks the only question that touches the bill: “would the cheap model's answer be good enough here?” If yes, keep the money. If no, pay up.

click any figure to zoom · scroll to magnify · drag to pan

Where entry 01's RouteLLM learned this judgement from human votes on Chatbot Arena, Hybrid LLM learns it from a number it can compute for itself — which is the whole story of the next two figures.

the catch

The signal is blind, and it is fuzzy

Two things make this harder than it sounds. First, the same catch as every upfront router: it must decide before any answer exists, from the question alone. It never gets to see whether the cheap answer was actually fine.

Second, and specific to this paper: “good enough” is not a fixed fact. Ask the same model the same question twice and you get two slightly different answers, of two slightly different qualities. The quantity the router is chasing — how far short does the cheap model fall? — is a random variable, not a number. Everything that follows exists to squeeze a reliable decision out of a signal that is both unseen and noisy.

figure 2 · the grader

First, a cheap price on quality

You cannot pay a human to grade every answer, and asking GPT-4 to judge them all is exactly the expensive-model bill you are trying to avoid. So the authors reach for a cheap, consistent automatic metric: the BART score.

It runs a small (~400M) sequence-to-sequence model over an answer and returns a single number — loosely, how likely a fluent, faithful model was to have written this, measured against a known-good reference. It is not a perfect judge, but it correlates with human ratings well enough, it costs a fraction of a cent, and it never changes its mind. That last property is what lets you label an entire training set.

the label is the contribution, again as in entry 01, a router is only as good as the thing that grades its training data. BART score is the pragmatic pick: a good-enough quality signal that is free at scale. Swap in a better grader and the whole method improves underneath you.

figure 3 · the signal

The quality gap — the whole training signal

Now the trick is almost embarrassingly simple. Score the small model's answer, score the large model's answer, and subtract:

H(x) = quality(small) − quality(large)

That single number is the entire signal. When it is very negative, the large model was much better — a genuinely hard question, route it up. When it is near zero, the two answers were a wash — an easy question, and there is nothing to buy by paying more.

The definition is oriented so that H ≥ 0 means “small was at least as good” — the easy case. And remember the catch: H is that random variable. Sample the answers again and the gap shifts a little. Deciding from a single draw would be reckless — which is exactly the problem the next section solves.

figure 4 · the designs

Deterministic, probabilistic, and the clever relaxation

All three train the same small DeBERTa router to read a question and output a score. They differ only in the label they train it against.

1. 01

   deterministic

   Sample one answer from each model and take a hard label: 1 if the small answer scored at least as high, else 0. Simple — and brittle. One noisy draw decides the label, and a different draw could flip it.

2. 02

   probabilistic

   Sample ten answers from each and let the label be the fraction of draws the small model won — an estimate of P[H ≥ 0]. Now the router learns “this one is 30% winnable by the cheap model,” a calibrated doubt instead of a false certainty.

3. 03

   probabilistic, with a transformation

   The fix that makes it work. When the large model is genuinely stronger, nearly every query scores P[H ≥ 0] ≈ 0 — the labels are 90% zeros and a router learns nothing from a flat field. So relax the bar: ask not “is small as good?” but “is small within t of the large model?” — P[H ≥ −t]. The signal returns, and because the genuinely easy questions still score highest, they still rank first.

figure 5 · the toy example

The soft label, with actual numbers

Take one query. Sample the small and large models ten times each, and for every pair record the gap, quality(small) − quality(large). Count how often the small model wins — the gap lands at or above zero. In our toy draws that happens three times, so the probabilistic label is 3 / 10 = 0.30.

Now relax the bar by t and count again — how often did the small model come within t, gap at or above −t? Now it is six of ten, and the transformed label is 0.60. Same draws, a gentler question, a stronger signal. That 0.60 is the target the DeBERTa router learns to predict from the question text alone.

The shift quietly rewrites the question from “is the cheap model better?” to “is the cheap model close enough?” — which is the actual product question. You never needed the small model to win; you needed it to not lose by an amount anyone would notice. The threshold t that draws that line is picked once, by grid search, to make the labels as informative as possible.

figure 6 · the dial

Tune the operating point without retraining

At serving time the router reads a query and emits a score — its estimate of “the small model is good enough here.” Route to the small model when that score clears a threshold τ, otherwise to the large one. That threshold is the dial: slide it low and almost everything goes cheap and risky; slide it high and almost everything goes to the large model, safe and dear.

The point is that τ is set after the router is trained, on a held-out set, at the operating point your product tolerates — the paper picks the spot where average quality drops no more than 1%. Want to spend more for a touch more quality tomorrow? Move the dial. No retraining.

two papers, one conclusion “one router, one tunable threshold” is exactly the framing entry 01 arrived at independently. When two different teams converge on the same knob, it is usually load-bearing — it is what separates a router you can operate from a brittle if/else you cannot.

results · MixInstruct

The numbers

Router: DeBERTa-v3-large (300M), one forward pass per query. Grader: BART score. Benchmark: MixInstruct — a broad instruction mix, split 10k train / 5k validation / 5k test. The headline, held constant across pairs: route enough traffic to the small model to make 40% fewer calls to the large one, and read off the quality you gave up.

Three things worth taking from that table.

The transformation earns its keep when the large model dominates. The harder the pairing — a weak small model against a strong large one — the more the labels collapse to all-zeros, and the more the relaxation matters. It is also why the residual drop grows: a genuinely feeble small model (FLAN-t5-800m) can only carry so much traffic before quality gives, transformation or not.

The router is almost free. It is a single 300M encoder pass, off the autoregressive hot path — negligible next to generating even one token from either model. The cost of routing is not what you are trading against.

The headroom is a property of your traffic. Where many queries are genuinely easy, 40% of the expensive calls were waste and the router reclaims them. Where every query is hard, there is simply less to save — the same lesson entry 01 drew from MT Bench versus MMLU.

figure 7 · the trade-off curve

40% fewer calls, at the same quality

Sweep the threshold and plot quality retained against the share of calls sent to the large model. The router's curve climbs steeply and then saturates: it reaches essentially full quality at around 60% of the calls, which means the final 40% of large-model calls were buying nothing. That flat stretch to 100% is the saving.

Against the honest baseline — routing at random, the diagonal — the curve dominates everywhere: more quality at every budget, or the same quality for less. On the harder GPT-3.5 pairing the margin narrows but holds: 22% fewer calls for a 1% dip.

applied · our reading

Routing in a domain system

We build specialised LLM systems for regulated domains — tax, law, finance — where a strong and a weak tier already coexist and the traffic is a long tail of easy lookups under a thin head of genuine interpretation. Three things from this paper travel straight across:

• The grader is buildable from your own outputs. You do not need anyone to annotate difficulty — you need a cheap, consistent quality score and a reference set. BART score is one option; a short domain rubric graded by a cheap model is another. The router is downstream of that choice, so spend the care there.
• Predict a probability, not a verdict. The soft label is honest about ambiguity, and it is what makes the threshold mean anything. A hard yes/no throws away the very uncertainty you most need to route on.
• The transformation is the transferable trick. In a regulated domain the strong model usually should win, so naive labels are mostly “route up” and a plain router learns nothing. Relaxing the bar to “is the cheap answer within tolerance?” is exactly the right reframing for that skew.

the caveat we would flag the same asymmetry as entry 01. In tax or law a wrong cheap answer is not a lost 40% — it is a compliance incident. So set τ well to the conservative side of the curve's optimum, and choose your quality tolerance (the paper's 1%) against the cost of being wrong on the worst question, not the average one.