Run the cheap model first — then check its work

figure 1 · the whole idea

A ladder of models, climbed only as far as needed

Line your models up cheapest to dearest. Hand every question to the cheap one first. Then — and this is the whole paper — put a small checker on its answer: if the answer looks reliable, return it and stop; if it looks shaky, throw it away and try the next model up. Keep climbing until an answer passes.

Because most questions are handled fine by the cheapest model, most never leave the bottom rung. The expensive model is reached only by the few questions that genuinely stump everything below it — so you pay for it rarely, and pay the cheap toll always.

click any figure to zoom · scroll to magnify · drag to pan

The cascade above is not invented for the illustration — it is the one the paper actually learned on its financial-news benchmark: GPT-J → J1-Large → GPT-4, with the checker's bars set to 0.96 and 0.37. We will come back to those two numbers.

figure 2 · the distinction that matters

Guess before, or check after

This is the fork in the whole field, and it is worth pinning down. An upfront router — entries 01 and 02 — looks at the bare question and predicts which model to use, before any answer exists. Exactly one model runs. It is the cheapest possible design, and its weakness is that a wrong guess is never noticed: a hard question misrouted to the cheap model comes back with a confident, wrong answer and ships.

A cascade makes the opposite trade. It runs the cheap model, looks at the actual answer, and only then decides whether to escalate. That costs a second call on the questions it escalates — but nothing wrong ever ships unexamined.

Neither is strictly better. Routing is a prediction and cascading is a measurement. The more expensive it is to be wrong, the more the measurement is worth paying for — which is the whole reason this paper exists.

figure 3 · the menu

Three levers, one star

Prompt adaptation shrinks what you send — fewer few-shot examples, or several questions batched into one call. LLM approximation avoids the call altogether — cache answers you have seen before, or fine-tune a small model to mimic a big one. Both help; neither is the headline.

The LLM cascade is. Everything below is about how it decides when an answer is good enough to keep.

figure 4 · the checker

A small model that reads the answer

The piece the upfront routers never had is a reliability scorer: a function that takes the question and the generated answer and returns a number between 0 and 1 — how likely this answer is to be correct. In the paper it is a small DistilBERT tuned for regression, trained on nothing more exotic than past answers labelled right or wrong.

That is the crucial difference from routing. The scorer gets to look at what the cheap model actually said. A confident, on-topic answer scores high and is kept; a hedged, evasive one scores low and is escalated — and the mistake is caught before it reaches anyone.

the verifier is the contribution as with the label in entries 01–02, the whole system rises or falls on this one small model. A cascade with a good cheap checker is powerful; a cascade with a bad one escalates everything (or ships junk). Invest here.

figure 5 · the bar

How sure is sure enough?

Each rung of the ladder gets its own threshold τ. Score above it, keep the answer; score below it, pay for the next model. Slide the threshold up and you demand more certainty — fewer answers kept, more escalated, higher quality and higher cost. Slide it down and the cheap model carries more of the load.

The thresholds are not set by hand. FrugalGPT chooses which models to include, in what order, and every threshold together — as one optimisation that maximises accuracy subject to a cost budget. On the financial-news benchmark that search returned the cascade we started with: keep a GPT-J answer only if it scores above 0.96, keep a J1-Large answer above 0.37, otherwise fall through to GPT-4.

figure 6 · the economics

Follow 1,000 queries through the ladder

The cost of a cascade is dominated by the shape of its funnel. If the cheapest model settles the overwhelming majority of questions, almost everyone pays only its tiny toll, and the expensive model — the one that sets the sticker price — is touched by a handful of queries. Add up the two contributions and the total barely registers against “send everything to GPT-4.”

The real cost spread makes this vivid: on the paper's pricing, ten million tokens cost $0.2 through GPT-J and $30 through GPT-4 — a 150× difference. Every query you keep on the cheap rung instead of the dear one is that ratio, working in your favour.

results · three benchmarks

The numbers

FrugalGPT is evaluated over 12 commercial APIs from five providers (OpenAI, AI21, Cohere, ForeFrontAI, Textsynth) on three tasks: HEADLINES (financial news), OVERRULING (legal), and COQA (reading comprehension). Here is the learned HEADLINES cascade, end to end:

Two headline outcomes come out of that machinery.

Match the best model for a fraction of the cost. On HEADLINES, the cascade equals GPT-4's accuracy with up to a 98% cost reduction — the same answers, at roughly one-fiftieth of the bill.

Or spend the same and do better. Held to GPT-4's cost instead, the cascade improves accuracy by up to 4% — because pooling several models and keeping only their confident answers beats leaning on any single one.

figure 7 · the frontier

GPT-4 quality, a fraction of the cost

Plot every model as a point — cost against accuracy — and the individual APIs march up and to the right: cheaper is worse, dearer is better. The cascade is the curve that dominates them all. It climbs to the best model's accuracy almost immediately, then runs flat: the last sliver of budget buys essentially nothing, and if you spend it anyway you edge past GPT-4 rather than merely matching it.

applied · our reading

Cascades in a domain system

For the regulated-domain systems we build — tax, law, finance — a wrong cheap answer is not a rounding error, it is a compliance event. That asymmetry is exactly where a cascade earns its extra call. Three things we carry across:

• Reach for a cascade when a cheap check exists. If you can verify an answer cheaply — a structured-output schema, a retrieval-grounding check, a small trained scorer — a cascade never ships an unexamined cheap answer, which an upfront router cannot promise. That guarantee is worth the occasional second call.
• The checker is the whole system. A DistilBERT reliability model trained on your own graded logs is inexpensive, and everything depends on it. In a high-stakes domain we would pair it with a hard grounding check, so that being confident and being right are not the same signal.
• Tune the thresholds to a budget, then step back from the optimum. The joint cost-constrained search gives you the operating point; in a domain where a false accept is costly, deliberately demand more certainty than the curve's sweet spot.

the caveat we would flag a cascade moves the risk from “misrouted question” to “wrongly accepted answer” — but it does not remove it. A checker that is confidently wrong ships the same bad answer an upfront router would have. In regulated work, a calibrated abstain (escalate on doubt) beats a clever accept.