Learn · Track
Faster RAG with Opus and Sonnet
Build a retrieval-augmented GST assistant, measure it honestly, then make it faster and cheaper without making it worse — by sending each question to the smallest Claude model that can handle it and not paying twice for the same tokens.
You will use: Claude API, Opus, Sonnet, your eval set
-
Step 1 of 9 How RAG works
1 Watch & read
-
RAG — The original — look up passages first, then generate from them. Hindi video coming soon Read the paper ↗
-
Dense Passage Retrieval — Find passages by meaning, not shared words — the idea behind every embedding search. Hindi video coming soon Read the paper ↗
2 Concepts explain each one in your own words before moving on
- Chunking: the unit you retrieve. For statutes, a section with its provisos and explanations belongs together.
- Embeddings find text by meaning; keyword search finds exact terms like 'Rule 42'. Hybrid search uses both.
- Reranking: fetch many candidates cheaply, then let a stronger model order them and keep a few.
- Top-k is a dial: more chunks raise the chance the right one is present, and raise cost, latency and distraction.
3 Quick quiz
-
A user asks about "Section 17(5)(d)". Which retrieval method is most likely to find the exact provision?
Exact identifiers are where keyword matching is strongest. Embeddings capture meaning but can blur one section number into its neighbours.
-
Why rerank instead of just retrieving the top 5 directly?
The two stages trade speed for precision at different scales — wide and cheap first, narrow and accurate second.
4 Action
Build the plainest pipeline that works over the CGST Act and Rules: chunk by section, embed, hybrid search, top-5, one Claude call to answer with citations. Run the eval set from the fine-tuning track through it.
-
-
Step 2 of 9 Evaluate it in two halves
1 Watch & read
-
RAGAS — Score a RAG system in parts — was the right text found, and did the answer stick to it? Hindi video coming soon Read the paper ↗
-
How Much Do Legal RAG Systems Still Hallucinate? 2026 — Eight legal RAG systems measured claim by claim — and questions with a false premise are where they break. Hindi video coming soon Read the paper ↗
2 Concepts explain each one in your own words before moving on
- Two separate questions: did retrieval return the governing text, and did the answer stay faithful to it?
- Your own finding fits here: the right sections were usually retrieved, and the model applied them wrongly.
- Claim-level checking: split an answer into claims and test each against the retrieved text.
- False-premise questions — 'since X is exempt, how do I…' when X is not exempt — are a known weak spot of legal RAG.
3 Quick quiz
-
The right section was retrieved, but the answer contradicts it. Which half failed?
Retrieval did its job. Better search will not fix this — the fix is in the model, the prompt, or fine-tuning.
-
Why include questions built on a false premise in the eval set?
Recent measurement of legal RAG systems found these questions among the most hallucination-prone. Rejecting the premise is the correct answer.
4 Action
For every eval question record two things separately: was the governing provision in the retrieved chunks, and was each claim in the answer supported by them. Add ten false-premise questions.
-
-
Step 3 of 9 Find where the time goes
1 Watch & read
2 Concepts explain each one in your own words before moving on
- A RAG answer has stages: retrieve → rerank → generate. Each has its own clock.
- Time to first token vs. total time: what the user feels vs. what you pay for.
- Output tokens are generated one at a time; long answers are slow answers.
3 Quick quiz
-
Before optimising anything in a RAG pipeline, you need…
Until retrieval, reranking and generation are timed separately you are guessing at which one to fix.
-
Two answers are equally correct; one is three times longer. The longer one is…
Generation is token by token and output tokens are billed, so length costs both time and money.
4 Action
Instrument the pipeline: for every eval question, log milliseconds per stage, tokens in, tokens out, and which model answered. Run every question through Opus and through Sonnet and keep both answers.
-
Step 4 of 9 Route — pick the model before answering
1 Watch & read
-
RouteLLM — A small router decides which model answers each question. Hindi video coming soon Read the paper ↗ How to read it
-
Hybrid LLM — Predict how hard the query is; send only the hard ones to the big model. Hindi video coming soon Read the paper ↗ How to read it
2 Concepts explain each one in your own words before moving on
- Strong model / weak model: most questions do not need the strong one.
- A router is a small, fast classifier with a threshold you can tune.
- The labels are the hard part: for which questions did the cheaper model do just as well?
3 Quick quiz
-
What does a router need in order to be trained?
Routing is supervised on outcomes — RouteLLM uses preference data, Hybrid LLM uses quality gaps between the two models' answers.
-
You raise the router's threshold so more queries go to the strong model. What happens?
The threshold is a dial along the cost–quality curve. That it can be turned at serving time, without retraining, is the point of both papers.
4 Action
Label your rows: was Sonnet's answer as good as Opus's? Build the simplest router that could work — a few rules, or nearest-neighbour on question embeddings — and report what share of questions it can send to Sonnet with no drop in score.
-
-
Step 5 of 9 Cascade — answer cheap, verify, escalate
1 Watch & read
-
FrugalGPT — Run the cheap model first — then check its work. Hindi video coming soon Read the paper ↗ How to read it
-
AutoMix — Let the small model grade its own answer against the retrieved context. Hindi video coming soon Read the paper ↗ How to read it
2 Concepts explain each one in your own words before moving on
- Cascade: try the cheap model first; escalate only if a check fails.
- In RAG the check is concrete: is the answer supported by the retrieved text?
- The catch: hard questions now pay for two calls, so a cascade can be slower on exactly those.
3 Quick quiz
-
How does a cascade differ from a router?
A router predicts difficulty from the question alone. A cascade spends one cheap call to get evidence, then decides.
-
When does a cascade make latency worse?
Every escalated question pays for the cheap call, the check, and the strong call. If that is the common case, route up front instead.
4 Action
Sonnet answers; a short second call checks the answer against the retrieved sections; on failure, Opus answers. Measure the escalation rate, the score, and p50/p95 latency against always-Opus.
-
-
Step 6 of 9 Compare — a curve, not a number
1 Watch & read
-
RouterBench — One ruler for every router — and a humbling result. Hindi video coming soon Read the paper ↗ How to read it
2 Concepts explain each one in your own words before moving on
- Every routing setup is a point on a cost-vs-quality plot; a tunable one traces a curve.
- The oracle — always picking the cheapest model that got it right — is the ceiling.
- RouterBench's humbling result: simple baselines are hard to beat. Check yours.
3 Quick quiz
-
Why report a curve instead of a single accuracy figure?
The same router is cheap-and-worse or costly-and-better depending on its threshold. Only the curve lets you compare two routers fairly.
-
What is the oracle router?
It is unreachable in practice but tells you how much headroom exists between your router and perfect routing.
4 Action
Plot cost against score for always-Sonnet, always-Opus, your router at three thresholds, your cascade, and the oracle computed from your labelled rows.
-
-
Step 7 of 9 Caching, trimming and batching
1 Watch & read
2 Concepts explain each one in your own words before moving on
- Prompt caching: a long, unchanging prefix (system prompt, standing statute text) is processed once and reused.
- Order matters: stable content first, the user's question last.
- Fewer, better chunks: reranking to a small top-k cuts input tokens and often improves the answer.
- Streaming does not shorten the work, but the user sees the first token sooner.
- Batch: work nobody is waiting for — building the index, running evals — can go through the cheaper asynchronous route.
3 Quick quiz
-
For prompt caching to help, the cached part of the prompt must be…
Caching matches on the prefix. Anything that varies per request — the question, the retrieved chunks — has to come after the stable part.
-
You cut retrieved context from 20 chunks to the 5 best after reranking. Expect…
Fewer input tokens are faster and cheaper, and removing marginal chunks takes away distractors. Verify on your eval set.
4 Action
Restructure the prompt so the stable part comes first and turn on prompt caching. Add a rerank step and cut top-k. Move eval runs to batch. Re-run the questions and add both setups to the chart.
-
Step 8 of 9 · recent research Frontier — time-aware retrieval, long context, agentic search
1 Watch & read
-
Temporal Misgrounding in Legal RAG 2026 — Tax-law RAG keeps citing today's version of a provision when the question is about another date. Hindi video coming soon Read the paper ↗
-
The Token Tax 2026 — Retrieve a few passages, or load every document into the prompt? An accuracy-versus-cost comparison. Hindi video coming soon Read the paper ↗
-
Judge-R1 2026 — A legal system that pairs agentic retrieval of statutes with rubric-guided RL to cut invented citations. Hindi video coming soon Read the paper ↗
2 Concepts explain each one in your own words before moving on
- Temporal misgrounding: the system retrieves and cites the version of a provision in force today when the question is about another date. In GST, rates and rules move by notification, so the date is part of the question.
- The fix is in the index: every chunk carries the period it was in force, and retrieval filters on the question's date.
- Long context instead of retrieval: for a bounded corpus, load the whole Act and cache it. Nothing is missed by search, but every question pays for more input and the model has more to be distracted by.
- Agentic retrieval: the model searches in several rounds and decides what to fetch next. Stronger on questions needing connected provisions, and slower and costlier per question.
- None of these wins by default. Each is one more row on your chart.
3 Quick quiz
-
A user asks which rate applied to a supply made in March 2023. The index holds only the current rate schedule. The system will most likely…
That is temporal misgrounding — a well-cited answer to the wrong date. It cannot be fixed at generation time; the versioned text has to be in the index.
-
When is loading the whole corpus into a cached prompt a reasonable alternative to retrieval?
A bounded, slow-changing corpus suits caching. Large or fast-moving corpora, and tight cost budgets, still favour retrieval.
-
Agentic retrieval helps most on which kind of question?
Multi-step search can follow a cross-reference that a single retrieval pass never sees — which is your "connected provisions" failure type.
4 Action
Three experiments on the same eval set. One: add in-force dates to every chunk, write ten date-dependent questions, and filter retrieval by date. Two: put the whole CGST Act in a cached prompt with no retrieval. Three: give the model a search tool and let it search up to three times. Add all three to the chart.
-
-
Step 9 of 9 Put the two tracks together
1 Watch & read
-
RAFT — Train the model with the right document mixed in among distractors, so it learns to pick. Hindi video coming soon Read the paper ↗
-
LLMs for Austrian VAT Law — Fine-tuning and RAG compared head to head on real VAT cases from a tax firm — the closest study to this project. Hindi video coming soon Read the paper ↗
2 Concepts explain each one in your own words before moving on
- Retrieval supplies the text; the tuned model supplies the reading of it. They fix different failures.
- RAFT-style training — with distractors in context — is what prepares a fine-tuned model to sit behind a retriever.
- A sensible shape: the tuned Qwen reasons over retrieved text, and a Claude model is the fallback when a check fails.
- Combining is not automatically better. Published comparisons on VAT law show mixed results — measure it.
3 Quick quiz
-
Your tuned model was trained on bare questions with no context. Behind a retriever it performs worse than expected. Why?
The serving input looks unlike the training input. Training with retrieved-style context, distractors included, closes that gap.
-
Fine-tuning plus RAG scores lower than fine-tuning alone on your eval. The right response is…
This outcome has been reported in the literature. Per-question, per-failure-type comparison shows whether retrieved text is distracting the tuned model.
4 Action
Put the fine-tuned Qwen in the pipeline as the reasoner, with the cascade check escalating to Claude. Run the shared eval set through four setups — Sonnet, Opus, tuned Qwen, tuned Qwen with fallback — and report score, cost and latency for each.
-
← all tracks · home · progress is saved in this browser only