Learn · Track

Faster RAG with Opus and Sonnet

Build a retrieval-augmented GST assistant, measure it honestly, then make it faster and cheaper without making it worse — by sending each question to the smallest Claude model that can handle it and not paying twice for the same tokens.

You will use: Claude API, Opus, Sonnet, your eval set

0 of 9 steps done
  1. Step 1 of 9

    How RAG works

    1 Watch & read

    • RAG — The original — look up passages first, then generate from them. Hindi video coming soon Read the paper ↗
    • Dense Passage Retrieval — Find passages by meaning, not shared words — the idea behind every embedding search. Hindi video coming soon Read the paper ↗

    2 Concepts explain each one in your own words before moving on

    • Chunking: the unit you retrieve. For statutes, a section with its provisos and explanations belongs together.
    • Embeddings find text by meaning; keyword search finds exact terms like 'Rule 42'. Hybrid search uses both.
    • Reranking: fetch many candidates cheaply, then let a stronger model order them and keep a few.
    • Top-k is a dial: more chunks raise the chance the right one is present, and raise cost, latency and distraction.

    3 Quick quiz

    1. A user asks about "Section 17(5)(d)". Which retrieval method is most likely to find the exact provision?

    2. Why rerank instead of just retrieving the top 5 directly?

    4 Action

    Build the plainest pipeline that works over the CGST Act and Rules: chunk by section, embed, hybrid search, top-5, one Claude call to answer with citations. Run the eval set from the fine-tuning track through it.

  2. Step 2 of 9

    Evaluate it in two halves

    1 Watch & read

    • RAGAS — Score a RAG system in parts — was the right text found, and did the answer stick to it? Hindi video coming soon Read the paper ↗
    • How Much Do Legal RAG Systems Still Hallucinate? 2026 — Eight legal RAG systems measured claim by claim — and questions with a false premise are where they break. Hindi video coming soon Read the paper ↗

    2 Concepts explain each one in your own words before moving on

    • Two separate questions: did retrieval return the governing text, and did the answer stay faithful to it?
    • Your own finding fits here: the right sections were usually retrieved, and the model applied them wrongly.
    • Claim-level checking: split an answer into claims and test each against the retrieved text.
    • False-premise questions — 'since X is exempt, how do I…' when X is not exempt — are a known weak spot of legal RAG.

    3 Quick quiz

    1. The right section was retrieved, but the answer contradicts it. Which half failed?

    2. Why include questions built on a false premise in the eval set?

    4 Action

    For every eval question record two things separately: was the governing provision in the retrieved chunks, and was each claim in the answer supported by them. Add ten false-premise questions.

  3. Step 3 of 9

    Find where the time goes

    1 Watch & read

    2 Concepts explain each one in your own words before moving on

    • A RAG answer has stages: retrieve → rerank → generate. Each has its own clock.
    • Time to first token vs. total time: what the user feels vs. what you pay for.
    • Output tokens are generated one at a time; long answers are slow answers.

    3 Quick quiz

    1. Before optimising anything in a RAG pipeline, you need…

    2. Two answers are equally correct; one is three times longer. The longer one is…

    4 Action

    Instrument the pipeline: for every eval question, log milliseconds per stage, tokens in, tokens out, and which model answered. Run every question through Opus and through Sonnet and keep both answers.

  4. Step 4 of 9

    Route — pick the model before answering

    1 Watch & read

    2 Concepts explain each one in your own words before moving on

    • Strong model / weak model: most questions do not need the strong one.
    • A router is a small, fast classifier with a threshold you can tune.
    • The labels are the hard part: for which questions did the cheaper model do just as well?

    3 Quick quiz

    1. What does a router need in order to be trained?

    2. You raise the router's threshold so more queries go to the strong model. What happens?

    4 Action

    Label your rows: was Sonnet's answer as good as Opus's? Build the simplest router that could work — a few rules, or nearest-neighbour on question embeddings — and report what share of questions it can send to Sonnet with no drop in score.

  5. Step 5 of 9

    Cascade — answer cheap, verify, escalate

    1 Watch & read

    2 Concepts explain each one in your own words before moving on

    • Cascade: try the cheap model first; escalate only if a check fails.
    • In RAG the check is concrete: is the answer supported by the retrieved text?
    • The catch: hard questions now pay for two calls, so a cascade can be slower on exactly those.

    3 Quick quiz

    1. How does a cascade differ from a router?

    2. When does a cascade make latency worse?

    4 Action

    Sonnet answers; a short second call checks the answer against the retrieved sections; on failure, Opus answers. Measure the escalation rate, the score, and p50/p95 latency against always-Opus.

  6. Step 6 of 9

    Compare — a curve, not a number

    1 Watch & read

    2 Concepts explain each one in your own words before moving on

    • Every routing setup is a point on a cost-vs-quality plot; a tunable one traces a curve.
    • The oracle — always picking the cheapest model that got it right — is the ceiling.
    • RouterBench's humbling result: simple baselines are hard to beat. Check yours.

    3 Quick quiz

    1. Why report a curve instead of a single accuracy figure?

    2. What is the oracle router?

    4 Action

    Plot cost against score for always-Sonnet, always-Opus, your router at three thresholds, your cascade, and the oracle computed from your labelled rows.

  7. Step 7 of 9

    Caching, trimming and batching

    1 Watch & read

    2 Concepts explain each one in your own words before moving on

    • Prompt caching: a long, unchanging prefix (system prompt, standing statute text) is processed once and reused.
    • Order matters: stable content first, the user's question last.
    • Fewer, better chunks: reranking to a small top-k cuts input tokens and often improves the answer.
    • Streaming does not shorten the work, but the user sees the first token sooner.
    • Batch: work nobody is waiting for — building the index, running evals — can go through the cheaper asynchronous route.

    3 Quick quiz

    1. For prompt caching to help, the cached part of the prompt must be…

    2. You cut retrieved context from 20 chunks to the 5 best after reranking. Expect…

    4 Action

    Restructure the prompt so the stable part comes first and turn on prompt caching. Add a rerank step and cut top-k. Move eval runs to batch. Re-run the questions and add both setups to the chart.

  8. Step 8 of 9 · recent research

    Frontier — time-aware retrieval, long context, agentic search

    1 Watch & read

    • Temporal Misgrounding in Legal RAG 2026 — Tax-law RAG keeps citing today's version of a provision when the question is about another date. Hindi video coming soon Read the paper ↗
    • The Token Tax 2026 — Retrieve a few passages, or load every document into the prompt? An accuracy-versus-cost comparison. Hindi video coming soon Read the paper ↗
    • Judge-R1 2026 — A legal system that pairs agentic retrieval of statutes with rubric-guided RL to cut invented citations. Hindi video coming soon Read the paper ↗

    2 Concepts explain each one in your own words before moving on

    • Temporal misgrounding: the system retrieves and cites the version of a provision in force today when the question is about another date. In GST, rates and rules move by notification, so the date is part of the question.
    • The fix is in the index: every chunk carries the period it was in force, and retrieval filters on the question's date.
    • Long context instead of retrieval: for a bounded corpus, load the whole Act and cache it. Nothing is missed by search, but every question pays for more input and the model has more to be distracted by.
    • Agentic retrieval: the model searches in several rounds and decides what to fetch next. Stronger on questions needing connected provisions, and slower and costlier per question.
    • None of these wins by default. Each is one more row on your chart.

    3 Quick quiz

    1. A user asks which rate applied to a supply made in March 2023. The index holds only the current rate schedule. The system will most likely…

    2. When is loading the whole corpus into a cached prompt a reasonable alternative to retrieval?

    3. Agentic retrieval helps most on which kind of question?

    4 Action

    Three experiments on the same eval set. One: add in-force dates to every chunk, write ten date-dependent questions, and filter retrieval by date. Two: put the whole CGST Act in a cached prompt with no retrieval. Three: give the model a search tool and let it search up to three times. Add all three to the chart.

  9. Step 9 of 9

    Put the two tracks together

    1 Watch & read

    • RAFT — Train the model with the right document mixed in among distractors, so it learns to pick. Hindi video coming soon Read the paper ↗
    • LLMs for Austrian VAT Law — Fine-tuning and RAG compared head to head on real VAT cases from a tax firm — the closest study to this project. Hindi video coming soon Read the paper ↗

    2 Concepts explain each one in your own words before moving on

    • Retrieval supplies the text; the tuned model supplies the reading of it. They fix different failures.
    • RAFT-style training — with distractors in context — is what prepares a fine-tuned model to sit behind a retriever.
    • A sensible shape: the tuned Qwen reasons over retrieved text, and a Claude model is the fallback when a check fails.
    • Combining is not automatically better. Published comparisons on VAT law show mixed results — measure it.

    3 Quick quiz

    1. Your tuned model was trained on bare questions with no context. Behind a retriever it performs worse than expected. Why?

    2. Fine-tuning plus RAG scores lower than fine-tuning alone on your eval. The right response is…

    4 Action

    Put the fine-tuned Qwen in the pipeline as the reasoner, with the cascade check escalating to Claude. Run the shared eval set through four setups — Sonnet, Opus, tuned Qwen, tuned Qwen with fallback — and report score, cost and latency for each.

← all tracks · home · progress is saved in this browser only