knowledge base
Knowledge Base
Papers we've read carefully enough to redraw. Each entry starts with no assumptions at all and adds the technical layer only once the idea has landed — the version we wish had existed when we first opened the PDF.
20 illustrated · 22 in the base
Send each query to the cheapest model that can actually answer it — the upfront routers, the cascades that check their own work, and how to measure any of them honestly.
-
entry 01 · arXiv:2406.18665
RouteLLM — The model that decides which model answers
A small router sends each question to the cheapest model that can handle it, and PGR, CPT and APGR tell you whether it worked. Nine figures, with toy examples for the metrics and the matrix factorization.
9 figures → -
entry 02 · arXiv:2404.14618
Hybrid LLM — The router that knows when cheap is good enough
A BERT-style router predicts whether the cheap model's answer would be good enough, learned from a measured quality gap and routed by one tunable threshold. Seven figures, with a worked soft-label example.
7 figures → -
entry 03 · arXiv:2305.05176
FrugalGPT — Run the cheap model first — then check its work
The canonical LLM cascade: run models cheapest-first and escalate only the answers a small checker judges unreliable, matching GPT-4 for up to 98% less. Seven figures, routing-vs-cascading laid out, the cost funnel by hand.
7 figures → -
entry 04 · arXiv:2310.12963
AutoMix — A model that grades itself — and a judge that knows better
The small model self-verifies its own answer, and a noise-aware POMDP meta-verifier decides when to trust that shaky signal. Seven figures, with the three hidden states and the belief-update worked out.
7 figures → -
entry 05 · arXiv:2403.12031
RouterBench — One ruler for every router — and a humbling result
The shared testbed (405k precomputed outcomes) and the AIQ metric that put every router on one cost/quality plane, bookended by an oracle and a zero router. Seven figures, and the finding that today's routers barely beat the trivial baseline.
7 figures →
The same specialist quality on a fraction of the parameters — by copying a bigger model, by finding the subnetwork that mattered, or by training only a thin adapter.
-
entry 06 · arXiv:1503.02531
Distillation — Teaching a small model to think like a big one
A large teacher's soft probabilities carry more signal than the single right answer — train a small student on them and it punches far above its size. The origin of every compression story that follows. Six figures, with temperature worked by hand.
6 figures → -
entry 07 · arXiv:1803.03635
Lottery Ticket — The winning subnetwork was in there all along
A dense network hides a small subnetwork that, trained alone from the right starting weights, matches the full model. Prune, rewind, repeat — and most of the parameters turn out to be scaffolding. Six figures, with pruning worked by hand.
6 figures → -
entry 09 · arXiv:2106.09685
LoRA — Fine-tune by adding, not rewriting
Freeze the whole model and train two tiny low-rank matrices alongside it — the update a fine-tune needs is far lower-rank than the weights themselves. A specialist for a fraction of the parameters and none of the serving cost. Six figures, with the parameter count worked by hand.
6 figures →
How much data, of what quality, for how big a model — the arithmetic of where to spend a compute budget, and the case that quality beats quantity.
-
entry 08 · arXiv:2306.11644
Textbooks Are All You Need — Better data beats a bigger model
A 1.3B model trained on textbook-quality data outscores models many times its size on code. The lesson under the phi series: curation is a lever at least as strong as scale. Six figures, from what "textbook quality" means to the fine-tune that unlocked it.
6 figures → -
entry 10 · arXiv:2203.15556
Chinchilla — Most big models are undertrained
For a fixed compute budget, model size and training tokens should grow together — and the models of the day were far too large for the data they saw. The scaling law that reset how everyone budgets a training run. Six figures, with the isoFLOP method and the 20:1 rule worked out.
6 figures → -
entry 11 · arXiv:2401.00448
Beyond Chinchilla — Once you pay to serve it, train it smaller
Chinchilla optimizes training cost alone. Add the cost of every future inference and the optimum shifts: train a smaller model on more tokens, because you pay for its size on every single call. Six figures, with the train-vs-serve crossover and why modern models are "overtrained".
6 figures →
How a base model becomes an assistant — the data bar, the human-feedback template, the loss that made preferences cheap, the hardware that fits one GPU, and the fully open recipe that ties it together.
-
entry 14 · arXiv:2305.11206
LIMA — A thousand examples, carefully chosen
The Superficial Alignment Hypothesis: the knowledge is pretrained, and 1,000 curated examples teach the format. Judges scored the tiny fine-tune at least as good as GPT-4 on 43% of prompts — and doubling the data bought nothing. Seven figures, the curation funnel and the ablations worked out.
7 figures → -
entry 15 · arXiv:2203.02155
InstructGPT — The three-step template every assistant follows
SFT, then a reward model on human rankings, then PPO on a KL leash — the pipeline that defined post-training, and the 1.3B model preferred over 175B GPT-3. The alignment tax and its pretraining-mix refund. Seven figures, with the ranking arithmetic worked by hand.
7 figures → -
entry 16 · arXiv:2305.1829
DPO — Your language model is secretly a reward model
The change of variables that collapses RLHF's reward model and RL loop into one classification-style loss — the partition function cancels, the gradient self-weights by how wrong the model is, and preference tuning becomes an afternoon job. Plus the family: IPO, KTO, ORPO, SimPO. Seven figures, one pair worked by hand.
7 figures → -
entry 17 · arXiv:2305.14314
QLoRA — A 65B fine-tune on the GPU you already have
NF4 quantile bins, double quantization and paged optimizers put a frozen 4-bit base under 16-bit LoRA adapters — 780GB of training memory becomes one 48GB card with no measured quality loss. Guanaco hits 99.3% of ChatGPT in 24 GPU-hours, and 9k curated examples beat 450k. Seven figures, the memory bill worked line by line.
7 figures → -
entry 18 · arXiv:2411.15124
Tulu 3 — The recipe, published — every stage, every dataset
The fully open post-training pipeline: skill-targeted curation with decontamination tooling, SFT, length-normalized DPO on on-policy pairs, and RL with verifiable rewards — finishing ahead of Llama 3.1 Instruct, GPT-4o-mini and Claude 3.5 Haiku. The capstone that turns entries 14–17 into one runnable recipe. Seven figures.
7 figures → -
entry 19 · arXiv:2602.15763
GLM-5 · Kimi K3 — What powers the open frontier
The arc's coda, dated July 2026: GLM-5's five-stage pipeline (SFT, then reasoning, agentic and general RL, then cross-stage distillation) on an asynchronous RL factory with 10k+ verifiable environments — and Kimi K3's 2.5×-intelligence-per-FLOP bet with 104B of 2.8T parameters active. The recipe from entries 14–18, industrialized. Seven figures.
7 figures →
The bets on what comes after the standard recipe — reasoning learned from reward, sequence models without attention, models that keep learning after training ends, and prediction in latent space.
-
entry 12 · arXiv:2501.12948
DeepSeek-R1 — Reasoning that trained itself
Reasoning emerges from pure reinforcement learning against verifiable rewards — no human step-by-step traces required. The model learns to think longer because thinking longer earns the reward. Six figures, from the verifiable reward and GRPO to the emergent "aha moment".
6 figures → -
entry 13 · arXiv:2312.00752
Mamba — Sequence modeling without the quadratic bill
A selective state-space model that reads a sequence in linear time and keeps a compact state instead of re-attending to everything — the most serious challenger to attention's dominance for long context. Six figures, from the quadratic wall to content-based selection.
6 figures → -
entry 20 · arXiv:2512.24695
Nested Learning (HOPE) — The model that never stops learning
Google's reframe: a model and its training are one nested system of associative memories, each updating on its own clock — and today's LLMs have only two clocks, every-token and never. HOPE fills the spectrum with a self-modifying sequence model and a continuum memory system that keeps consolidating. Eight figures, with the momentum arithmetic worked by hand.
8 figures → -
in the works · arXiv:2606.23444
SkyJEPA — Predicting the world, not the next token
LeCun's JEPA thesis made concrete: model the dynamics in embedding space rather than rolling out tokens, for long-horizon control. A world model in practice, and the counterpoint to autoregression.
read the paper ↗
What all of the above looks like in a regulated domain, where the data can't leave the building and "approximately right" isn't good enough.
No entries match that.
Every numbered entry is live — illustrated end to end. Dashed
cards are in the works — the wider reading list under
papers/, built in curriculum order — and link to the source paper until
their entry lands. As the base grows: search and the tag chips cut across every arc,
the arc bar above jumps within the page, and ≡ compact flips to a
one-line-per-paper view.
Prefer a guided tour of the architecture itself? The Illustrated Decoder covers a GPT-style model one figure at a time, or run the forward pass in 3D.
Working on routing, fine-tuning, evaluation, or sovereign deployment — or want to? hello@attention.sh