A 65B fine-tune on the GPU you already have

figure 1 · the memory wall

Where 780 gigabytes actually go

Fine-tuning is memory-hungry far beyond the weights themselves: 16-bit weights, then gradients of the same size, then Adam’s two optimizer states on top — before any activations. For a 65B model the full-fine-tune bill exceeds 780GB — a rack of A100s to change the tone of a chatbot. LoRA already deleted the gradient and optimizer cost for the base weights; the weights themselves, 130GB in 16-bit, remained the floor.

click any figure to zoom · scroll to magnify · drag to pan

QLoRA’s question: how far down can the frozen base go — without the fine-tune noticing?

figure 2 · the architecture

A 4-bit statue with 16-bit hands

The recipe: quantize the entire frozen base model to 4 bits, and train ordinary 16-bit LoRA adapters on top. On each forward pass, weights are dequantized block-by-block to BFloat16 just long enough to multiply, so gradients flow through the frozen 4-bit weights into the adapters — only the adapters (a fraction of a percent of parameters) ever have gradients or optimizer states.

Nothing about LoRA changes — same low-rank update, same mergeability. QLoRA is a storage claim: the fine-tune’s signal lives in the adapters, so the base can be compressed brutally as long as compute happens in 16-bit.

figure 3 · innovation one

NF4 — bins shaped like the weights

Four bits buy sixteen values, so the question is which sixteen. Uniform bins (INT4/FP4) waste levels where weights rarely fall. Pretrained weights are approximately zero-centered normal — so 4-bit NormalFloat places its sixteen levels at the quantiles of that normal distribution: equal mass per bin, dense near zero where the weights crowd, sparse in the tails. Information-theoretically, for normal data, that is the optimal spend of sixteen values.

Empirically the shaping is worth about a percentage point on MMLU over FP4 — the difference between “quantization degraded it” and “matches 16-bit.”

figure 4 · innovations two & three

Quantize the bookkeeping, page the spikes

Two supporting tricks. Each block of 64 weights carries a 32-bit scaling constant — half a bit per parameter of pure bookkeeping. Double quantization quantizes those constants themselves (8-bit, blocks of 256), cutting the overhead to 0.127 bits per parameter. And paged optimizers use NVIDIA unified memory to spill optimizer states to CPU RAM during the rare long-sequence memory spikes that would otherwise kill the run.

figure 5 · worked by hand

The 65B memory bill, line by line

Add it up for 65 billion parameters: 4-bit weights are 32.5GB; double-quantized constants add about 1GB (instead of ~4GB); LoRA adapters and their optimizer states are a rounding error; what remains is activations and cache. Under 48GB, with ~3GB saved by double quantization alone — against 780GB for the full fine-tune.

results · fidelity and Guanaco

Matches 16-bit — and the fine-tune proves it

The fidelity claim is tested across sizes and benchmarks: 4-bit NF4 QLoRA matches 16-bit full fine-tuning and 16-bit LoRA — provided adapters go on all linear layers, not just attention (the finding that became the modern default). As a demonstration, the authors fine-tune Guanaco on the OASST1 dataset: the 65B model reaches 99.3% of ChatGPT’s Vicuna-benchmark score after 24 hours on one GPU.

figure 7 · the data echo

9,000 beats 450,000 — LIMA, re-proven by accident

Buried in the evaluation is the arc’s recurring lesson, found independently: Guanaco trained on 9k curated OASST1 conversations beat models trained on the 450k-example FLAN v2 mixture at chat — a 50× smaller dataset, better suited to the target behavior. Dataset suitability dominated dataset size; the effect dwarfed everything scaling the data bought.

the caveat we would flag QLoRA democratizes the compute; it does nothing about the data. After this paper the binding constraint on a small team’s fine-tune is entirely entry 14’s problem — a thousand excellent examples — which is exactly the division of labor the arc’s capstone (entry 18) industrializes.

the equations · worked

Dequantize, multiply, and count the bits

applied · our reading

The default hardware recipe

• Start every fine-tune as QLoRA. NF4 base, adapters on all linear layers, rank 8–16, BF16 compute. Escalate to full precision only when an eval — not a feeling — shows the 4-bit run leaving quality on the table. For most domain work, it never does.
• Sovereignty runs through this paper. A regulated-domain model that must train on-premise no longer needs a rack: one workstation GPU fine-tunes a 33B–65B base. The hardware excuse for shipping data to someone else’s cloud is gone.
• Adapters compose with the whole arc. The frozen 4-bit base plus swappable LoRA adapters is an architecture: one base model, one adapter per behavior or client, hot-swapped at serve time — and the same trick runs DPO (entry 16) on the pairs your review queue emits.