The Illustrated Decoder

figure 1 · the map

The decoder, in one look

Everything that follows is this one picture. Tokens enter at the bottom, become vectors, pass through the repeated block — attention, then a feed-forward network, each adding its update onto the residual stream — and end as a probability distribution over the vocabulary. One token is sampled and loops back into the context.

click any figure to zoom · scroll to magnify · drag to pan

figure 2 · autoregression

One token per pass

The model has no plan and writes no drafts. Each pass through the stack predicts exactly one next token, conditioned on everything before it — including its own previous output. Appending that token and running again is the whole writing loop.

figure 3 · embeddings

Tokens become vectors

Each token id looks up a learned column of numbers, and a positional pattern is added so word order survives — below, the real arithmetic: E[8402] plus the sin/cos ladder PE(2), summed element by element into x. From here on the model never sees words again — only geometry, where nearby meanings sit at nearby points.

figure 4 · the atom

Matrix multiplication, by hand

Before anything else: the one operation underneath it all. Watch each output cell get computed — a row from A dotted with a column from B, products summed, cell filled. Every projection, every attention score, and every layer below is exactly this, just billions of times larger.

figure 5 · q · k · v

Three lenses on every token

Three learned weight matrices project each vector into a query (what am I looking for?), a key (what do I contain?), and a value (what do I pass on?). These matrices are the weights — what training learns and fine-tuning reshapes. On the right: one output cell being computed live, term by term.

figure 6 · masked attention

Score, mask, softmax, blend

Attention is four small steps: score every pair of tokens, mask the future so a decoder can only look back, softmax the scores into weights that sum to one, and blend the values with those weights. That's the entire mechanism this site is named after.

figure 7 · multi-head

Eight readers, one sentence

That mechanism runs eight times in parallel, each head in its own low-dimensional subspace — one tracks syntax, another coreference, another position. Their readings are concatenated and mixed back together.

figure 8 · residual + ffn

The stream and the knowledge layer

Nothing overwrites anything. Each sublayer reads the residual stream, computes a small update, and adds it back — watch it happen with real numbers below: the stream [1.0, -0.5, 2.0] gets nudged, element by element, never replaced. Then layernorm recenters and rescales so values stay tame across dozens of layers. The FFN itself is just two matrix multiplications around a gate: expand to 4d, gelu, project back — with d = 768 that's ≈4.7M weights per layer, which is where most of the "knowledge" lives.

figure 9 · sampling

From vector to word

After N layers, one projection scores every word in the vocabulary, softmax turns scores into probabilities, and one token is sampled — temperature decides how adventurous the dice are. The winner drops back into the context, and the loop runs again.