05 Inside the Transformer: The Complete Forward Pass

April 2026 (22252 Words, 124 Minutes)

1. Introduction

Welcome to Week 5. Last week you learned that text becomes token IDs, and token IDs get looked up in an embedding matrix to produce dense numerical vectors. That is where we stopped. The vectors were waiting at the door of the big grey box labelled Transformer Model, and we let them wait.

Today we open that box.

By the end of this lecture you will be able to trace the full journey from an embedding vector to a sampled next token, one operation at a time: add a position signal, project into queries, keys and values, compute an attention pattern, mix values through it, push the result through a feed-forward block, normalise, stack, project to vocabulary scores, apply softmax, sample. Every arrow on the way has a name, a shape, and a reason. By the end, none of them should feel like magic.

The structural pay-off you should hold on to from the first sentence is this. A trained transformer is a fixed pile of numbers — call it \(\theta\). Its size is a polynomial in five hyperparameters: vocabulary size \(V\), embedding dimension \(d\), number of layers \(N\), number of heads \(H\), and (only sometimes) maximum context length \(n_{\max}\). What is conspicuously missing from that list is the actual sequence length \(T\) of the input you happen to be running. The same \(\theta\) handles a 5-token prompt, a 5,000-token prompt, and (for RoPE-style models) a 5,000,000-token prompt. \(T\) lives in activation shapes, never in weight shapes. That is what makes a transformer one model that generalises across input lengths instead of a different model per length, and the lecture builds toward a precise statement of why.

We will reach that punchline in chapter 13, after the architecture is on the table. Three Lecture-4 mysteries will also find at least partial answers along the way — but, unlike the previous draft of this lecture, we keep CLAUDE.md and the mystery board in one place at the end (chapter 16) instead of interrupting every chapter with callbacks. The architecture comes first; the implications come once it is built.

This is the most mathematical lecture of the course. That is on purpose. You cannot reason about why a model hallucinates (Lecture 6) or why CLAUDE.md sometimes fails (Lecture 7) without understanding the mechanism that produces every single one of its tokens. But you do not need a background in machine learning to follow. We will build the geometry (dot products, cosine similarity, embedding direction) from scratch in chapter 5. We will walk the attention computation with concrete numbers in chapter 7. The only formula that really matters —

— will be something you have computed by hand before we write it down in that compact form.

Let us begin where Lecture 4 left off.

2. LLMs Are Next-Token Predictors

Before we open the transformer, look at one full forward pass from the outside.

Take the running example from Lecture 4 §6: a tiny ten-token fantasy vocabulary (\(V = 10\)) — {<BOS>, wizard, mixes, glowing, potion, dragon, drinks, fire, troll, smashes} — embedding dimension \(d = 6\), input prompt <BOS> dragon drinks (token IDs [0, 5, 6], so \(T = 3\)). It is small enough to trace by hand, specific enough to mean something. Whenever you see a \(\mathbb{R}^{3 \times 6}\) shape badge later, that is the dragon-drinks matrix flowing through the pipeline.

End-to-end, here is what happens:

1. Tokenise. The prompt text becomes the integer sequence \([0, 5, 6]\). \(T = 3\).
2. Embed. Each integer indexes into the embedding matrix \(E \in \mathbb{R}^{V \times d} = \mathbb{R}^{10 \times 6}\). Three lookups, three rows pulled out, stacked into the input matrix \(X_0 \in \mathbb{R}^{3 \times 6}\).
3. Apply the transformer. \(X_0\) goes into the big grey box. \(N\) layers later (12 for GPT-2 small, 96 for GPT-3) it comes out the other side as a matrix \(Z \in \mathbb{R}^{3 \times 6}\) — same shape, completely different numbers.
4. Pick the last row. For next-token prediction we only care about \(\mathbf{z}_T \in \mathbb{R}^{d}\), the row corresponding to the most recent token (drinks). That row has, by virtue of the attention machinery inside the box, attended to every preceding token.
5. Project to vocabulary. Multiply \(\mathbf{z}_T\) by the output matrix \(W_U \in \mathbb{R}^{d \times V}\) to produce \(V = 10\) logits — one real number per vocabulary entry.
6. Softmax. Exponentiate and normalise the logits into a probability distribution over the 10 tokens.
7. Sample. Pick one token from the distribution. For our prompt the model is overwhelmingly confident in potion — the next-token probability lands around 62 %.

That is the whole forward pass at the black-box level. Six steps. Every chapter from here on opens one of them.

Two observations to carry forward.

The output is not a sentence; it is one token. The model emits exactly one next token per forward pass. To produce the sentence <BOS> dragon drinks potion mixes glowing fire, the model runs the forward pass six times — appending its own output to the input, each time, and re-running everything. We come back to this in chapter 15.

The transformer preserves the matrix shape \(\mathbb{R}^{T \times d}\). \(X_0\) goes in as \(3 \times 6\); \(Z\) comes out as \(3 \times 6\). The internal pipeline is a sequence of shape-preserving transformations. That property is not an accident — it is the discipline that makes the architecture stack cleanly and that, eventually, makes the same \(\theta\) handle any \(T\). We will trace it through every chapter.

What is inside the grey box? That is the next chapter.

3. Inside the Transformer

Open the box one ring at a time.

Outermost ring. The transformer is a function \(\mathbb{R}^{T \times d} \to \mathbb{R}^{T \times d}\). Same shape in, same shape out, content remixed. That is what the previous chapter promised; it remains true at every depth we drill into.

Second ring. The function is a composition. First, positional encoding is added: \(H_1 = X_0 + P\), with \(P \in \mathbb{R}^{T \times d}\). Then, \(N\) transformer layers in series:

Every \(H_\ell\) is in \(\mathbb{R}^{T \times d}\). The layers do not share weights — each \(\text{Layer}_\ell\) has its own parameters \(\theta_\ell\) — but they share the blueprint. What you understand about layer 1 you understand about layer 96.

Third ring. Each layer is itself two sub-layers: multi-head attention (which mixes information across token positions) and a feed-forward network (which transforms each position independently), wrapped in residual connections and layer norm. We will hold off on residuals and norm until chapter 12 — the architectural meat is in the two sub-layers.

That is the unzipping. From here on, every chapter takes one of these objects and asks a different question:

• Where do the inputs come from? — chapter 4, the embedding lookup that produces \(X_0\).
• How do we visualise vectors in a 12,288-dimensional space? — chapter 5, geometric foundations.
• What does one layer look like? — chapter 6, attention + FFN named.
• How does the attention sub-layer actually compute? — chapter 7, the deep dive.
• What changes when we make the model decoder-only? — chapter 8, causal masking.
• Why H heads in parallel instead of one? — chapter 9, multi-head attention.
• Why does position have to be injected at all? — chapter 10, positional encoding.
• What does the FFN do? — chapter 11.
• How do residuals and layer norm hold a 96-deep stack together? — chapter 12.
• How do we get from \(Z\) back to vocabulary logits, and what is the architectural punchline? — chapter 13.
• How do we pick a token? — chapter 14.
• And how does the loop produce a whole response? — chapter 15.

The two final chapters return to CLAUDE.md and the lab.

A neural network is, end to end, just function composition — each layer is a map between vector spaces, and the whole network is \(f = f_N \circ \cdots \circ f_1\). Stack enough layers and that composed function becomes the largest object humanity has ever computed with. But every individual piece you can hold in your head. The rest of the lecture is showing them to you, one at a time.

4. Embedding Vectors as Inputs

Lecture 4 ended with a clean statement: tokenization turns text into a sequence of integer IDs. A prompt like def validate_email becomes something like [748, 11140, 62, 12888]. Those integers are the handshake between the tokenizer and the model. But neural networks cannot do arithmetic on “token ID 11140” as if it were a single scalar — a token ID is a label, not a quantity. You cannot multiply label 11140 by a weight and expect the result to mean anything. You cannot take the dot product of two labels.

So the first thing that happens inside the Transformer box is not a computation. It is a lookup.

The model carries a giant table called the embedding matrix:

\(V\) is the vocabulary size — 50,257 for GPT-2 and GPT-3, 129,280 for DeepSeek-V3, 202,048 for Llama 4. \(d\) is the embedding dimension: 768 for GPT-2 small, 12,288 for GPT-3, 5,120 for Llama 4, 7,168 for DeepSeek-V3. The matrix has one row per vocabulary entry. Row 11140 is the vector the model keeps for the token whose ID is 11140. That row is a list of \(d\) floating-point numbers.

To turn a token ID into something the model can compute with, we simply index into \(E\):

The token at position \(t\) becomes the \(\text{id}(t)\)-th row of \(E\). Nothing clever. A table lookup. Stack \(T\) such lookups for a sequence of \(T\) tokens and you get a matrix:

This \(X\) is the first object inside the model that is made of numbers you can do calculus on. Token IDs were discrete; \(X\) is continuous. Every subsequent operation in this lecture is an operation on matrices like this one. The shape will change (we will see \(T \times T\) attention patterns, \(T \times 4d\) feed-forward expansions, \(T \times V\) vocabulary logits at the end), but the type never goes back to discrete. Differentiation is possible from this point onward, which is why the embedding matrix itself is learned during pre-training: each row is nudged by gradient descent along with the rest of the model.

4.1 What is stored in a row?

Ask a simpler question first: what kind of information could a single row store? Concretely: \(d = 768\) real numbers per token, for a vocabulary of tens of thousands of tokens. That is tens of millions of learnable parameters just in the embedding table.

After training on a trillion tokens of text and code, the rows end up encoding something genuinely useful. Tokens with similar meanings sit in similar directions in this \(d\)-dimensional space. The row for python is closer to the row for javascript than it is to the row for tuesday. The row for open is closer to read than to blue. The rows for single digits 0 through 9 form a cluster. The rows for common English words cluster away from the rows for Python keywords, which cluster away from the rows for German words, which cluster away from the rows for emoji.

But — and this is the critical qualifier — each row is the same every time, regardless of context. The row for the token bank is whatever gradient descent converged on during training. It is a single fixed vector. It does not know whether this particular occurrence of bank means river bank or savings account. The embedding is the model’s first impression of each token, independent of its neighbours. Making sense of bank in this sentence is the job of the layers that come after — and the mechanism that does it is attention.

4.2 The parameter cost across model families

The embedding matrix is also where a meaningful chunk of the model’s total parameters live. Different model families pick very different \((V, d)\) trade-offs. Side by side:

A few things jump out. GPT-3 has the highest per-token capacity (\(d = 12{,}288\)), reflecting the 2020 design philosophy of “make every dimension count”. Modern open-weight models like Llama 4 keep \(d\) deliberately smaller — 5,120 — but blow up \(V\) into the hundreds of thousands to handle multilingual and code vocabularies. DeepSeek-V3 sits between them. The product \(V \times d\) — the size of the embedding table itself — converges around the gigabyte mark for frontier models. It is one of the biggest single parameter blocks in any LLM, even though it is dwarfed in total by the attention and FFN parameters that we will count in chapter 13.

The takeaway: the embedding lookup is the entrance to the model and a sizeable chunk of its parameter budget. But it is just an entrance. After this row-pulling, every further operation is computation, not lookup.

With the entrance explained, we have a \(T \times d\) matrix sitting inside the Transformer. Nothing has happened to it yet. No token has looked at any other token. We will need to give the model a sense of direction in this high-dimensional space before any of the attention machinery makes sense — that is the geometry recap in the next chapter.

5. Vectors You Can Reason About

Before we go any further, we need to make these high-dimensional vectors feel concrete. If \(d = 768\) or \(d = 12{,}288\), you cannot draw the vectors. You cannot visualise them as arrows on a page. You may be tempted to treat them as a black box: “the model does things with numbers; as long as I trust it, the details do not matter.”

That is the wall we need to get over. Not by pretending \(d = 768\) is easy — it is not — but by noticing that the handful of geometric facts the rest of this lecture relies on are all already true in two dimensions. If you understand them on a piece of paper, you understand them in \(\mathbb{R}^{768}\). The model is doing the same three things — measuring similarity, adding and subtracting directions, grouping neighbours — just with more coordinates.

This chapter is the geometry recap. We will meet dot products, cosine similarity, the famous king − man + woman ≈ queen experiment, and two visualisation techniques (t-SNE and UMAP) that let us see a handful of real embeddings projected down to 2D or 3D. Everything we do here will come back, directly, inside the attention mechanism.

5.1 Dot product and cosine similarity

Take two vectors \(\mathbf{a}, \mathbf{b} \in \mathbb{R}^d\). Their dot product is a single number:

Write each vector as a list of numbers, multiply them componentwise, sum the result. That is the whole operation. It works the same for \(d = 2\) and for \(d = 12{,}288\).

What does the output mean? In two dimensions the dot product has a famous geometric identity:

where \(\lVert \mathbf{a} \rVert\) is the length of \(\mathbf{a}\) (Euclidean: \(\sqrt{a_1^2 + a_2^2}\) in 2D, \(\sqrt{\sum a_i^2}\) in general) and \(\theta\) is the angle between the two vectors. The identity holds in every dimension. Three consequences matter.

• If \(\mathbf{a}\) and \(\mathbf{b}\) point in the same direction, \(\cos(\theta) = 1\) and the dot product is maximal for their lengths.
• If they are perpendicular, \(\cos(\theta) = 0\) and the dot product is zero.
• If they point in opposite directions, \(\cos(\theta) = -1\) and the dot product is maximally negative.

So a dot product tells you how aligned two vectors are, scaled by how long they are. That is one number carrying both “do they agree?” and “how confident should we be?”.

Often we only care about the direction, not the length. We strip lengths out with the cosine similarity:

This is always between \(-1\) and \(+1\). It is the angle-only measure of how similar two vectors are. The embedding world loves cosine similarity. When someone says “the embedding of dog is similar to the embedding of puppy”, they usually mean the cosine similarity between those two rows of \(E\) is close to \(1\).

5.2 Stacking vectors and right-multiplication

One small but load-bearing convention before we use these vectors at scale. Throughout this lecture we treat each token’s embedding as a column vector by default: \(\mathbf{x}_t \in \mathbb{R}^d\) means a \(d \times 1\) column. This is the standard convention in linear algebra.

When we stack \(T\) tokens into the embedding matrix \(X\), we lay them out as rows of transposes:

So row \(t\) of \(X\) is \(\mathbf{x}_t^\top\) — a \(1 \times d\) row, the transpose of the token’s column vector. Every weight matrix in the model lives “to the right” of these rows, with shape \(d \times d’\) for some output dimension \(d’\). Right-multiplying \(X\) by such a \(W\) applies the same linear map to every row:

Each row \(\mathbf{x}_t^\top W\) is a \(1 \times d’\) row vector. It is the transpose of the projected column vector \(W^\top \mathbf{x}_t\) — same linear map, two notations. The per-token form we will use in every subsequent chapter, \(\mathbf{q}_t^\top = \mathbf{x}_t^\top W_Q\), is just one row of \(Q = X W_Q\). The transpose on the left of \(\mathbf{x}_t\) is doing the bookkeeping that puts the column on its side so the matrix multiplication typechecks.

This layout — vectors as columns when written alone, rows when stacked into a matrix — is what every machine-learning library actually implements (PyTorch, TensorFlow, JAX all use it, with the feature dimension as the last axis). It is also the only layout that lets the matrix-stacked form \(Q = X W_Q\) double as the per-row formula \(\mathbf{q}_t^\top = \mathbf{x}_t^\top W_Q\) without any transposes leaking into the matrix product. Whenever you see a transpose on a per-token formula in the rest of the lecture, this is what it is for.

5.3 Why neural networks care

Nothing we just said is specific to language models. The dot product shows up everywhere inside a neural network, for one simple reason: it is the cheapest useful summary of “how well do these two vectors match?”.

• A fully-connected layer computes \(\mathbf{z} = W \mathbf{x}\), and each entry of the output is a dot product between a row of \(W\) and the input \(\mathbf{x}\).
• A convolutional filter slides over the input and computes a dot product at every location.
• An attention head — which we are about to meet — compares every token against every other token using, you guessed it, a dot product.

Hold on to this. When chapter 7 says “compute the dot product between this token’s query and every token’s key”, that is not a new concept. It is the same operation, reused. What is new in attention is not the dot product itself but what the model does with the resulting numbers.

5.4 Meaning lives in direction: the King/Queen experiment

There is a classic result from the early days of distributed word embeddings (Mikolov et al., Word2Vec, 2013) that is worth pausing on, because it does more than any formula to rebuild your intuition for what a high-dimensional embedding is. If you take the pretrained embedding vectors for the words king, man, woman, queen, compute

and ask which vocabulary word has the highest cosine similarity to the result, the answer — more often than not — is queen.

Read that again. We are not doing anything semantic. We subtracted one list of \(d\) numbers from another, added a third, and looked for the closest entry in the table. And the direction we ended up in is roughly the direction of a word related to the others in exactly the way we would expect: queen is to woman what king is to man. The model did not represent “royalty” or “gender” as labels. It represented them as directions in the embedding space, and the algebra worked out.

Two calibration notes are important here before we move on.

First, this is a Word2Vec-style demonstration, not a guarantee about every modern transformer embedding. Later work (see Nissim, van Noord, van der Goot (2019)) showed that the experiment is fragile — you have to exclude the three input words themselves from the nearest-neighbour search or queen never wins, and the effect is weaker for less common analogies. The story is real but partially an artifact. Take it as a vivid illustration of a principle, not as a universal law.

Second, and more important for us, the principle is correct even when the individual example is not. Embeddings really do encode semantic axes as directions. Grammatical number, tense, gender, language (English vs. German), domain (code vs. prose) — each of these tends to correspond to some direction in the embedding space, learned by the model during pre-training. That is why a dot product between two embeddings can carry useful information: the angle between them is not random, it has been shaped by a trillion tokens of context. When attention asks “how similar are these two tokens?” it is asking a question the embedding geometry is built to answer.

5.5 Latent-space visualisation — t-SNE and UMAP

We cannot draw vectors in 768 dimensions. But we can take a collection of them and ask a dimensionality-reduction algorithm to find a projection to 2D or 3D that preserves as much of the original neighbourhood structure as possible. Two algorithms dominate this job today:

• t-SNE (t-distributed Stochastic Neighbour Embedding) focuses on preserving local neighbourhoods: tokens that were close in the original space stay close in the projection. Clusters look tight and separated.
• UMAP (Uniform Manifold Approximation and Projection) tries to preserve both local neighbourhoods and the broader shape of the data. It tends to give you something that works both as a map (global structure) and as a neighbourhood viewer (local structure).

Neither is a perfect distance-preserver — they cannot be, they are discarding information. Distances in the projection are not exactly distances in the original space. Trust the groupings, not the precise numbers.

Here is what it looks like in practice. Take the embedding matrix from a small pre-trained model, pick a few hundred tokens that you have a semantic story for, project them down to 2D with UMAP, and plot. You will see clusters emerge: Python keywords (def, class, return, if, else) sit together. Common English words sit together, away from the keywords. German words sit together, near but distinguishable from the English. Digits form a thin strand. Punctuation marks form their own tight cluster. Nothing told the projection algorithm about any of these categories — it only saw the \(d\)-dimensional vectors. The structure it found is the structure the model itself learned during training.

The accompanying notebook (chapter 17) lets you produce one of these plots yourself. Open it, run it, rotate the 3D view, and spend two minutes finding the cluster you expected and two minutes finding something surprising. Once you have done that, the phrase “high-dimensional embedding” will never feel like a wall again. It will feel like a map with extra axes you cannot quite see, but the neighbourhoods on it are real.

That is the geometry. Three facts to carry forward into attention: (1) dot products measure alignment, (2) directions carry meaning, (3) the embedding space has structure you can, at least partially, see. Everything in the attention mechanism will exploit some combination of these three.

6. Inside the Layer

Chapter 3 zoomed into the transformer and named the layer; this chapter zooms into the layer and names its two sub-layers. Each of \(\text{Layer}_1, \ldots, \text{Layer}_N\) is the same function applied with its own learned weights. Once you understand what one layer is, you understand what 96 of them are.

A single layer takes a hidden-state matrix \(H_{\ell-1} \in \mathbb{R}^{T \times d}\) and produces \(H_\ell \in \mathbb{R}^{T \times d}\). The transformation factors into two sub-layers in series:

Both maps preserve the \(T \times d\) shape. They differ in what they do with it.

6.1 Attention mixes across rows. FFN does not.

This is the only sentence you need to take out of the chapter, but it is a load-bearing one. The two sub-layers split the work along a clean axis:

• Attention is the mechanism that lets one row of \(H_{\ell-1}\) read information from the other rows. It is the only sub-layer in the whole transformer that mixes information across token positions. If the next token is it and we want to know it refers to cat, attention is the operation that pulls cat’s representation into it’s row.
• The feed-forward network (FFN) is a small two-layer perceptron applied independently to each row. Same weights, applied to row 1, then to row 2, then to row 3 — never combining information from different rows. Its job is to take whatever the attention sub-layer produced for one position and transform it: amplify some features, suppress others, look up stored knowledge in its weights.

Two sub-layers, two complementary jobs. Attention gathers; FFN processes. Without attention, the model would be 96 stacked MLPs operating on each token in parallel — no token would ever see another. Without the FFN, the model would be 96 stacked routers — information would move around but never be reshaped. Both are needed.

6.2 The attention sub-block, with shapes

We will rebuild the attention mechanism rigorously in the next chapter. For now, a sketch with shapes — enough so that nothing in chapter 7 lands as a surprise.

The attention sub-layer has three learned matrices per head, each of shape \(d \times d\) in the single-head case (smaller in multi-head — chapter 9):

It produces three intermediate matrices by projecting the input:

with \(Q, K, V \in \mathbb{R}^{T \times d}\). It computes a \(T \times T\) score matrix from \(Q\) and \(K\), softmaxes it row-wise into an attention matrix \(A \in \mathbb{R}^{T \times T}\), and uses \(A\) to mix the rows of \(V\) into an output \(\tilde H_\ell \in \mathbb{R}^{T \times d}\). We will derive every piece of this in chapter 7.

The shape worth flagging now is the \(T \times T\) score matrix. It is the only place in the entire forward pass where \(T\) appears squared in an activation. It is not a weight — there is no parameter in the model with shape \(T \times T\). The matrix is computed fresh on every input. That distinction will matter a lot when we get to chapter 13.

6.3 The FFN sub-block, with shapes

The FFN has four parameters per layer:

The factor of 4 is conventional; it appears in GPT-2, GPT-3, and most modern open-weight models. Applied per token (column vector \(\mathbf{h} \in \mathbb{R}^d\), working on its row \(\mathbf{h}^\top\) per the §5.2 layout):

where \(\varphi\) is a pointwise non-linearity (GeLU for GPT-2/3, SwiGLU for LLaMA-style models). On the whole hidden-state matrix \(H\) it broadcasts cleanly — the bias rows \(\mathbf{b}_1^\top, \mathbf{b}_2^\top\) are added to every row of the intermediate product:

Same weights applied to every row. We will look at what the FFN actually does in chapter 11.

6.4 The promise of the next four chapters

The next four chapters open the attention sub-block from four angles. Chapter 7 derives the formula on real numbers. Chapter 8 modifies it to be causal — the change that distinguishes a decoder-only LLM from a bidirectional encoder. Chapter 9 generalises one head to many. Chapter 10 deals with the elephant we have not yet acknowledged: nothing in attention as defined here actually distinguishes “the cat sat on the mat” from “the mat sat on the cat”. Position needs to be injected, and the mechanism is more interesting than it sounds.

7. Understanding Attention

The previous chapter named the two sub-layers of a transformer block. This chapter opens the attention sub-block — the architecturally novel idea, the part that distinguishes a transformer from every neural network that came before it.

We will introduce the formula in one line and then earn every symbol in it.

By the end of the chapter you will be able to explain, without reference to any analogy, what each piece does and why it has to be there. The intuition will follow, not lead.

7.1 A new kind of layer

Every neural-network layer that existed before the transformer fits a single pattern. A fully-connected layer takes an input \(X\) and applies a fixed weight matrix \(W\):

A convolutional layer is a structural variant of the same idea — same fixed kernel, applied at every spatial position. A recurrent layer applies the same recurrence weights at every timestep:

In every case, once training is finished, the weights are frozen. Every input the model ever sees gets multiplied by the same \(W\). The function \(f\) is a fixed nonlinearity. The layer’s behaviour is, in a precise sense, not a function of the input — it is a fixed function applied to the input.

Attention breaks this pattern. Attention produces an output \(Z\) of the same shape as the input \(X\), but it does so through a fundamentally different operation:

Read that slowly. The weight matrix \(W\) now depends on the input \(X\) itself. The transformation applied to \(X\) is built, on the fly, from \(X\). There is no single fixed matrix that gets multiplied by every input. The matrix changes with the input.

For a sequence like shutil.copy(src, dst), the routing is one thing. For shutil.copy(dst, src), the routing is completely different. Same parameters in the layer; different effective transformation, because the input gets to participate in computing its own routing.

This is the conceptual leap. Every prior architecture committed, at training time, to one transformation that had to work for every possible input. A fully-connected layer trained on English must apply the same \(W\) to code, to German, to punctuation, to numbers. A convolutional filter looking for edges must apply the same kernel to bright regions and dark regions alike. Attention is not locked in. It computes a tailored transformation for every input it sees.

That single change — letting the layer’s effective weights depend on the input — is the architectural innovation that turned deep learning into the architecture you can hold a conversation with.

The remaining question is mechanical: how does the layer build the input-dependent matrix \(W(X)\)? The answer is the rest of this chapter.

7.2 Three roles per token: Query, Key, Value

The transformer’s specific construction of \(W(X)\) routes through three intermediate matrices, each derived from the input by a fixed projection. Three roles per token:

• Query (Q). What am I looking for? — the question a token is asking about its context.
• Key (K). What do I contain? — the advertisement a token publishes so others can find it.
• Value (V). What should I transmit, if I am selected? — the actual payload a token offers when another token has decided to listen to it.

Three vectors per token, one per role, produced by three learned projection matrices. Following the §5.2 layout — column vectors by default, stacked as rows-of-transposes when packed into \(X\) — the per-token projections read with a transpose on the left to make the matrix multiplication typecheck:

with \(W_Q, W_K, W_V \in \mathbb{R}^{d \times d}\) for the single-head case. Each \(\mathbf{x}_t^\top\) is a \(1 \times d\) row, multiplied by a \(d \times d\) matrix to give another \(1 \times d\) row. Equivalently, transposing both sides, \(\mathbf{q}_t = W_Q^\top \mathbf{x}_t\) with all three vectors as columns in \(\mathbb{R}^d\) — same linear map, different bookkeeping. The same three matrices are used for every token: the matrices define the recipe, the embeddings provide the ingredients.

Stacking every token’s embedding row by row into the \(T \times d\) matrix \(X\) we have been carrying around since chapter 4, the projections become three matrix multiplications:

with \(Q, K, V \in \mathbb{R}^{T \times d}\). One query per token, one key per token, one value per token, computed in parallel.

Notice the careful separation: query and key together determine who listens to whom; value determines what actually gets sent. A token can advertise its relevance for one reason (its key) and carry useful information for another (its value). That decoupling is what gives attention its expressive power.

In the library analogy that comes up at every conference talk: you walk in with a question (query). Every book on the shelf has a title (key). You match your question against each title to decide which books are relevant — then you read the content (value) of the most relevant ones and summarise. Three artefacts, each doing a different job. We will not lean on this analogy further; everything that follows works directly on the math.

Compare back to §7.1. We promised a content-dependent routing matrix \(W(X)\). The fixed parameters of attention are \(W_Q\), \(W_K\), \(W_V\). The content-dependent objects are \(Q\) and \(K\), which change with every input. The actual routing matrix — the content-dependent \(W(X)\) the chapter promised — is built from \(Q\) and \(K\) in the next subsection. It will turn out to be \(\operatorname{softmax}(Q K^\top / \sqrt{d})\): a content-derived \(T \times T\) matrix that decides which tokens listen to which.

7.3 The six-step walk on “the cat sat on it”

We have queries, keys, and values. We have not yet done anything with them. The next six steps turn three matrices of raw material into one output vector per token. We will do it on real numbers small enough to verify by hand.

The example sentence: “the cat sat on it” — five tokens. We will follow what the model computes when it processes the token it.

Step 1 — Project. Three projections through \(W_Q\), \(W_K\), \(W_V\) produce \(Q, K, V \in \mathbb{R}^{5 \times d}\). Five rows, one per token. Done.

Step 2 — Score. For every pair of tokens, take the dot product between one’s query and the other’s key:

For five tokens this is twenty-five dot products. The cleanest way to write this is one matrix multiplication:

The \(i\)-th row of \(S\) holds the scores for query \(i\) — one number for each candidate key. Higher means more relevant.

The next four matrix grids show \(S\) being filled in cell by cell. The first frame is empty. The second highlights one entry — \(S_{1,1}\) — to make the indexing visible. The third fills the entire first row, showing what “query the against all keys” looks like as a single matrix-vector product. The fourth shows the complete \(S\) matrix, all twenty-five scores produced by the single product \(Q K^\top\).

Frame 1: the score matrix \(S \in \mathbb{R}^{5 \times 5}\) before any entry is computed. Rows index queries, columns index keys.

Frame 2: \(S_{1,1} = \mathbf{q}_{\text{the}} \cdot \mathbf{k}_{\text{the}} = 2.1\). One dot product, one cell. The orange outline marks "the cell currently being computed".

Frame 3: the entire first row of \(S\) — query the dotted with all five keys. This is what \(\mathbf{q}_{\text{the}} K^\top\) gives you in one matrix-vector product.

Frame 4: the complete \(S = Q K^\top\) matrix. All twenty-five dot products in one matrix multiplication. Row five is the row we will follow through the rest of the recipe.

The bottom row — query it against all five keys — is the row we care about for this walkthrough. Read off the numbers: \(S_{5,:} = [3.2, 12.8, 5.6, 2.4, 7.2]\). The strongest score by far goes to cat. That is the geometry of the projections asserting itself: \(\mathbf{q}_{\text{it}}\) and \(\mathbf{k}_{\text{cat}}\) point in similar directions, so their dot product is large. The model has not yet committed to attending to cat — that takes the next two steps — but the raw affinity is already there.

Step 3 — Scale. We divide every entry of \(S\) by \(\sqrt{d}\). The reason is in §7.4 — for now, accept it.

Step 4 — Softmax. Apply softmax row-wise:

For each row, every entry is in \((0, 1)\) and the row sums to \(1\). The result \(A \in \mathbb{R}^{5 \times 5}\) is the attention matrix. Row \(i\) is a probability distribution over tokens describing where token \(i\) should look.

This is the content-dependent \(W(X)\) we promised at the start of the chapter. It is a function of \(X\) (through \(Q\) and \(K\), which are projections of \(X\)). It is not a learned parameter — it is freshly computed for every input.

For row five (query it) at, say, \(d = 64\) so \(\sqrt{d} = 8\), Steps 3 and 4 unfold like this:

Without any rule about pronouns, without a dictionary, without a parse tree — just dot products and a softmax — the model has put 40 % of it’s attention onto cat, the largest share of any token. That is the structure of the input asserting itself through the geometry of the projections. How the projections come to encode that structure is a question for the training chapters; for now, the point is that the architecture can express it.

Step 5 and Step 6 — Weight, then sum. The attention matrix \(A\) is a routing pattern. To produce the output, multiply by the value matrix:

Row \(i\) of \(Z\) is:

Each output token is a weighted average of every token’s value vector, with weights read off the corresponding row of \(A\). For it, the output is mostly cat’s value (40 %), partly it’s own (20 %), partly the rest. The token that started as a generic pronoun embedding is leaving the layer with most of cat’s information mixed in — exactly the resolution a human reader would want.

That is the entire attention sub-layer. Six numbered steps, four matrix shapes. No new mathematics has been introduced that was not in a first-year linear algebra course.

7.4 Why we divide by \(\sqrt{d}\)

Step 3 looked arbitrary. We divided every score by \(\sqrt{d}\), the ranking did not change, and we moved on. What does that division actually buy us, and why is it worth a section?

The dot product of two vectors \(\mathbf{q}\) and \(\mathbf{k}\) is a sum of \(d\) componentwise products. If the components are roughly standard-normal (mean 0, variance 1 — a typical initialisation), each product has expected value 0 and variance 1. Sum \(d\) of them and the variance adds: the total has variance \(d\), and its typical magnitude is \(\sqrt{d}\).

So the raw scores grow as \(\sqrt{d}\). For \(d = 64\) (a per-head dimension in GPT-2), a typical score lies in roughly \([-8, +8]\). For \(d = 768\) (GPT-2’s full embedding dimension if the model used a single huge head), it would lie in roughly \([-28, +28]\). The number is not meaningful — it is an accident of how many components the dot product summed over.

That accident matters because the next step is softmax, and softmax is extremely sensitive to the scale of its inputs.

Consider softmax applied to three scores \([1, 2, 3]\): you get roughly \([0.09, 0.24, 0.67]\). Peaked but not extreme. Now multiply all three scores by 10: \([10, 20, 30]\). Softmax gives \([2 \times 10^{-9}, 4.5 \times 10^{-5}, 0.99995]\). One value dominates. Everything else is effectively zero.

This is softmax saturation. When inputs span a large range, softmax collapses onto a single winner. Two bad things follow.

First, the distribution loses information. A 0.67 / 0.24 / 0.09 distribution tells you “mostly the third one, but the second matters and the first is a minority voice”. A 0.99995 / 0.00005 / 0 distribution tells you “only the third thing exists”. Attention that cannot distribute weight is attention that cannot mix.

Second, and worse, gradients die. A saturated softmax has derivatives that are essentially zero everywhere except at the winning index. During training, gradient descent receives, at every position, a signal of effectively do nothing. The model never learns.

These are not theoretical concerns — this is exactly what happens if you train a transformer without the \(\sqrt{d}\) scaling. The original Vaswani et al. (2017) paper notes it explicitly: “we suspect that for large values of \(d_k\), the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients.” One division by \(\sqrt{d}\) is the difference between training smoothly and not training at all.

The fix is elegant because the problem is purely a scale issue. The ranking of dot products carries the information — which token is more relevant than which other. The absolute magnitude is the accident. So we normalise the magnitude back down:

Dividing by \(\sqrt{d}\) brings the scores back to constant expected magnitude, independent of dimension. The softmax receives numbers it can handle. The ranking is preserved. Training works.

7.5 The whole formula on one line

We can now read the formula end to end:

Tracing the pieces:

• \(Q K^\top\) — Step 2. Every query dotted with every key, in one matrix product. Result: scores in \(\mathbb{R}^{T \times T}\).
• \(/ \sqrt{d}\) — Step 3. Scale to keep softmax in its usable range.
• \(\operatorname{softmax}(\cdot)\) — Step 4. Row-wise. Turns each row into a probability distribution over keys.
• \(\cdot\, V\) — Steps 5 and 6 fused. Multiply the routing matrix by the values; sum implied by matrix multiplication. Result: one output vector per query in \(\mathbb{R}^{T \times d}\).

Step 1 — the projection that produces \(Q\), \(K\), \(V\) from the embeddings — is the price of admission. It happens before this formula and is the only part that involves learned parameters: the three matrices \(W_Q\), \(W_K\), \(W_V\). The rest is fixed arithmetic — matrix multiplies, a division, an exponential, another matrix multiply — composed in a way that lets the layer build a custom routing pattern for every input.

That is attention. Same input shape \(\mathbb{R}^{T \times d}\) on both ends. Internally, a content-dependent \(T \times T\) routing matrix that mixes the value vectors into context-aware outputs. The architectural pattern \(Z = f(X \cdot W(X))\) made concrete with one specific construction of \(W(X)\).

We did this on a sentence — “the cat sat on it” — that flows left to right, but nothing in the formula stops the (position 1) from attending to it (position 5). Every query sees every key. That is fine for an encoder model like BERT, which has access to the whole input. For a decoder-only LLM that has to predict the next token, every query seeing every key is a problem: it would let the model peek at the answer it is supposed to generate. The next chapter shows how a single one-line modification — a mask added inside the softmax — converts this bidirectional attention into the causal attention every GPT-style model actually uses.

8. Causal Attention

The attention we built in the previous chapter is bidirectional. For input “the cat sat on it”, the query at position 1 (the) looks at the key at position 5 (it) and pulls in its value. That is fine if your job is to take a complete sentence and produce, say, sentiment scores for it — which is what BERT does. It is wrong if your job is to predict the next token.

This chapter modifies the attention formula in exactly one place to make it causal. The change is mechanically tiny and conceptually load-bearing: it is the difference between an encoder model and a GPT-style decoder.

8.1 Why “next token” forces causality

A decoder-only LLM is trained to predict, at every position \(i\), the token at position \(i + 1\). For the prefix the cat sat, the training target is on. For the prefix the cat sat on, the target is it. For each prefix, the model emits a probability distribution over the next token, and the loss compares that distribution to the actual next token in the training data.

Now, training is efficient because it processes the entire sentence in one forward pass. Position 1’s query, position 2’s query, position 3’s query, …, all run in parallel. The model emits a distribution at every position simultaneously, and the loss sums across all positions. That is great — but it means the position-3 query (predicting on) is computed in the same forward pass as the position-4 token (on itself). If we let the position-3 query attend to the position-4 key, it would simply read the answer. Loss collapses to zero. The model learns nothing — it just learns to copy.

So we have to handicap training to match inference. At inference time, when the model is generating, position 4 does not yet exist when we compute position 3’s prediction. To stop training from using information that inference cannot, we forbid every query from attending to any key strictly to its right. Each position can use only its causal prefix — positions 1 through \(i\).

8.2 The mask, in one line

The attention formula changes by exactly one term:

where \(M \in \mathbb{R}^{T \times T}\) is the causal mask:

Adding \(-\infty\) to a score and then exponentiating gives \(0\). So the softmax sends every “look-into-the-future” entry to exactly zero, and the surviving entries on each row renormalise to sum to 1 over the causal prefix alone. Every other shape is unchanged. No new parameters. The training loop still runs all positions in parallel; only the routing pattern is now triangular.

8.3 Convention vs. structure

The mask is usually drawn as an upper-triangular matrix of \(-\infty\)s above the diagonal, with the lower triangle and diagonal at \(0\). That visual is a layout convention: rows are queries (top to bottom in time order), columns are keys (left to right in time order), so “future keys” land in the upper-right. The math would be unchanged if we drew time bottom-to-top — the mask would just look like a different triangle.

What is not convention is the asymmetry of \(i\) and \(j\). The query index is “the position making a prediction”, and the key index is “the position whose information might be used”. The constraint \(j \leq i\) is a statement about time, not about matrix geometry. The triangular mask is the geometric image of that temporal constraint.

In code, every framework spells this the same way: torch.triu(ones, diagonal=1) * -inf, or tf.linalg.band_part(-inf, 0, -1), or equivalent. The mask is a tensor that lives at the very top of the attention block, added before the softmax on every layer.

8.4 When the mask is applied: training, prefill, decoding

Three regimes use the same mask, for slightly different reasons.

Training (teacher forcing). All \(T\) tokens are fed in parallel. The mask prevents position \(i\) from cheating by looking at the ground-truth token at position \(i + 1\) which is sitting in its own input. Without the mask, the loss collapses to zero — the model learns the identity function.

Inference, prompt prefill. Your full prompt of \(T\) tokens is pushed through in one pass — this is what happens when you hit Send on a chat interface and the cursor takes a moment before the response starts. The mask is still required so every position is processed causally consistently with how the model was trained. Removing it would leak future-prompt context into earlier positions and produce out-of-distribution activations.

Inference, decoding. Generation is autoregressive. At step \(t + 1\), the model has produced tokens 1 through \(t\) and must predict the next. The new query — length 1 — attends back over positions 1 through \(t\): exactly the past. There are no “future” tokens in this particular input, so the upper triangle of the row would be empty anyway. But the architecture still applies the mask, because the mask is part of the architecture, not a training trick. You cannot turn it off at inference. We come back to the autoregressive loop in chapter 15.

8.5 What carries forward

Causal masking has the same shape, the same parameters, the same \(O(T^2)\) compute as bidirectional attention. The single change — one matrix added inside the softmax — is what makes the model decoder-only. Every GPT, every LLaMA, every Mistral, every Qwen does exactly this. The causal mask is the line between “LLM that generates text” and “model that classifies whole inputs”.

With causality understood, we are ready to scale the attention mechanism along its other axis: not just what it attends to, but how many simultaneous things it can attend to. That is multi-head attention.

9. Multi-Head Attention

One attention mechanism learns one routing pattern. We have seen what that looks like: a single \(W_Q\), \(W_K\), \(W_V\) triple produces one view of “what is relevant to what” for the entire sequence. For our five-token sentence, the routing put 40 % of it’s attention onto cat. That is one perspective on the sentence — the coreference one.

Now consider everything else that could plausibly be relevant to it. Its grammatical role (subject of a verb). Its sentence-level topic (a sentence about something tired). Its position (near the end). Its relationship to was, the verb it acts as the subject of. In a longer passage: the paragraph it belongs to, the topic of the paragraph, the parallel structures earlier in the document. A single attention head has to pick one of these perspectives to specialise in. It cannot attend strongly to cat and to the verb and to the topic word simultaneously — softmax forces the weights to sum to 1, so putting more on cat means taking it away from elsewhere.

The fix is the one that by now you can guess: do not have one attention head. Have many.

9.1 The design

A transformer runs many attention heads in parallel, side by side, at every layer. Each head has its own learned \(W_Q\), \(W_K\), \(W_V\). Each head produces its own routing pattern, completely independently. They do not share weights. They do not share attention scores. They do not know about each other.

Concretely:

• GPT-2 small: 12 heads per layer.
• GPT-3: 96 heads per layer.
• Claude-class frontier models: 100+ heads per layer, sometimes considerably more.

How does the accounting work? We split the embedding dimension across the heads. If the full model dimension is \(d = 768\) and we want 12 heads, each head gets \(d_h = 768 / 12 = 64\) dimensions to work with. Its three projection matrices \(W_Q^{(h)}, W_K^{(h)}, W_V^{(h)}\) have shape \(d \times d_h = 768 \times 64\). Head \(h\) produces \(Q^{(h)}, K^{(h)}, V^{(h)} \in \mathbb{R}^{T \times 64}\). It runs the full six-step attention computation we built in chapter 7 — independently, on its own \(Q/K/V\) matrices — and produces an output \(Z^{(h)} \in \mathbb{R}^{T \times 64}\).

Notice the parameter accounting: the Q/K/V budget is unchanged from single-head. \(H\) heads × \(3 d \cdot d_h\) per head = \(3 d \cdot H \cdot (d/H) = 3 d^2\). Multi-head is not “more expensive per projection” — it is the same parameter budget rearranged into \(H\) parallel narrower subspaces. The scaling denominator inside each head also changes from \(\sqrt d\) to \(\sqrt{d_h}\), because each head now works in a \(d_h\)-dimensional subspace.

Every head does this. Twelve heads produce twelve \(T \times 64\) output matrices. Concatenate them along the feature axis and you get a single \(T \times (12 \times 64) = T \times 768\) matrix. Then a final linear projection \(W_O\) of shape \(768 \times 768\) mixes the heads back together:

That \(W_O\) projection has no analogue in single-head attention; it is the one extra \(d^2\) of parameters that multi-head adds on top of the rearranged Q/K/V budget. Total per layer: \(4 d^2\) attention parameters (Q, K, V projections plus \(W_O\)). Without \(W_O\), each head’s output would live in its own \(64\)-dimensional slice of the embedding space, isolated from the others. \(W_O\) lets downstream layers combine information across heads — they can produce outputs that depend on what head 3 and head 7 together said.

9.2 What do different heads actually do?

This is one of the genuinely beautiful empirical findings of interpretability research. Nobody tells the individual heads what to specialise in. There is no supervision signal that says “head 4, be the coreference head”. And yet — given enough training data and enough model scale — heads differentiate naturally. Researchers probing trained transformers have found heads that, with surprising consistency, implement recognisable functions:

• Previous-token heads. They attend almost exclusively to the token immediately before them. A built-in short-range bigram memory.
• Coreference heads. They attend from pronouns and variable references to their antecedents or bindings.
• Bracket-matching heads. For code, heads that attend from a closing bracket to its matching opener, tracking nesting depth across arbitrary distances.
• Syntactic-role heads. One head specialises in finding the subject of a verb. Another finds the verb’s object. Another finds adjectives modifying nouns.
• Positional heads. Heads that attend to specific relative offsets — three tokens back, eight tokens back — implementing a kind of learned positional pattern.
• Induction heads. Heads that implement a pattern like “whenever I see the bigram X Y, and later I see X again, predict Y”. These are thought to be central to in-context learning and few-shot prompting.

None of these were designed. They all emerge from training. The model discovers that these patterns are useful for next-token prediction and allocates heads to them.

9.3 Why this matters for code

Code is rich in long-range structure: matching brackets, scope, variable bindings, indentation, function-signature awareness. Each of those is a routing pattern — exactly the kind of work a single head can specialise in. Take a snippet:

def total_price(items, tax_rate):
    subtotal = sum(item.price for item in items)
    tax = subtotal * tax_rate
    return subtotal + tax

Several relationships at once need to fire for the model to generate or reason about this:

• tax_rate on line 3 must bind back to tax_rate in the parameter list (variable-binding head).
• subtotal on line 4 must respect its definition on line 2 (scoping head).
• The closing ) on line 2 must match the opening ( (bracket-matching head).
• Each line’s leading whitespace must respect the function’s indentation level (indentation/positional head).
• The return line must understand it is inside a function whose signature listed two parameters (function-signature awareness — typically several heads cooperating across layers).

One head per pattern, in parallel, every layer. This is why code models work: many simple routing rules combine into structural understanding. It is also part of why a single model trained on next-token loss alone learns to produce well-bracketed, well-indented, scope-respecting code without ever being told the rules — different heads carve out the rules during training because the prediction loss rewards them.

9.4 What multi-head adds

Multi-head attention is not a different mechanism from attention. It is several narrower attention mechanisms run in parallel on disjoint slices of the embedding space, plus one extra mixing projection \(W_O\). Single-head total parameters per layer: \(3 d^2\). Multi-head total: \(4 d^2\) (the extra \(d^2\) is exactly \(W_O\)). For our toy with \(d = 6\), that is 144 attention parameters per block — independent of \(T\), independent of the actual content of the input.

We have now fully built the attention sub-layer — single-head, multi-head, causal. There is one elephant in the room we have been postponing: nothing in any of these formulas distinguishes “the cat sat on the mat” from “the mat sat on the cat”. Position needs to be injected. The next chapter is the proof that we cannot get away without it, and the survey of how three families of models actually do it.

10. Positional Encoding

The attention sub-block works on a matrix \(H \in \mathbb{R}^{T \times d}\). We have been pretending that the row order of \(H\) carries some meaning — that row 3 is the third token, and that is why it sits at row 3. This chapter shows that, as far as attention is concerned, that pretence is empty: attention is order-blind by construction unless we explicitly add a position signal. We will state this as a small lemma, prove it, then show the three mechanisms real models use to fix it.

10.1 The blind spot: attention sees a multiset

Take the sentence the cat sat on the mat and scramble it into the cat mat on the sat. Same multiset of tokens. What changes in attention?

Structurally, nothing. The set of pairwise scores \(\mathbf{q}_i \cdot \mathbf{k}_j\) is identical — the same set of pairs is being scored, just relabeled. Each score depends on which two tokens are involved, never on where they sit. Attention’s input is, mathematically, an unordered set of token vectors.

Without help, the transformer is just a very expensive bag-of-words model. A human reader tells the cat sat on the mat from the cat mat on the sat instantly — in one of them the cat is sitting, in the other the mat is. An attention layer given only the bare embeddings would see two identical bags and produce the same output for both.

10.2 Permutation equivariance, formally

Let us make this precise. A permutation matrix \(P \in {0, 1}^{T \times T}\) has exactly one 1 per row and per column. Two facts: \(P X\) permutes the rows of \(X\), and \(P^\top P = I\) (permutation matrices are orthogonal).

Lemma (permutation equivariance of attention). For every permutation matrix \(P\) and every \(X \in \mathbb{R}^{T \times d}\),

Proof sketch. Substitute \(P X\) into the formula:

Pull \(P\) out: \((P X W_Q)(P X W_K)^\top = P (X W_Q)(X W_K)^\top P^\top\). Row-wise softmax commutes with row permutations, so the \(P^\top\) inside softmax cancels the \(P\) on the same side, leaving \(P \cdot \operatorname{softmax}(\ldots) \cdot \operatorname{nothing}\). The right factor \(P X W_V\) keeps its leading \(P\). Combined: \(P\) factors out front. \(\square\)

Consequence in plain English. Each output row depends only on which tokens are present, never on their order. Permute the inputs, and you get exactly the same output rows in the same permuted order.

You might think the causal mask saves you. It does not, quite. The causal mask restricts which set of past tokens is visible to each query — but not their order within that set. Take prefix {the, cat, sat, on, the, mat} versus prefix {the, mat, sat, on, the, cat} — same multiset. The query at position 7 sees an identical set of past tokens in both cases; the model cannot tell the two prefixes apart from their causal contents alone. A weaker equivariance survives — \(\operatorname{Attention}_M(P X) = P \cdot \operatorname{Attention}_M(X)\) holds iff \(P M P^\top = M\), which for the strict lower-triangular mask collapses to “the only \(P\) that works is the identity” — but the loss of order inside the past is still real.

The conclusion is the same with or without the mask: without a position signal, the model literally cannot tell “John loves Mary” from “Mary loves John”. The two sentences embed identically as multisets. The attention output for loves is the same in both. We need to break equivariance on purpose.

10.3 The fix: inject position into the input

We do not want to change attention itself — equivariance is what gives us parallelism, length generalisation, and the absence of recurrence. Instead, we change the input, so that the same token at different positions becomes a different vector:

Now \(\operatorname{Attention}(P (X + E_{\text{pos}})) \neq P \operatorname{Attention}(X + E_{\text{pos}})\) in general, because \(P E_{\text{pos}} \neq E_{\text{pos}}\). The equivariance is broken on purpose. The model can now distinguish position 1 from position 5 because their input vectors differ.

10.4 Three standard ways to do it

Sinusoidal positional encodings (the original 2017 paper). Each dimension of the position signal is a sine or cosine at a different frequency:

No learnable parameters. The matrix is computed. For a new \(T\), evaluate the formula for the new number of rows — there is no “stored sinusoidal matrix” that has to be resized.

Learned absolute positional embeddings (GPT-2 / GPT-3 style). A second lookup table, keyed on position instead of token identity:

For input length \(T \leq n_{\max}\), take the first \(T\) rows. This is the one place in the entire architecture where a constant derived from the maximum sequence length appears in a weight shape. The constant is \(n_{\max}\), not \(T\). The parameter shape is fixed at training time; it does not change when an input of different length comes in. If the input is shorter, we use a prefix of rows. If the input is longer than \(n_{\max}\), the architecture has no row to look up — the model literally cannot process it. This is the hard context-window ceiling of GPT-2 and GPT-3.

Rotary Positional Embeddings, RoPE (LLaMA, Mistral, Qwen, most modern frontier models). Instead of adding a position vector to \(X\), rotate pairs of coordinates in \(Q\) and \(K\) inside attention by an angle that depends on position:

Apply this to consecutive 2-coordinate slices of every query and key vector. The dot product \(\mathbf{q}_t \cdot \mathbf{k}_s\) then depends only on the relative offset \(t - s\) — RoPE is a relative encoding, not absolute. Zero learned parameters, no \(n_{\max}\)-shaped matrix, and the formula extrapolates (with degrading quality) beyond the training length. That last property is the main reason RoPE has won the modern open-weight market.

The differences between these matter for long-context behaviour, for extrapolation, and for how well the model handles context windows it was not trained on. For our purposes today the shared principle is what counts: some per-position signal gets injected before attention sees it. That is the + Position step in our chapter-3 picture.

10.5 The position-dilution observation

One observation to log before moving on. With learned absolute positions, the same token at position 50 and position 50,000 picks up very different position vectors. The model’s representation of an instruction at the start of a 50,000-token conversation is, position-wise, far away from where it would be at the start of a 500-token conversation. In principle, attention can learn to handle this. In practice, models with learned absolute positions can struggle with tokens that sit far back — partly because those positions are seen less often, and less variably, during training, partly because the routing patterns were shaped on shorter contexts. How badly, and whether it happens at all, depends on the positional scheme and the training distribution (RoPE-style models extrapolate further than learned-absolute ones). We come back to this in chapter 16.

With position injected, the input matrix \(H_0 = X + E_{\text{pos}}\) carries both what each token is and where it sits. Attention can now do its work without losing the difference between cat sat on mat and mat sat on cat. The next chapter turns to the second sub-block of the layer: the FFN.

11. Feed-Forward Network

After multi-head attention, each token has a new representation — one that has absorbed information from all the other tokens relevant to it. Attention has done its job of gathering. But it has not yet done much thinking. All attention does is produce weighted sums of existing vectors. It cannot, on its own, transform those vectors into fundamentally different kinds of representations — for example, turning “a pronoun token that has absorbed information about its antecedent” into “the concept of the animal it refers to, enriched with facts about animals”.

That next step is the job of the feed-forward network, or FFN, also called the MLP sub-layer. It is the second half of a transformer block.

11.1 The shape

The FFN is a two-layer multi-layer perceptron applied independently to each token’s vector. There is no mixing across tokens here — every position is processed in isolation. Concretely, for a token column vector \(\mathbf{z} \in \mathbb{R}^d\) (working on its row \(\mathbf{z}^\top\) per the §5.2 layout):

where \(W_1 \in \mathbb{R}^{d \times 4d}\) and \(W_2 \in \mathbb{R}^{4d \times d}\). Two linear transformations, with a non-linearity in the middle. The critical detail is the expansion factor: the first matrix expands the dimension by a factor of 4 (or sometimes 8 in newer architectures). For GPT-3 with \(d = 12{,}288\), the hidden dimension inside the FFN is \(4d = 49{,}152\). The model briefly blows the token’s representation up to four times its usual size, applies a non-linearity, and then compresses it back down.

11.2 Where the parameters live

Here is the punchline that tends to surprise people when they first encounter it: the FFN, not the attention, is where most of the transformer’s parameters sit. For each layer:

• Attention’s weights: \(4 d^2\) per layer (Q, K, V projections plus \(W_O\)). For \(d = 768\): about 2.4 million parameters.
• FFN’s weights: \(W_1\) is \(d \times 4d = 4d^2\), and \(W_2\) is \(4d \times d = 4d^2\). That is \(8 d^2\) per layer. For \(d = 768\): about 4.7 million parameters.

Per layer, the FFN holds roughly two-thirds of the parameter budget. Over the full 96-layer stack of GPT-3, that is tens of billions of parameters dedicated to per-token, non-attention-based transformation. Most of what we call “the model’s knowledge” — that is, the facts and patterns it picked up from its training corpus — lives in these weights, not in attention.

11.3 What the FFN actually does

A useful way to think about the FFN is as a per-token key-value memory. The first linear layer \(W_1\) reads the token’s current representation and computes a score for each of \(4d\) internal “keys”. The GeLU non-linearity gates which of those keys are active (roughly: a soft mask that says “these \(k\) keys match”). The second linear layer \(W_2\) then produces the output as a weighted combination of \(4d\) stored “values” — one per key. So the FFN behaves like a giant associative lookup: given a token representation, retrieve the relevant pieces of the model’s learned knowledge and blend them into the output.

This framing — which comes from a line of interpretability research in 2020–2023 — helps explain why the FFN is the parameter-heavy sub-layer. Attention is routing. FFN is storage. Routing needs relatively few parameters; storage needs a lot.

11.4 The practical consequence

The short version of the contrast, worth carrying with you into Lecture 6: attention gathers information, the feed-forward network processes it. A token first collects relevant context from other tokens (attention), then thinks about what it has just seen (FFN). The two sub-layers alternate across the full depth of the model, each handing the enriched representation to the next.

That alternation, stacked many times, is what we look at next.

12. The Complete Block, Stacked

We have built the attention sub-block (chapters 7–9), justified positional encoding (chapter 10), and named the FFN (chapter 11). Two pieces are missing before the architecture stacks cleanly: the residual connections that let gradients survive 96 layers of depth, and the layer normalisation that keeps activations sane along the way.

12.1 Residual connections

For each sub-layer (attention or FFN), the input is added to the output before being passed on:

The sub-layer does not replace the representation — it refines it. The original vector can “skip” through the layer, unchanged, via the residual path. The sub-layer only needs to compute a correction that gets added on top.

Two consequences follow. First, a residual path exists from the input all the way through all 96 layers, end to end. Gradients have a clean highway back to every early layer during training, which is why deep transformers can be trained at all — vanishing gradients stop being a death sentence when the gradient has a direct path. Second, the final representation at layer 96 still contains a trace of the original embedding at layer 0. Each layer built on top of the previous rather than erasing it. The model accumulates. It does not overwrite.

12.2 Layer normalisation

After each sub-layer (or, in most modern transformers, before each sub-layer), a layer norm step renormalises the representation. For each token’s \(d\)-dimensional vector, it subtracts the mean across features, divides by the standard deviation, and then applies a learned per-dimension scale and shift:

where \(\mu\) and \(\sigma\) are the mean and standard deviation of the \(d\) entries of \(\mathbf{z}\), and \(\boldsymbol{\gamma}, \boldsymbol{\beta} \in \mathbb{R}^d\) are learnable. The \(\odot\) is elementwise multiplication.

Think of it as recalibration after each step. Without it, the magnitude of activations can drift — growing or shrinking unpredictably as the stack deepens — and the model becomes impossible to train stably.

The 2017 paper put layer norm after the sub-layer and the residual: \(\operatorname{LN}(\mathbf{z} + \operatorname{SubLayer}(\mathbf{z}))\). This is post-LN. It worked at small scale but became unstable at larger scales. Modern practice puts layer norm before the sub-layer: \(\mathbf{z} + \operatorname{SubLayer}(\operatorname{LN}(\mathbf{z}))\). This is pre-LN, used by GPT-2, GPT-3, Claude, LLaMA, and every mainstream production model today.

12.3 The standard block

Putting attention, FFN, residuals, and layer norms together, a single transformer block in a modern model looks like this:

1. \(\mathbf{z}’ \leftarrow \mathbf{z} + \operatorname{MultiHeadAttention}_M(\operatorname{LN}_1(\mathbf{z}))\)
2. \(\mathbf{z}’’ \leftarrow \mathbf{z}’ + \operatorname{FFN}(\operatorname{LN}_2(\mathbf{z}’))\)

Two sub-layers, two residuals, two layer norms. Every intermediate quantity is in \(\mathbb{R}^{T \times d}\). The input matrix enters at step 1; the output matrix emerges at step 2, ready for the next block. That is one layer.

12.4 The architecture, in retrospect: a 5-stage build

It is worth pausing to look at the same architecture from a pedagogical angle: what does each ingredient add, and in what order would you discover it if you were inventing transformers from scratch?

1. Single-head attention plus FFN. Attention mixes information across tokens; FFN transforms each token locally. This minimal pair is already enough to do “content-dependent routing” — the architecturally novel thing.
2. Add residual connections. Now sublayers learn corrections, not replacements. The block becomes stackable.
3. Replace single-head attention with multi-head attention. Several routing patterns in parallel. Same parameter budget on Q/K/V, plus one extra \(W_O\) projection.
4. Add layer normalisation. Stabilises the activations across depth so the stack can be trained reliably.
5. Stack \(N\) such blocks. Now you have the full decoder-only transformer.

Each stage adds one genuinely new idea. The slide deck and the manuscript present this as a separate progression; the lecture leads with the more concrete deck order (overview → attention → masking → multi-head → …), but knowing both views helps you talk about the architecture later.

12.5 Stacking N blocks

A real model stacks many blocks. The number of blocks — the model’s depth — is one of its headline hyperparameters.

• GPT-2 (2019): 12 layers in the small model, up to 48 in the XL.
• GPT-3 (2020): 96 layers.
• Modern frontier models: 80 to 120+ layers.

Each layer applies the same recipe with its own independently learned parameters. The hidden state shape is preserved at every depth: \(H_0, H_1, \ldots, H_N \in \mathbb{R}^{T \times d}\). This shape stability is what makes the architecture so modular — different ideas can be inserted into the block while preserving the same outer interface.

Empirically, different layers specialise. Early layers (~1–20) capture low-level patterns: token adjacency, basic syntax. Middle layers (~20–60) handle semantic relationships: coreference, entity tracking. Late layers (~60–96) carry task-specific reasoning: code style, instruction following. None of this is designed; it is an emergent property of gradient descent on next-token prediction at scale.

12.6 Parameter accounting per block

A quick reality check. Each layer has:

• Attention parameters: \(\approx 4 d^2\) (Q, K, V, O projections).
• FFN parameters: \(\approx 8 d^2\) (\(W_1\) and \(W_2\)).
• Layer norm parameters: \(\approx 4d\), negligible.

Total per layer: \(\approx 12 d^2 + 9d\). For GPT-3, \(d = 12{,}288\), so that is roughly \(1.8\) billion parameters per layer. Times 96 layers: \(\approx 175\) billion — which matches GPT-3’s headline parameter count. Every single forward pass through this model, for every single generated token, exercises all 175 billion of those parameters. This is the mechanical reason behind the observation in chapter 15: 500 output tokens means 500 complete forward passes through a 175-billion-parameter machine.

We have now followed a token’s representation all the way to the top of the stack. It has been embedded, positionally encoded, passed through attention and FFN \(N\) times, and emerged at the top with a representation that encodes both what it is and what is likely to come after it. The next chapter projects that final representation back into the vocabulary — and lands the architectural punchline this lecture has been building toward.

13. The Output Head and the Key Insight

After the final transformer layer, we have a matrix \(H_N \in \mathbb{R}^{T \times d}\) where row \(t\) is the fully-processed representation of the token at position \(t\). Every row has been through every layer. Every row has absorbed context from every other token through every attention head. Every row has been rewritten \(N\) times.

For next-token prediction we only care about one of those rows: the last one. That is the row corresponding to the final input token, and it is the one whose representation has attended to every previous token and is therefore the most informed about “what should come next”.

13.1 The output projection

The output head turns that single \(d\)-dimensional vector into a prediction over the full vocabulary. Architecturally it is one linear projection:

where \(\mathbf{h}_T \in \mathbb{R}^d\) is the last row of \(H_N\) and \(W_U \in \mathbb{R}^{V \times d}\). The output \(\boldsymbol{\ell} \in \mathbb{R}^V\) is one real number per vocabulary entry — the logits. They are unbounded; they can be positive, negative, or zero. Higher logits mean the model thinks that vocabulary entry is more likely to come next.

13.2 Weight tying

Look at the shapes: the embedding matrix \(E\) is \(V \times d\), mapping token IDs to \(d\)-dimensional vectors. The output projection \(W_U\) is also \(V \times d\), mapping back to vocabulary scores. They are the same shape.

In most modern transformers, they are literally the same matrix: \(W_U = E\). This is weight tying. Two reasons it makes sense. First, parameter efficiency — you save \(V \times d\) parameters (for GPT-3, about 600 million) by not duplicating the work. Second, more conceptual: if the embedding direction for the token cat is what “means” cat on the input side, then the best way to score “is the next token cat?” is to check how well the final hidden state aligns with that same direction. The logit for cat ends up as \(\mathbf{h}_T \cdot E_{\text{cat}}\) — a dot product. The model bookends itself with the same vocabulary in the same geometry.

Weight tying turns the output head into a dot-product similarity check between the final token’s contextual representation and every row of the embedding matrix. Whichever vocabulary row is most aligned wins.

13.3 The full parameter count

Time to count everything in the model. With weight tying, the final tally is:

For GPT-3 (\(V = 50{,}257\), \(d = 12{,}288\), \(N = 96\), no \(n_{\max}\) term needed because we use the simpler form): plugging in gives \(\approx 175\) billion parameters.

For our toy (\(V = 10\), \(d = 6\), \(N = 3\), \(H = 2\), \(n_{\max} = 128\), learned positional, tied output head): \(60 + 768 + 1{,}458 = 2{,}286\) parameters total.

Now look at what is not in the formula.

13.4 The key insight: T is not in the parameters

Look at every term in the parameter count: \(V \cdot d\), \(n_{\max} \cdot d\), \(N \cdot (12 d^2 + 9 d)\). Each one is a function of \(V\), \(d\), \(N\), \(H\), and at most \(n_{\max}\). The actual sequence length \(T\) does not appear anywhere.

This is not a coincidence. It is the deep architectural fact that makes the transformer one model instead of a family of models. Walk through every weight you have met:

• Embedding \(E \in \mathbb{R}^{V \times d}\). No \(T\).
• Positional embedding \(E_{\text{pos}} \in \mathbb{R}^{n_{\max} \times d}\) (only if learned absolute). No \(T\) — the constant is \(n_{\max}\), an architectural ceiling, not the input length.
• Attention projections \(W_Q, W_K, W_V \in \mathbb{R}^{d \times d_h}\) per head. No \(T\).
• Output projection \(W_O \in \mathbb{R}^{d \times d}\). No \(T\).
• FFN \(W_1 \in \mathbb{R}^{d \times 4d}\), \(W_2 \in \mathbb{R}^{4d \times d}\). No \(T\).
• LayerNorm \(\gamma, \beta \in \mathbb{R}^d\). No \(T\).
• Output head \(W_U \in \mathbb{R}^{V \times d}\). No \(T\).

Where does \(T\) live? Only in activations. The hidden states \(H_\ell \in \mathbb{R}^{T \times d}\) carry \(T\) in the row dimension. The attention scores \(S \in \mathbb{R}^{T \times T}\) carry \(T\) in both dimensions — but \(S\) is an activation computed afresh on every input, not a stored weight. \(T\) is a summation index inside the formulas, not a dimension of any parameter.

The key sentence. All these dimensions — \(V\), \(d\), \(N\), \(H\) — are baked into the parameters \(\theta\). Once chosen, they are fixed for the lifetime of the model. The sequence length \(T\) is not — the same parameters handle any \(T\) up to \(n_{\max}\).

That is what makes a transformer one model that generalises across sequence lengths instead of a different model per length. The training pass sees thousands of length values and updates \(\theta\) in a single shared parameter space. Inference at a brand-new length just plugs that length into the activation shapes and runs the same \(\theta\). Differentiation, optimisation, parameter storage all happen in the shared space. \(T\) only ever appears as a summation index inside activations.

This is the architectural pay-off the lecture has been building toward. Every previous chapter is, retrospectively, an instance of it: the embedding lookup is a per-row operation, the attention projections are per-row, the FFN is per-row, and even the attention mixing — which couples rows — does so through a content-dependent activation \(A \in \mathbb{R}^{T \times T}\) that is computed, not stored. The model is shape-uniform in \(T\).

13.5 A brief look at modern open-weight LLMs

A short note before we move on. The block we built — dense multi-head attention, GeLU FFN, learned absolute positions, tied output head, around \(\sim 1/3\) attention / \(\sim 2/3\) FFN parameter split at scale — is the GPT-2 / GPT-3 baseline. Modern open-weight models depart from this baseline along four named axes:

• RoPE instead of learned absolute positions (LLaMA, Mistral, Qwen). Zero learned-position parameters; better length extrapolation.
• GQA / MQA (Grouped-Query / Multi-Query Attention). Reduces the K and V projections by sharing them across multiple query heads — pushes the dense models’ parameter share even further toward the FFN. Llama-2 70B sits at roughly 17 % attention / 82 % FFN.
• SwiGLU instead of GeLU FFN. Slightly different non-linearity; usually a wider FFN to match.
• MoE (Mixture of Experts), e.g. Mixtral. The FFN block becomes a router that picks a sparse subset of experts per token. Total parameters and active parameters per token become two different numbers.

None of these change the forward-pass story this lecture told. They change parameter counts, length-extrapolation behaviour, and inference cost, but the shape \(\mathbb{R}^{T \times d}\), the attention operation, and the “T not in the parameters” property all survive. Lecture 6 picks up here.

We have now mapped the entire forward pass from input to logits. Two operational steps remain: turning logits into a chosen token (next chapter), and looping the whole thing to generate more than one token at a time (chapter 15).

14. Picking the Next Token

We have a vector of logits — for our toy, ten of them; for GPT-3, 50,257. We need to produce one token. The question of how to turn logits into a concrete choice has more moving parts than you might expect, and every moving part corresponds to a setting you can tune from a real API call.

14.1 Softmax — from logits to probabilities

The first step is unambiguous. Run softmax across the logit vector:

Same operation we used in attention (chapter 7, Step 4), applied now to vocabulary scores. All outputs are positive; they sum to 1; higher logits become exponentially higher probabilities. A logit of 8.3 for validate and a logit of 7.1 for check might, after softmax, become probabilities like 0.77 and 0.23 respectively — most of the probability on the top candidate, a non-trivial share on the runner-up, long tail of nearly-zero for everything else.

Why those properties hold is a one-line check. The numerator \(e^{\ell_i}\) is positive because \(e^x > 0\) for any real \(x\) — so every \(p_i > 0\), and no token ever gets exactly zero probability (the worst-ranked token in a 50,000-vocab distribution still has some mass, however astronomically small). And the sum being 1 falls straight out of the definition:

The denominator is literally the sum of all the numerators, so dividing by it normalises the vector into a proper probability distribution. That normalisation step is the whole point of softmax — exponentiation alone would give you positive numbers; the division by the sum is what turns positive numbers into probabilities.

This is the probability distribution over the next token. It is what every AI coding tool is quietly carrying around every time it generates.

A subtle but important point about what happens next. The model does not just pick the highest-probability token. It draws a random token from this distribution, where each token’s chance of being drawn equals its probability. If validate has \(p = 0.71\) and check has \(p = 0.21\), then across many runs of the same prompt you would expect validate to be picked ~71 % of the time, check ~21 % of the time, and the long tail of low-probability tokens almost never — but not impossibly. This is what “sampling from a distribution” means, and it is the reason the same prompt can give you different completions on different calls.

A natural follow-up question at this point: why bother with the random draw at all? Why not just pick the highest-probability token every time and be done with it? Three honest reasons, each developed in detail later in the course.

1. Argmax discards almost all of the information the model produced. The forward pass spent enormous compute — every attention head, every FFN, every layer — building a full probability distribution over the vocabulary. When that distribution is sharp, argmax and sampling agree. When the distribution is flat — say, the top token at 38 % with several runners-up between 15 % and 25 % — the model is honestly uncertain, and argmax pretends it isn’t. Sampling uses the distribution the way it was meant to be used; argmax throws away 99 % of it and reports only the rank.

2. Argmax is fully deterministic — same prompt, same output, every time. For factual lookup this is a feature (“Paris” every time, please). For anything creative or open-ended (taglines, name generation, alternative phrasings, brainstorming) it is a fatal limitation. The “variation desired” regime in Lecture 6 §10.1 only exists because sampling exists as the default.

3. Pure greedy decoding tends to break in practice. Always picking the argmax has well-documented failure modes — looping (“…the cat sat on the mat. The cat sat on the mat. The cat sat…”), phrase repetition, mode collapse, and the beam search curse that Lecture 6 §10.3 names by name. There is also a closely related problem — the first-token cascade, in which a single wrong argmax pick at step 1 conditions every subsequent token — which we develop in §14.5. A small amount of stochasticity breaks these attractor states and escapes the cascades; pure argmax often does neither.

A natural concern, once you accept that sampling is random: can a result ever be reproduced? Yes — sampling uses a seeded pseudorandom number generator (PRNG), so with control of the seed the same \((\boldsymbol{\ell}, \text{seed})\) pair always produces the same token. On a local model the whole call can be made bit-for-bit reproducible if you also fix the model checkpoint and use deterministic GPU kernels. On a hosted API, however, reproducibility is only “mostly”: even with a seed and even at \(T = 0\), model-version updates, request batching, replica routing, and GPU floating-point non-determinism can shift the output. The practical consequence is to write tests against properties of LLM outputs (does it return valid JSON? does it mention X?) rather than against exact byte-for-byte strings. The engineering side of this lives in Lecture 6 §10.3.

The special case where the model does always pick the argmax — equivalent to making the highest-probability token’s share equal to 1 — has a name and a knob, and that is the next subsection.

14.2 Temperature — controlling how peaked the distribution is

Temperature is a single scalar knob that scales logits before softmax:

Notice what this division does not do. Dividing every logit by the same positive \(T\) preserves their ordering: if \(\ell_k > \ell_i\) for all \(i \neq k\), then \(\ell_k / T > \ell_i / T\) too, and since \(e^x\) is monotonically increasing, the argmax of the softmax is unchanged. Temperature shifts the height of the peak, not its location. Whatever token has the largest logit before scaling still has the largest probability after — at any \(T > 0\). What changes is how much of the probability mass that winning token captures, and how much leaks to the runners-up.

There is also a cleaner way to see what this whole formula is doing, geometrically. Dividing the logit vector by \(T\) is just horizontal scaling — the same operation that turns a 1D function \(f(x)\) into \(f(x/T)\), generalised to several dimensions and applied to the input of softmax. For \(T > 1\) the curve stretches horizontally; for \(T < 1\) it compresses; the peak height stays the same. The Gaussian below is just a generic 1D function chosen for illustration — the operation is what matters.

Softmax also has one more property worth naming explicitly: it is invariant to additive constants. Adding the same scalar \(c\) to every logit changes nothing — \(\text{softmax}(\boldsymbol{\ell} + c\mathbf{1}) = \text{softmax}(\boldsymbol{\ell})\). What softmax actually sees is not the raw logits, only their pairwise differences. So scaling logits by \(1/T\) is in effect scaling all of those pairwise differences by \(1/T\) too: shrink them (\(T > 1\)) and the exponentials cluster, the distribution flattens; blow them up (\(T < 1\)) and the exponentials spread, the distribution peaks. Every quantitative observation about temperature in the rest of this chapter — and in Lecture 6 chapter 10 — follows from those two facts: horizontal scaling of the input and invariance to additive shifts.

(The reason this knob is called temperature rather than just scaling factor goes back to statistical mechanics. We come back to that in Lecture 6 §10.6.)

The effect of this one division is dramatic.

• \(T = 0\) (greedy). Divide by zero, effectively. The output collapses onto the highest-logit token with probability 1. The model becomes deterministic: “always pick the best”. This is what you use when you want the safest, most predictable completion — generating code that compiles or answering a factual question.
• \(T = 0.3\) (low). The distribution is sharpened but not collapsed. Top tokens dominate but occasionally a runner-up wins. Good for code that needs to vary slightly across runs (e.g. name generation, unit-test synthesis).
• \(T = 0.7\) (typical). The default for most chat-style APIs. A balance: top tokens are strongly preferred, but there is enough breadth that outputs feel natural and not robotic.
• \(T = 1.0\) (unscaled). The distribution that actually came out of the model’s head, unmodified. Often what researchers report.
• \(T = 1.5\) or higher. The distribution is flattened; improbable tokens get meaningful probability mass. Creative, weird, occasionally incoherent. Useful for brainstorming. Dangerous for code.

The same probability distribution at different temperatures is strikingly different to sample from. At \(T = 0.3\), “The capital of France is ___” produces Paris every time. At \(T = 1.5\), it will sometimes produce Lyon, occasionally Marseille, very rarely Berlin. The model has not changed. The sampling has.

14.3 Top-k and top-p (nucleus) sampling

Temperature alone is not enough. Even at moderate temperatures, softmax assigns some probability to pathologically rare tokens — the long tail of the vocabulary that almost never makes sense in any context. Sampling from it occasionally produces genuinely broken output.

Two filtering techniques tame the tail.

• Top-k sampling. Before sampling, keep only the \(k\) highest-probability tokens; zero out the rest; renormalise. Typical \(k\) values are 40, 50, 100. Simple, effective, but \(k\) is a fixed cut-off that does not adapt to how confident the model is.
• Top-p sampling (nucleus). Before sampling, sort the tokens by probability and walk down the list, accumulating mass until the cumulative probability crosses \(p\). Keep that set (the nucleus); zero out everything below it; renormalise the survivors so they sum to 1 again. Typical \(p\): 0.9 or 0.95. The goal is the same as top-k — prevent the model from ever sampling a token it considered nearly impossible — but the cut is adaptive: when the model is confident, very few tokens qualify and the cut is sharp; when uncertain, many qualify and the cut is wide. Nucleus sampling has largely replaced plain top-k in production for exactly this reason.

A concrete look at the nucleus. Take the dragon-drinks running example from chapter 1: at \(T = 1.0\) the next-token distribution over the 10-token toy vocabulary sorts like this (highest probability first):

At \(p = 0.9\), the nucleus stops after glowing (cumulative just crosses 0.9). Sampling sees three tokens. At \(p = 0.95\), five. At \(p = 0.99\), eight. The two deep-tail tokens (Drinks, <BOS>) are excluded regardless of which threshold you choose — they were never going to be plausible continuations, and excluding them costs the model nothing while removing a small but real risk of weird output. That is the entire point of top-p.

On a flatter distribution — for example at \(T = 2.0\), or at a genuinely ambiguous step — the same \(p\) would keep many more tokens. The number of survivors emerges from the shape of the distribution rather than being chosen up front, and that is the adaptivity that top-k cannot match.

In practice, modern API calls combine the techniques: a temperature (typically 0.7), a top-p (typically 0.95), and occasionally a top-k. You are tuning multiple knobs on the same distribution. Once those filters are applied, a single sample is drawn — a token is picked — and we move on.

14.4 A worked mini-example

Suppose the logits for the next token look like this:

(These numbers are illustrative.) Watch how the shape of the distribution shifts. At \(T = 0.3\) the model essentially always picks validate. At \(T = 1.0\) there is meaningful chance of picking check or even verify. At \(T = 2.0\) a long tail of improbable tokens suddenly has appreciable combined mass — which is where weird outputs come from.

14.5 What sampling has to do with Lecture 6

Two threads to mark for next week. First: temperature is the primary knob that controls the hallucination vs. reliability trade-off. High temperature makes the model more likely to emit a plausible-sounding but incorrect answer, because it dips into the long tail. Second: even at temperature 0, the model still emits one token at a time, and each of those tokens conditions everything that follows. A single wrong greedy pick early in a generation can cascade into a completely hallucinated paragraph. We will return to this in Lecture 6.

For now: we have picked one token. That token becomes part of the sequence. And then — the part people tend to miss — the entire forward pass runs again. That is the auto-regressive loop, and it is the last mechanism we need.

15. The Auto-Regressive Loop and Live Demo

It is tempting to think that once the model has “understood” a prompt, generating 500 tokens of response is cheap — maybe 500 small local steps building off some shared internal state. It is not. Every single generated token is the output of a complete, full-depth forward pass, over the entire sequence so far.

15.1 The loop, concretely

Starting from an input prompt of \(T_0\) tokens:

1. Run the full forward pass — embedding, all \(N\) layers of attention and FFN, output head — on the \(T_0\)-token sequence.
2. Extract the last token’s logits.
3. Apply softmax + temperature + top-p; sample one token.
4. Append that token to the sequence. Now the sequence has \(T_0 + 1\) tokens.
5. Run the entire forward pass again, now on \(T_0 + 1\) tokens.
6. Extract the last token’s logits. Sample the next token.
7. Repeat until a stopping condition (end-of-sequence token, max length, tool-call trigger).

A 500-token response is 500 full forward passes. A 5,000-token response is 5,000 full forward passes. Generating long outputs is not a subtle optimisation — it is a linear multiplier on compute.

15.2 Why the sequence grows monotonically

At every step we append. We never remove anything. We never overwrite. The sequence grows strictly monotonically, and the model sees every previous token — including every previous generated token — at every pass.

This has a subtle but important consequence: the model is free to condition on its own output. The token it generates at step \(k\) becomes an input at step \(k + 1\). If the model made a small mistake at step \(k\), that mistake is now in its context and shapes everything after. If the model committed at step \(k\) to a particular structure (“I am going to answer this in three bullet points”), that commitment is in the context and the model will tend to follow through.

15.3 KV caching — the optimisation everyone uses

A reasonable question: if every token requires a full forward pass over the whole growing sequence, isn’t this quadratic? For a 50,000-token context and a 500-token response, are we doing 500 × 50,500-token forward passes?

In naive implementations, yes. In every production system, no — because of an optimisation called the KV cache. The observation is that the keys \(K\) and values \(V\) computed for the first \(T_0\) tokens on the first forward pass do not change on subsequent passes (the tokens are the same). So we cache them. Pass \(T_0 + 1\) only needs to compute \(K\) and \(V\) for the newly appended token and then reuse the cached matrices for everything else. The attention computation for each new token is linear in the accumulated sequence length rather than recomputing the whole thing.

This is purely an engineering optimisation. It does not change anything about the conceptual model. Every token still sees every previous token. The arithmetic is identical; only the work we redo is reduced. KV caching is the reason 200K-token contexts are tractable at all, and it is the reason the very first token of a response sometimes takes a long time to appear (building the cache) while every subsequent token is fast (reusing it).

15.4 Live demo — Transformer Explainer

Everything we have done until now has been on paper. Six steps, three matrices, 12,288 dimensions — none of it has been a thing you could see happening. To close out the architecture, put a concrete transformer on a browser page in front of you, with live attention heatmaps, per-layer inspection, and a prompt you can change.

The tool is Transformer Explainer by the Polo Club of Data Science at Georgia Tech. It runs a real GPT-2 in your browser and visualises every sub-block from this lecture on actual inputs. Open poloclub.github.io/transformer-explainer and walk it in chapter order:

• Embedding layer. Type a prompt, see each token become a coloured \(d\)-dimensional bar. Hover over different tokens — the bars are different because they are different rows of \(E\). Chapter 4 made visible.
• Positional encoding. Toggle “show positional encoding” and watch the bars shift. Chapter 10’s + E_pos step in motion.
• A single attention head. Click on a layer, then a head. You get a heatmap — rows are query positions, columns are key positions, brightness is the attention weight. Exactly what we computed as \(\operatorname{softmax}(QK^\top / \sqrt{d_h})\) in chapter 7. Try def binary_search(arr, target): and hover on the : — on at least one head you will see strong attention back to binary_search.
• Multiple heads per layer. Scroll through the 12 heads. Some look almost diagonal (previous-token heads). Some have strong columns. Some are nearly uniform. Chapter 9 in living colour.
• Layer by layer. Compare attention patterns at layer 1 vs layer 6 vs layer 12. Early layers tend to produce short-range, syntax-like patterns; late layers, long-range and task-relevant. Chapter 12.5 at work.
• The output distribution. A bar chart at the top shows the next-token probabilities — chapters 13 and 14.

For your between-lecture work, pick one head in one layer, change the prompt three or four times, and write a paragraph in your reflection log describing what you think that head is “for”. There is no right answer; you are doing a miniature version of professional interpretability research.

16. CLAUDE.md, the Mystery Board, and What Breaks Next Week

We have spent the previous fifteen chapters building the architecture as a self-contained mathematical object. In this closing chapter we connect the architecture back to the concerns that motivated the course — CLAUDE.md, the mystery board, and the puzzle pieces we deliberately left for Lecture 6.

16.1 How CLAUDE.md actually enters the model

When you add a CLAUDE.md file to your project, every one of its tokens encounters the model through the exact same pipeline as every other token. There is no “instruction channel”. Three concrete mechanisms govern its eventual influence on output, in the order an instruction encounters them.

Embedding-layer entrance. Each instruction line — say, “Always use snake_case for function names” — becomes a sequence of token IDs, which become rows of \(E\), which get stacked with the rest of the context into the input matrix \(X\). The tokens are indistinguishable in type from user prompts or from the model’s own previous output. Any leverage CLAUDE.md ultimately has is earned through attention weight downstream, not through any privileged channel at the entrance.

Multi-head dilution. Different attention heads pay attention to different things (chapter 9). Whether the model follows the instruction at some later token depends on whether there is a head, at some layer, that happens to attend from the current generation position back to the instruction tokens. Some heads do; some specialise in patterns that have nothing to do with CLAUDE.md. The effective influence is therefore partial — the rule wins in some heads and loses in others, and the final output is a mixture weighted by \(W_O\). This is one mechanical reason why CLAUDE.md compliance is never 100 %, only high-ish, and why some rules stick more reliably than others.

Context drift over the autoregressive loop. Suppose your CLAUDE.md sits at tokens 200–700 of a 70,000-token conversation. At token 1,000 it is most of what the model has seen. At token 60,000 it is still in the context and still in every forward pass, but it now sits far behind a long history of intervening tokens. Its share of the raw token count has dropped — but that fraction is not the mechanism. Softmax is winner-take-most, so a head that has learned to route back to “rules” can still place high weight on the instruction tokens regardless of how many tokens sit in between. What actually erodes compliance as the conversation grows is subtler — long-range positional behaviour, more competing and possibly conflicting instructions later in the context, and drift away from the training distribution — so the effect depends on the model, the task, and how the context is managed, not on the percentage of the window CLAUDE.md occupies. The model does not forget CLAUDE.md; under accumulated, possibly conflicting context the relevant heads can fail to prioritise it. We call this context drift — a behavioural tendency, not a fixed proportional law.

These three mechanisms — the un-privileged embedding entrance, the fact that only some heads route back to the instruction (so its influence is partial and mixed by \(W_O\)), and the behavioural tendency for instructions to lose priority as the context grows and accumulates competing material — are the architectural reasons CLAUDE.md works partially rather than perfectly. Lecture 7 will build on top of them, looking at the orchestration layer (where does CLAUDE.md text come from and how does it get into the prompt) and at training-time effects.

16.2 Mystery status after Lecture 5

We opened this lecture with three open Lecture-4 mysteries. Time to update the board.

One largely resolved. Four partially resolved. One still mostly untouched. The next two lectures close the rest.

16.3 What breaks next week

We have spent this lecture building the mechanism as if it works perfectly. For well-posed prompts on tasks close to its training distribution, it mostly does. But push any part of this pipeline off-distribution — unusual tokens, confusing instructions, a context window full of irrelevant junk, a topic the training data did not cover — and the same mechanism that produced validate_email for you in Lecture 1 will happily produce a call to a function that does not exist, confidently and with no warning.

Why? Every output is a sample from a probability distribution. The model has no “I don’t know” token. If every plausible next token is wrong, it will still pick the most plausible wrong one. Attention can route to irrelevant context just as easily as to relevant context — it routes by learned similarity, not by truth. The FFN stores patterns from training data, not verified facts. Softmax always gives you a distribution, even when none of the candidates are correct.

Next week we use the exact mechanism we built today to trace where things go wrong. Hallucination, prompt sensitivity, drift, plausibility-bias. You now have the vocabulary for all of them.

17. Lab and Deliverables

17.1 Notebook: Embeddings and Attention Playground

Open the Embeddings and Attention Playground notebook. It runs directly in your browser via JupyterLite (no setup needed) for most sections; the embedding-loading section may prefer Colab — the notebook will tell you which cells need which runtime.

You will:

• Load a small set of real pretrained word embeddings and explore them with cosine similarity — nearest neighbours, code vs prose clusters, the king − man + woman ≈ queen experiment in your own hands.
• Project hundreds of token embeddings down to 2D and 3D with t-SNE and UMAP.
• Implement scaled dot-product attention from scratch in about fifteen lines of NumPy and replicate the worked example from chapter 7 — verify that you get roughly 40 % attention on cat for query it.
• Run the permutation-equivariance demonstration from chapter 10: apply attention to the bare embeddings of shutil.copy(src, dst) and shutil.copy(dst, src) and show the outputs are identical up to permutation. Then add a sinusoidal position signal and show the outputs diverge.

Work through it top to bottom. Each section has questions to answer; write your findings in your reflection log.

17.2 Deliverables

Document your work from today as a reflection log with the following parts.

1. The forward pass in your own words. A paragraph or a drawing of your own making, in any format, tracing the journey of a token from raw text all the way to the sampled next token. Name every stage. This is the single most important artefact you will produce in this lecture, because if you can write it, you understand it.

2. Transformer Explainer head probe. The attention head you chose in chapter 15’s homework task. Describe it: layer number, head number, the prompts you tried, what patterns you saw, and your best hypothesis for what that head is “for”. Under 200 words.

3. Notebook findings. At least three results from the Embeddings and Attention Playground that clarified or surprised you. The order-matters demo in particular: did the numbers actually come out identical (up to permutation) without position, and different with it? Paste one numerical example.

4. A prompting change. Based on everything you now know about attention competition, context curation, and context drift, one concrete change you will make to how you write prompts or CLAUDE.md files going forward.

5. The Key-Insight test. In your own words, in two or three sentences, explain the chapter-13 climax: why is \(T\) not in the parameters of a transformer, and what does that buy us? No formulas required — just the architectural argument.

Bring these to next week. Lecture 6 will assume you have internalised this one — every hallucination we discuss will trace back to a specific part of the pipeline you built today.