05 Lecture 5 Quiz: Inside the Transformer

A transformer emits exactly one token per forward pass. To produce a 500-token response the model runs the forward pass 500 times — appending its own output to the input each time and re-running everything. There is no “summary head”, no parallel multi-token output, no built-in beam-search ranking. The auto-regressive loop is what turns one-token-at-a-time into a full response.

A token ID is a name, not a number. Multiplying ID 11140 by a weight produces a meaningless quantity — there is no sense in which “token 11141” is one unit “more” than “token 11140”. The embedding lookup converts each ID into a row of d real numbers that can be added, dot-producted, and differentiated. This is also why the embedding matrix is learned: gradient descent shapes the rows so that semantically similar tokens end up in similar directions. One-hot encoding is one (very wasteful) way to make IDs continuous, but learned embeddings dominate it.

The embedding is the model’s first impression of each token, independent of context. Disambiguating river bank from savings account is the job of the layers that come after — and the mechanism that does it is attention. Attention pulls information from neighbouring tokens into each token’s representation, so a bank row that has attended to river ends up at a different point in the embedding space from a bank row that has attended to account. Tokenization assigns the same ID to both, positional encoding only adds a position signal, and fine-tuning is not how a pretrained model resolves polysemy at inference.

The dot product is a · b = ||a|| ||b|| cos(theta): a single number that combines “do the two vectors agree on direction” with “how long are they”. Large and positive when the two point in similar directions, zero when perpendicular, negative when opposed. This identity holds in every dimension. Cosine similarity strips out the lengths to give an angle-only measure. The dot product is not a distance, not the angle itself, and not just ||a|| ||b|| (that ignores direction entirely) — it is the scalar that attention uses to score “how relevant is key j to query i” in a single number per pair.

This is the load-bearing distinction of the lecture: attention gathers, FFN processes. Attention is the only operation in the whole transformer that lets one token read information from other tokens. The FFN is a per-position MLP — same weights applied row by row, never combining rows. Without attention the model would be 96 stacked MLPs operating on each token in parallel, with no token ever seeing another. Without the FFN, information would route around but never get reshaped. Residuals carry information through the depth, but they do not mix across positions.

Q/K/V decouple “who listens to whom” from “what gets transmitted”. The query asks a question; keys advertise content for matching; values supply the actual signal that flows out of a selected token. A token can advertise relevance for one reason (its key) and carry useful information for another (its value) — that decoupling is what gives attention its expressive power. K and V are produced by different learned projections W_K and W_V, so they are not computed identically and there is no extra normalisation step on K. Both are used at training and inference alike.

Each entry of Q K^T is a sum of d componentwise products. With standard initialisation, each product has variance 1, so the sum has variance d and typical magnitude sqrt(d). When that magnitude is fed into a softmax, the distribution collapses onto a single winner and the gradient to all other entries dies. Dividing by sqrt(d) restores constant-magnitude inputs so softmax stays in its informative regime. The original Vaswani et al. (2017) paper called this out explicitly. It is not a normalisation into [-1, 1] (that would be cosine similarity, which strips lengths entirely); the row-sum-to-one property comes from softmax itself, not from the division.

The mask M inside softmax((Q K^T + M) / sqrt(d)) sets every above-diagonal entry to -infinity, which becomes 0 after the exponential. The result: query at position i can attend to keys at positions 1, …, i but not to i+1, …, T. This is what forces the model to predict the next token without peeking at it — without the mask, the parallel training loop would let position i read the ground-truth token at position i+1 and the loss would collapse to zero. The mask is part of the architecture, not a windowing scheme — every position sees its full causal prefix, not just the last k tokens. A token does attend to itself.

H heads each have Q, K, V of shape d × d_h with d_h = d/H. Multiplying out: H × 3 × d × d_h = 3 d^2 — exactly the single-head budget, just rearranged into H parallel narrower subspaces. The genuinely new parameters in multi-head are the output projection W_O of shape d × d, which mixes the heads back together and contributes one extra d^2. Total per attention sub-block: 4 d^2. The FFN parameter count is independent of head count.

This is the permutation-equivariance lemma: for any permutation P, Attention(P X) = P · Attention(X). Each output row depends only on which tokens are involved, never on where they sit. The set of pairwise scores is identical between the two sentences — same multiset of tokens, same set of pairs. Without a position signal, a transformer is a very expensive bag-of-words model. That is why E_pos (or RoPE) is added: the input X + E_pos is no longer permutation-equivariant, and the model can finally distinguish “cat sat on mat” from “mat sat on cat”. Residual connections preserve information but cannot recover what was never there to begin with.

Per block: attention is 4 d^2, FFN is 8 d^2 (W_1 is d × 4d and W_2 is 4d × d, each 4 d^2), layer norm is ~ 4d — negligible. So roughly two-thirds of every block’s weights sit in the FFN. Over the full stack of GPT-3, that is tens of billions of parameters dedicated to per-token, non-attention transformation. The interpretability framing is “attention is routing; FFN is storage” — most of what we call “the model’s knowledge” lives in those FFN weights, not in attention. The embedding matrix is shared with the output head and is not per-block.

This is the architectural punchline of Chapter 13. Walk through every weight: embedding V × d, attention W_Q / W_K / W_V / W_O of shape d × ·, FFN d × 4d and 4d × d, layer norm in R^d. None of them depend on T. T lives only in activation shapes — H_l of shape T × d, the score matrix S of shape T × T — which are recomputed on every input. The model is shape-uniform in T, which is what makes a transformer one model that generalises across input lengths instead of a different model per length. Weights do not resize; padding does not enable arbitrary length; there is only one θ.

Of every sub-layer in a transformer block, only attention scales quadratically with T. The score matrix Q K^T of shape T × T dots every query against every key — T^2 entries — and the subsequent mix A V is also O(T^2 · d). Everything else is linear in T: the FFN applies its weights row by row at O(T · d^2) (doubling T merely doubles the cost — it does not quadruple it); the embedding lookup is O(T · d) — V does not appear because each lookup just indexes one row; layer norm normalises within each row at O(T · d). The lecture’s framing is that T × T is “the only place in the entire forward pass where T appears squared in an activation” — and it is an activation, computed fresh on every input, not a stored parameter. This single fact is why every long-context optimisation — FlashAttention, sliding-window, sparse, and linear attention — targets the score matrix.

The auto-regressive loop runs the full forward pass per generated token: read the last position’s logits, sample one token, append, then re-run the entire forward pass on the now-longer sequence. The very first pass — the one that consumes the prompt — is the pass that generates the first token, so there is no separate “prompt pass” on top of 500 generation passes. A 500-token response is 500 full forward passes; a 5,000-token response is 5,000. Generating long outputs is genuinely a linear multiplier on compute, and stacked layers run inside one pass — they are not extra passes.