Home

05 Lecture 5 Quiz: Inside the Transformer

April 2026 (3179 Words, 18 Minutes)

quiz transformer attention embeddings positional-encoding sampling

New: Interactive Quiz Available!

Want to track your progress and get instant feedback? Try our new interactive version with progress tracking, immediate explanations, and score calculation.

Start Interactive Quiz →

Instructions

Question 1: The Forward Pass at the Black-Box Level

What does a transformer produce per single forward pass over a prompt?

A) One next token, sampled from a probability distribution over the vocabulary
B) A complete response of arbitrary length, terminated by an end-of-sequence token
C) A summary embedding that compresses the entire prompt into one vector
D) Several candidate continuations, ranked by probability

Show Answer

Correct Answer: A

A transformer emits exactly one token per forward pass. To produce a 500-token response the model runs the forward pass 500 times — appending its own output to the input each time and re-running everything. The other options describe systems that do not exist: there is no “summary head”, no parallel multi-token output, no built-in beam-search ranking. The auto-regressive loop is what turns one-token-at-a-time into a full response.

Question 2: Why Embeddings Exist

Why does a transformer convert token IDs into dense embedding vectors instead of feeding the integer IDs directly into the network?

A) The IDs are too large numerically and would cause overflow during matrix multiplication
B) Token IDs are labels, not quantities — the network needs vectors it can do calculus on
C) The IDs need to be normalised into the range [0, 1] before they can be passed through a non-linearity
D) The IDs must be one-hot encoded so that no two tokens share any non-zero component

Show Answer

Correct Answer: B

A token ID is a name, not a number. Multiplying ID 11140 by a weight produces a meaningless quantity — there is no sense in which “token 11141” is one unit “more” than “token 11140”. The embedding lookup converts each ID into a row of \(d\) real numbers that can be added, dot-producted, and differentiated. This is also why the embedding matrix is learned: gradient descent shapes the rows so that semantically similar tokens end up in similar directions. One-hot encoding is one (very wasteful) way to make IDs continuous, but learned embeddings dominate it on every dimension that matters.

Question 3: Embeddings Are Context-Independent

The row of the embedding matrix for the token bank is the same fixed vector whether the surrounding sentence is about a river or about a savings account. What part of the architecture is responsible for distinguishing those two senses?

A) The tokenizer, which assigns different token IDs in different contexts
B) The positional encoding, which shifts the embedding vector based on neighbouring words
C) Fine-tuning on the downstream task, which retrains the embedding row per sense
D) Attention in the transformer layers, which mixes information from neighbouring tokens into each row

Show Answer

Correct Answer: D

The embedding is the model’s first impression of each token, independent of context. Disambiguating river bank from savings account is the job of the layers that come after — and the mechanism that does it is attention. Attention pulls information from neighbouring tokens into each token’s representation, so a bank row that has attended to river ends up at a different point in the embedding space from a bank row that has attended to account. Tokenization assigns the same ID to both, positional encoding only adds a position signal, and fine-tuning is not how a pretrained model resolves polysemy at inference.

Question 4: Dot Product as a Similarity Measure

What does the dot product \(\mathbf{a} \cdot \mathbf{b}\) between two vectors of the same dimension actually measure?

A) How aligned the two vectors are, scaled by their lengths
B) The angle between them, measured in radians
C) The product of their lengths, ignoring direction
D) The straight-line (Euclidean) distance between the two vector tips

Show Answer

Correct Answer: A

The dot product is \(\mathbf{a} \cdot \mathbf{b} = \lVert \mathbf{a} \rVert \lVert \mathbf{b} \rVert \cos(\theta)\): a single number that combines “do the two vectors agree on direction” with “how long are they”. It is large and positive when the two point in similar directions, zero when perpendicular, negative when opposed. This identity holds in every dimension. Cosine similarity strips the lengths out for an angle-only measure. The dot product is not a distance, not the angle itself, and not just \(\lVert \mathbf{a} \rVert \lVert \mathbf{b} \rVert\) (that ignores direction entirely — which is what “ignoring direction” would mean) — it is the scalar that attention uses to score “how relevant is key \(j\) to query \(i\)” in a single number per pair.

Question 5: What Each Sub-Layer Does

Which sub-layer of a transformer block is the only one that mixes information across token positions?

A) Both attention and the FFN mix across positions, in roughly equal measure
B) The FFN only — attention’s job is to refine each token’s representation in isolation
C) Neither — cross-token mixing happens through the residual connections
D) Attention only — the FFN is applied independently to each token, never combining different rows

Show Answer

Correct Answer: D

This is the load-bearing distinction of the lecture: attention gathers, FFN processes. Attention is the only operation in the whole transformer that lets one token read information from other tokens. The FFN is a per-position MLP — same weights applied row by row, never combining rows. Without attention the model would be 96 stacked MLPs operating on each token in parallel, with no token ever seeing another. Without the FFN, information would route around but never get reshaped. Residuals carry information through the depth, but they do not mix across positions.

Question 6: Q, K, V — Three Roles per Token

In a single attention head, what is the role distinction between a token’s Key (K) and its Value (V)?

A) K stores the token’s grammatical role; V stores its semantic content
B) K and V are computed identically and stored separately only as an engineering optimisation
C) K advertises what the token contains; V is the payload transmitted if the token is attended to
D) K and V differ only because keys are normalised before the dot product while values are not

Show Answer

Correct Answer: C

Q/K/V decouple “who listens to whom” from “what gets transmitted”. The query asks a question; keys advertise content for matching; values supply the actual signal that flows out of a selected token. A token can advertise relevance for one reason (its key) and carry useful information for another (its value) — that decoupling is what gives attention its expressive power. K and V are produced by different learned projections \(W_K\) and \(W_V\), so they are not computed identically and there is no extra normalisation step on K. Both are used at training and inference alike.

Question 7: Why Divide by √d

Why are the raw attention scores \(Q K^\top\) divided by \(\sqrt{d}\) before the softmax?

A) Without the scaling, the dot products grow with \(d\) and saturate softmax, killing gradients
B) The division turns the dot products into cosine similarities, bounding them in \([-1, 1]\)
C) It enforces that each row of the attention matrix sums to exactly 1 after the softmax
D) It speeds up the matrix multiplication by reducing the magnitude of the operands

Show Answer

Correct Answer: A

Each entry of \(Q K^\top\) is a sum of \(d\) componentwise products. With standard initialisation, each product has variance 1, so the sum has variance \(d\) and typical magnitude \(\sqrt d\). When that magnitude is fed into a softmax, the distribution collapses onto a single winner and the gradient to all other entries dies. Dividing by \(\sqrt d\) restores constant-magnitude inputs so softmax stays in its informative regime. The original Vaswani et al. (2017) paper called this out explicitly. It is not a normalisation into \([-1, 1]\) (that would be cosine similarity, which strips lengths entirely); the row-sum-to-one property comes from softmax itself, not from the division.

Question 8: Causal Masking

What is the role of the causal mask in a decoder-only LLM?

A) It speeds up training by allowing the loss to skip padding tokens at the end of short sequences
B) It restricts attention to the most recent \(k\) tokens, bounding the attention matrix’s size for long inputs
C) It prevents a token from attending to itself, eliminating self-loops in the attention pattern
D) It zeros out attention to all future positions, so each query only sees itself and earlier keys

Show Answer

Correct Answer: D

The mask \(M\) inside \(\operatorname{softmax}((QK^\top + M)/\sqrt d)\) sets every above-diagonal entry to \(-\infty\), which becomes 0 after the exponential. The result: query at position \(i\) can attend to keys at positions \(1, \ldots, i\) but not to \(i+1, \ldots, T\). This is what forces the model to predict the next token without peeking at it — without the mask, the parallel training loop would let position \(i\) read the ground-truth token at position \(i+1\) and the loss would collapse to zero. The mask is part of the architecture, not a windowing scheme — every position sees its full causal prefix, not just the last \(k\) tokens. A token does attend to itself.

Question 9: Multi-Head Parameter Budget

How does the parameter budget of multi-head attention compare to single-head attention at the same total dimension \(d\)?

A) Multi-head adds significantly more parameters because each head has its own full \(d \times d\) Q, K, V triple
B) Multi-head uses fewer parameters because each head sees only \(d/H\) dimensions
C) The Q/K/V budget is unchanged at \(3 d^2\); multi-head adds one extra projection \(W_O\) for \(4 d^2\) total
D) Multi-head doubles the FFN parameter count because the FFN must process \(H\) parallel heads

Show Answer

Correct Answer: C

\(H\) heads each have Q, K, V of shape \(d \times d_h\) with \(d_h = d/H\). Multiplying out: \(H \times 3 \times d \times d_h = 3 d^2\) — exactly the single-head budget, just rearranged into \(H\) parallel narrower subspaces. The genuinely new parameters in multi-head are the output projection \(W_O \in \mathbb{R}^{d \times d}\), which mixes the heads back together and contributes one extra \(d^2\). Total per attention sub-block: \(4 d^2\). The FFN parameter count is independent of head count.

Question 10: Why Position Has To Be Injected

Without any positional information added to the input, what would attention output for “the cat sat on the mat” versus “the mat sat on the cat”?

A) Identical outputs (up to row permutation), because attention sees its input as an unordered multiset of token vectors
B) Different outputs because the dot products \(\mathbf{q}_i \cdot \mathbf{k}_j\) depend on the order in which the tokens appear
C) Identical outputs only at the first layer; later layers can recover order through residual connections
D) Different outputs because the row-wise softmax is inherently sensitive to the order of its inputs

Show Answer

Correct Answer: A

This is the permutation-equivariance lemma: for any permutation \(P\), \(\operatorname{Attention}(PX) = P \cdot \operatorname{Attention}(X)\). Each output row depends only on which tokens are involved, never on where they sit. The set of pairwise scores is identical between the two sentences — same multiset of tokens, same set of pairs. Without a position signal, a transformer is a very expensive bag-of-words model. That is why \(E_{\text{pos}}\) (or RoPE) is added: the input \(X + E_{\text{pos}}\) is no longer permutation-equivariant, and the model can finally distinguish cat sat on mat from mat sat on cat. Residual connections preserve information but cannot recover what was never there to begin with.

Question 11: Where the Parameters Live

Within a single transformer block (attention + FFN), where do the majority of parameters live?

A) In the layer-norm scale and shift parameters \(\gamma, \beta\)
B) In the feed-forward network — its two matrices \(W_1\) and \(W_2\) hold roughly twice the attention parameters
C) In the embedding matrix, which is part of every block by weight-tying
D) In the attention projections \(W_Q, W_K, W_V, W_O\), since attention is the architecturally novel part

Show Answer

Correct Answer: B

Per block: attention is \(4d^2\), FFN is \(8d^2\) (\(W_1\) is \(d \times 4d\) and \(W_2\) is \(4d \times d\), each \(4d^2\)), layer norm is \(\sim 4d\) — negligible. So roughly two-thirds of every block’s weights sit in the FFN. Over the full stack of GPT-3, that is tens of billions of parameters dedicated to per-token, non-attention transformation. The interpretability framing is “attention is routing; FFN is storage” — most of what we call “the model’s knowledge” lives in those FFN weights, not in attention. The embedding matrix is shared with the output head and is not per-block.

Question 12: The Key Insight — T Is Not in the Parameters

The same trained transformer \(\theta\) can process a 5-token prompt and a 5,000-token prompt. Architecturally, why?

A) The sequence length \(T\) does not appear in any weight shape — every weight depends on \(V, d, N, H\), and at most \(n_{\max}\)
B) The model dynamically resizes its weight matrices at inference time based on the input length
C) Padding tokens are inserted to bring every input up to the fixed length the model was trained on
D) A separate copy of \(\theta\) is loaded for each input length and the orchestration layer transparently routes the input to the right copy

Show Answer

Correct Answer: A

This is the architectural punchline of Chapter 13. Walk through every weight: embedding \(V \times d\), attention \(W_Q/W_K/W_V/W_O\) of shape \(d \times \cdot\), FFN \(d \times 4d\) and \(4d \times d\), layer norm \(\mathbb{R}^d\). None of them depend on \(T\). \(T\) lives only in activation shapes — \(H_\ell \in \mathbb{R}^{T \times d}\), the score matrix \(S \in \mathbb{R}^{T \times T}\) — which are recomputed on every input. The model is shape-uniform in \(T\), which is what makes a transformer one model that generalises across input lengths instead of a different model per length. Weights do not resize; padding does not enable arbitrary length; there is only one \(\theta\).

Question 13: The Quadratic Cost of Attention

Doubling the input length \(T\) roughly quadruples the per-pass compute and memory for one transformer block. Which sub-layer is responsible for that quadratic-in-\(T\) scaling, and why?

A) The FFN — its two matrices \(W_1\) and \(W_2\) each touch every token, so doubling \(T\) doubles the cost of two matmuls and quadruples the total
B) The embedding lookup — looking up \(T\) rows from a \(V \times d\) matrix involves \(T \times V\) work, which is quadratic in \(T\)
C) Attention — the score matrix \(Q K^\top \in \mathbb{R}^{T \times T}\) has \(T^2\) entries; both computing it and using it to mix \(V\) cost \(O(T^2 d)\)
D) Layer normalisation — it computes mean and variance across all \(T\) positions for every token, giving \(T \times T\) total work

Show Answer

Correct Answer: C

Of every sub-layer in a transformer block, only attention scales quadratically with \(T\). The score matrix \(Q K^\top \in \mathbb{R}^{T \times T}\) dots every query against every key — \(T^2\) entries — and the subsequent mix \(A V\) is also \(O(T^2 d)\). Everything else is linear in \(T\): the FFN applies its weights row by row at \(O(T \cdot d^2)\) (doubling \(T\) merely doubles the cost — it does not quadruple it); the embedding lookup is \(O(T \cdot d)\) — \(V\) does not appear because each lookup just indexes one row; layer norm normalises within each row at \(O(T \cdot d)\). The lecture’s framing is that \(T \times T\) is “the only place in the entire forward pass where \(T\) appears squared in an activation” — and it is an activation, computed fresh on every input, not a stored parameter. This single fact is why every long-context optimisation — FlashAttention, sliding-window, sparse, and linear attention — targets the score matrix.

Question 14: The Auto-Regressive Loop

To generate a 500-token response from a transformer, how many full forward passes through the network are required?

A) One — the model produces all 500 tokens in a single pass over the prompt
B) 500 — each generated token requires a complete forward pass over the entire sequence so far
C) 501 — one initial pass to process the prompt, plus 500 more passes to generate each token
D) One per layer — so 96 forward passes for a GPT-3-class model

Show Answer

Correct Answer: B

The auto-regressive loop runs the full forward pass per generated token: read the last position’s logits, sample one token, append, then re-run the entire forward pass on the now-longer sequence. The very first pass — the one that consumes the prompt — is the pass that generates the first token, so there is no separate “prompt pass” on top of 500 generation passes. A 500-token response is 500 full forward passes; a 5,000-token response is 5,000. Generating long outputs is genuinely a linear multiplier on compute, and stacked layers run inside one pass — they are not extra passes.

Scoring Guide

Key Takeaways to Remember

  1. One token per forward pass — A 500-token response means 500 full forward passes, each over the entire sequence so far
  2. Embeddings turn labels into geometry — Token IDs are names; the embedding matrix turns each name into a row of \(d\) real numbers the model can do calculus on
  3. Embeddings are context-independent; attention is what disambiguates — The row for bank is the same vector everywhere; attention mixes neighbours into each row to resolve sense
  4. Dot product = aligned-ness — \(\mathbf{a} \cdot \mathbf{b} = \lVert \mathbf{a} \rVert \lVert \mathbf{b} \rVert \cos\theta\); this single operation underlies every “how relevant is X to Y” question in attention
  5. Attention gathers, FFN processes — Attention is the only sub-layer that mixes across positions; the FFN is per-token and never combines rows
  6. Q advertises a question, K advertises content, V is the payload — The decoupling of Key from Value is what gives attention its expressive power
  7. The √d division keeps softmax in its informative regime — Without it, gradients die and training stalls
  8. The causal mask is one term added inside softmax — That single line is the difference between a bidirectional encoder and a GPT-style decoder
  9. Multi-head is the same Q/K/V budget rearranged into parallel subspaces — Plus one extra projection \(W_O\) to mix heads back together
  10. Without positional encoding, attention is a bag-of-words operator — cat sat on mat and mat sat on cat would produce identical outputs
  11. The FFN holds two-thirds of each block’s parameters — Most of the model’s “knowledge” lives in FFN weights, not in attention
  12. \(T\) is not a dimension of any weight — Every weight is a function of \(V, d, N, H\); the same \(\theta\) handles any sequence length up to \(n_{\max}\)
  13. Attention is the quadratic-in-\(T\) sub-layer — The \(T \times T\) score matrix \(Q K^\top\) is the only place \(T\) appears squared; FFN, embeddings, and layer norm all scale linearly
© 2026 Dominik Müller   •  Powered by Soopr   •  Theme  Moonwalk