Home

06 Lecture 6 Quiz: Steering the Distribution

May 2026 (4425 Words, 25 Minutes)

quiz hallucination prompting post-training in-context-learning chain-of-thought

New: Interactive Quiz Available!

Want to track your progress and get instant feedback? Try our new interactive version with progress tracking, immediate explanations, and score calculation.

Start Interactive Quiz →

Instructions

This quiz tests your understanding of key concepts from Lecture 6
Each question has 4 options with exactly one correct answer
Try to answer without referring to the lecture materials
Focus on understanding concepts, not memorizing details

The quiz is split into the lecture’s two halves: Block A (why the model fails) in Questions 1–10, and Block B (how you deliberately steer it) in Questions 11–20.

Question 1: Next-Token Prediction

Pre-training optimises a transformer against exactly one objective. What is it?

A) Maximising helpfulness, scored by a reward model trained on human labellers’ rankings of candidate responses
B) Following the instructions contained in its system prompt
C) Matching its outputs to a curated set of ideal question-answer conversations
D) Predicting the next token, scored by cross-entropy loss against the actual next token

Show Answer

Correct Answer: D

Pre-training is one loss function — next-token cross-entropy — applied over trillions of tokens, and everything that looks like a separate capability (code generation, translation, question answering) is emergent from it. The most tempting wrong answer is B: instruction-following feels fundamental, but the lecture singles it out as the one behaviour that does not emerge from pre-training — it needs the separate post-training step. A describes the RLHF reward signal and C describes SFT — both belong to post-training, not pre-training.

Question 2: What the Model Does Not Have

Which of the following does a pre-trained language model NOT possess by direct construction?

A) The ability to emit a probability distribution over the vocabulary
B) A learned representation of the statistics of the text it saw in training
C) A built-in concept of truth that flags whether a statement is correct
D) Attention heads that route information between token positions, mixing context into each token’s representation

Show Answer

Correct Answer: C

The training corpus contains both correct and incorrect text with no flag marking which is which, so the model has no concept of truth — a wrong answer is just text in the same statistical universe as a right one. The most tempting wrong choice is A: emitting a probability distribution is the model’s one operational behaviour — the very thing it does have by construction — so a reader who skims past the capitalised NOT may pick it. A, B, and D are all things the model genuinely possesses.

Question 3: Base vs Post-Trained Models

Why have you almost certainly never interacted with a “base” model (the raw artefact of pre-training)?

A) Every consumer/developer product ships a model that has additionally been post-trained to behave like an assistant
B) Base models are too large to run on any consumer or cloud hardware
C) Base models are kept secret and never released by any lab
D) Base models still respond like assistants, just less reliably, so labs hide them to avoid confusing users

Show Answer

Correct Answer: A

Base models exist and labs such as Meta, Mistral, and DeepSeek release them alongside their post-trained versions — but the products you use (Claude Code, ChatGPT, Copilot) ship the post-trained model. The most tempting wrong answer is D: post-training does not merely “polish” an already-assistant-like base — a base model has no preference for being helpful and might answer “What is the capital of France?” with more capital-of questions. C is false (base versions are published), and B is a wrong reason.

Question 4: SFT vs RLHF

In the standard post-training recipe, what is the division of labour between SFT and RLHF?

A) SFT teaches what an assistant reply looks like; RLHF teaches which reply humans prefer
B) SFT injects new factual knowledge into the model; RLHF then aligns the model with the truth of its answers
C) SFT uses reinforcement learning; RLHF uses supervised cross-entropy
D) SFT removes the model’s ability to refuse; RLHF adds it back

Show Answer

Correct Answer: A

SFT (instruction tuning) trains on curated example conversations with the same next-token loss, so the model imitates the shape of assistant replies; RLHF then ranks candidate responses by human preference and optimises toward the preferred ones. Crucially, RLHF rewards form, not truth. B is the tempting trap and is doubly wrong: post-training is not the main vehicle for new facts (those come from pre-training), and RLHF aligns the model with preferred forms, not truth. C swaps the two methods, and D is false — refusal is learned in both stages.

Question 5: Why AI Hallucinates

Mechanistically, why does a model confidently emit a fabricated API endpoint when asked about one it has never seen?

A) A bug in the sampling code occasionally corrupts the output token
B) The model has no stored fact to emit and no reliable way to abstain, so it does what next-token prediction always does — it emits the most plausible-looking continuation, which here is a grammatically valid but invented endpoint
C) The model deliberately guesses because it was trained to never admit ignorance
D) The model retrieved the closest real endpoint it had memorised, and the small mismatch is just an approximate-lookup error

Show Answer

Correct Answer: B

The endpoint was never in training, so there is nothing to look up — and the pipeline cannot abstain. A fluent next-token predictor with no concept of truth fills the gap with the most plausible-looking continuation, here a grammatically valid but invented endpoint. Note it can be confidently wrong: the distribution may be sharply pointed at a plausible falsehood, not flat. The most tempting wrong answer is D, but a hallucinated endpoint is “not a retrieval of a known fact” — the endpoint was never in training, so there is nothing to look up. C is wrong because the model has no goals or intent (it does not “deliberately” do anything), and A is wrong because this is the normal mechanism, not a code bug.

Question 6: The Hallucination Taxonomy

A model produces a long, fluent, well-structured prose explanation of a requests.DEFAULT_POOL_SIZE attribute that does not exist, full of plausible implementation detail. Which category is this, and what is the recommended mitigation?

A) Factual hallucination — mitigated by retrieving the documentation
B) Structural hallucination — caught automatically by the runtime (ImportError / AttributeError)
C) Confident-nonsense hallucination — caught automatically by a linter
D) Confident-nonsense hallucination — mitigated by external verification (running it, asking a domain expert)

Show Answer

Correct Answer: D

A long prose explanation that reads like a senior engineer’s blog post but is invented end-to-end is confident-nonsense. Unlike factual hallucinations (catchable by checking a source) or structural ones (catchable by running the code), it requires domain expertise or external verification to detect. The most attractive distractor is C: it names the right category but the wrong mitigation — a linter parses code style and syntax and cannot catch a wrong prose claim. B is the wrong category, since the scenario is invented prose, not code the runtime could reject.

Question 7: CLAUDE.md Compliance

A 615-line CLAUDE.md states a project convention, yet mid-session the model ignores it. What is the best mechanistic explanation?

A) The orchestration layer failed to load the file into the context
B) The model recognised the convention as low-priority and its safety training overrode it for the rest of the session
C) The file is still in the context, but it now sits among far more recent, task-local tokens; whether the model keeps following the early rule depends on the model and the prompt, and it may attend less to it
D) CLAUDE.md instructions expire after a fixed token count and are deleted from the context

Show Answer

Correct Answer: C

The file is still in the context (the sequence only grows). For each head and query position the softmax weights sum to one, so an early instruction now has to compete with everything added since, and the model may attend less to it — but this is a tendency, not a fixed proportional law: a well-trained long-context model often still follows an early rule, and attention weight is not the same as causal influence on the output. CLAUDE.md influences the distribution; it does not command. The most seductive wrong answer is A — “ignored” intuitively reads as “never loaded” — but the file has not been overwritten; the sequence only grows. It was not a refusal (B: there is no safety conflict), and tokens are not deleted (D).

Question 8: Instruction-File Standards

Which statement about instruction files (AGENTS.md, CLAUDE.md) is correct as of the lecture’s April 2026 snapshot?

A) AGENTS.md is an Anthropic-authored standard that Claude Code reads natively, while OpenAI and Google tools use their own separate convention files
B) AGENTS.md is a multi-vendor, foundation-governed standard, but Claude Code does not read it natively — it reads CLAUDE.md
C) CLAUDE.md and AGENTS.md are processed by fundamentally different attention mechanisms inside the model
D) Because the standard is foundation-governed, the model now enforces AGENTS.md rules as configuration

Show Answer

Correct Answer: B

AGENTS.md began as an OpenAI initiative, was donated to the Agentic AI Foundation (a Linux Foundation fund), and is read natively by many tools — but Claude Code reads CLAUDE.md, not AGENTS.md (the workaround is a one-line @AGENTS.md indirection). The most attractive distractor is A, the clean inversion of the truth: AGENTS.md is not Anthropic-authored, and Claude Code does not read it natively. The same attention mechanism applies to every instruction file (C wrong), and none are enforced as configuration — they are loaded as context (D wrong).

Question 9: Lost in the Middle

Liu et al.’s “lost in the middle” result implies which practical rule for a CLAUDE.md file?

A) Bury the most important rule in the middle, where the model reads most carefully
B) Make the file as long as possible so important rules appear more than once by chance
C) Order does not matter, because attention is global, so the model weighs an early rule and a buried rule equally regardless of position
D) Put the single most important rule near the top (primacy), and optionally repeat a critical instruction at the end of the prompt (recency)

Show Answer

Correct Answer: D

Retrieval accuracy is U-shaped — high at the beginning (primacy) and end (recency), worst in the middle. So front-load the most important rule and repeat a critical instruction at the bottom for recency. A is the precise inverse of the result: the middle is the ~20-point accuracy trough, the worst place to bury a rule. C’s premise is right (attention can in principle reach any position) but the empirical U-shaped positional bias is real, so order matters; B is wrong because padding a file with repetition mostly adds distractors and pushes the important rule toward the weak middle — it does not reliably help.

Question 10: The Three-Layer Model

In the three-layer model (influence / enforcement / human), why can’t you reach production reliability with a great influence layer alone?

A) The influence layer is probabilistic: it biases the distribution but cannot guarantee or enforce any outcome
B) The influence layer is deterministic and therefore too rigid for real tasks
C) The enforcement layer is deterministic, so with strict enough linters, types, and CI it can fully replace the influence layer
D) Running tests and hitting real APIs is more reliable than prompting, so the influence layer is never the real bottleneck

Show Answer

Correct Answer: A

The influence layer (prompts, AGENTS.md, few-shot, CoT triggers) only biases the distribution — it is position-dependent, attention-dilutable, and RLHF-bounded. Enforcement (linters, types, tests, CI) is deterministic and catches hard violations; the human layer catches wrong-but-plausible output. You cannot substitute one layer with more of another. The most attractive distractor is C, but strict enforcement still cannot tell whether a function does the right thing — only whether it parses and passes types. B inverts the influence layer’s property, and D mistakes the enforcement layer for a reason the influence layer is sufficient.

Question 11: Temperature as Engineering

Which statement about temperature (T) is correct?

A) Raising T can flatten the distribution enough that a previously lower-logit token overtakes the argmax and becomes most probable
B) Setting T = 0 eliminates hallucination by forcing the model toward correct answers
C) T scales the logits before softmax; it reshapes how peaked the distribution is but cannot move the argmax or add missing knowledge
D) Temperature can enforce a structural constraint such as “the output must be valid JSON”

Show Answer

Correct Answer: C

Temperature divides the logits before softmax, sharpening (low T) or flattening (high T) the distribution. The argmax never moves — only the logits decide the top token (A wrong: flattening shrinks the gaps but preserves their order). The most tempting trap is B: a sharper distribution feels safer, but on a flat distribution T = 0 just commits harder to the modal-but-wrong token, so hallucination becomes more repeatable, not gone. Temperature only redistributes mass; it cannot enforce a grammar — that is constrained decoding (D wrong).

Question 12: Few-Shot in Practice

You want the model to write a new error-handling function that matches your codebase’s house style. According to the lecture, what is the most reliable approach?

A) Write a long instruction listing every style rule (naming, logging, exception type, line limit) in prose, since a complete verbal spec removes all ambiguity
B) Provide two real error-handling functions from your repo as few-shot examples, and ask for a similar handler
C) Raise the temperature so the model explores more stylistic variety
D) Point the model at your project’s written style guide and ask it to follow the documented conventions

Show Answer

Correct Answer: B

For project-specific conventions, examples beat instructions — show, don’t tell. Few-shot lets the induction heads route over genuine instances of the style, producing output far more consistent with the existing code. The most attractive distractor is A, the instruction-style prompt the lecture explicitly stages against few-shot: each rule becomes a separate piece of conditioning the model can misinterpret, so the output drifts. D fails for the same reason as A — a referenced style guide is documentation the model must interpret, not concrete examples for the induction heads to route over; C changes variety, not fidelity.

Question 13: Induction Heads

What routine does an induction head implement?

A) It computes an equal-weighted average of all previous token embeddings to form a fixed-length summary of the context
B) It stores the demonstration examples in a cache and looks them up by exact string match when an input reappears
C) It updates the model’s weights based on the examples in the prompt
D) Given the current token A, it finds the earlier occurrence of A and routes attention to whatever token followed it, predicting that the same continuation repeats

Show Answer

Correct Answer: D

Olsson et al. (2022) identified induction heads doing exactly this copy-and-continue routing — tightly correlated with in-context-learning ability and emerging at a visible phase transition during training. The most tempting wrong answer is C, but in-context learning involves no weight updates; the behaviour change comes purely from the longer context. A describes mean-pooling and B describes an explicit example cache/lookup — neither is what an attention head does.

Question 14: Format Beats Content

Min et al. (2022) replaced the correct labels in few-shot demonstrations with random labels. What happened, and what does it imply?

A) Accuracy was only slightly lower than with correct labels and far above zero-shot — demonstrations mainly convey format, label space, and task identity, not the answers
B) Accuracy collapsed to zero, proving demonstrations work by showing the model the answer
C) Accuracy improved, because random labels prevent the model from overfitting to the demonstrations
D) Accuracy was completely unchanged — since the induction heads route to the demonstration positions regardless, the label content carries no signal at all

Show Answer

Correct Answer: A

Most of the few-shot lift survived randomising the labels. Demonstrations primarily transfer the format, input distribution, label space, and task identity; the induction heads route to the demonstration positions regardless of whether the labels there are correct. B is the naive expectation Min et al. overturned. D is the over-correction: correct labels still do slightly better, so labels are not entirely irrelevant. C is a category error — there are no weight updates in in-context learning, so “overfitting” does not apply.

Question 15: Chain-of-Thought

Mechanistically, why does chain-of-thought help on multi-step reasoning tasks?

A) It triggers a special reasoning module that is otherwise dormant
B) The orchestration layer dynamically increases the number of transformer layers N for that query, so each pass runs deeper
C) Each intermediate token is itself a full N-layer forward pass that later passes attend to, trading depth for length
D) It widens the sampling distribution (higher temperature / top-p) so the correct token becomes more probable

Show Answer

Correct Answer: C

A single forward pass gives N rounds of computation; CoT emits intermediate tokens — each itself the output of N layers — and later passes attend back over them, stretching effective depth to ~N × L. There is no special module (A), and N itself never changes (B) — it is the model’s fixed depth, not something the orchestration layer can grow. D confuses CoT (a depth/sequence mechanism) with temperature/top-p, which only reshape sampling, not computation.

Question 16: The Faithfulness Caveat

How should you treat a model’s visible chain-of-thought when verifying a high-stakes answer?

A) As a faithful, step-by-step record of how the model computed its answer, since each visible reasoning token comes from the same forward pass as the final token
B) As a useful debugging signal, but not proof of how the answer was reached — rely on tests, schemas, and human review instead
C) As confirmation the answer is right whenever the conclusion matches the final visible reasoning step
D) As a guarantee the answer is correct whenever the individual reasoning steps each look valid

Show Answer

Correct Answer: B

A CoT trace catches some cascading errors and reveals the model’s framing, but the model can generate a clean chain that justifies the answer post-hoc rather than deriving it (Lyu et al., 2023) — a signal, not a guarantee. A is the most tempting wrong answer and doubly wrong: the visible chain is not a faithful record (it justifies post-hoc), and its premise is false too — each reasoning token is a separate forward pass, not the same one as the final token. C and D both mistake internal consistency or plausible-looking steps for correctness; a CoT chain can be a longer wrong answer.

Question 17: Reasoning Models

How should you prompt a modern reasoning model (o1, DeepSeek-R1, extended-thinking models) differently from a standard chat model?

A) Add the phrase “Let’s think step by step” and several reasoning demonstrations, since reasoning models need even more explicit chain-of-thought
B) Always set temperature to 0 so the model’s internal reasoning is fully deterministic, since deterministic decoding is what lets it reach the verifiably-correct answer
C) Reasoning models run a separate planning module before the transformer, so the prompt only configures that module, not the model
D) Give the goal, constraints, context, allowed tools, output format, and verification criteria, and let the model reason internally — explicit step-by-step prompting can degrade quality

Show Answer

Correct Answer: D

Reasoning models are trained (often with RL against verifiable-correctness signals, e.g. DeepSeek-R1’s GRPO) to reason at length internally. They are not architecturally different — same attention, FFN, layers, and auto-regressive loop. So you prompt for the goal, constraints, context, tools, output format, and verification criteria, not a visible chain; vendors warn that “Let’s think step by step” can degrade quality (A wrong). C is wrong — there is no separate planning module (no new mechanism). B is wrong — temperature is a sampling lever, not the relevant control for reasoning quality.

Question 18: Self-Consistency

Self-consistency samples N chain-of-thought responses at T > 0 and takes the majority answer. When does it FAIL to help?

A) When the model’s most common reasoning failure produces the same wrong answer — the mode is wrong, so voting reinforces the misconception
B) When the task has many possible answers, so the votes split and no clear majority ever forms
C) When temperature is above 0, because the samples must be identical to vote
D) On any task where the single correct answer is already more likely than any individual wrong answer

Show Answer

Correct Answer: A

Self-consistency is a variance-reduction / law-of-large-numbers argument: it works only when correct paths are more likely than any individual wrong path. If the model is systematically miseducated so that a wrong answer is the mode, the vote concentrates on it — a more confident wrong answer. C is backwards (you need T > 0 so samples differ), D describes exactly when it works, and B is wrong — vote-splitting across many answers narrows the margin but is not the documented failure; a wrong mode is.

Question 19: Constrained Decoding

Constrained decoding (structured outputs / JSON mode) masks the sampling distribution so only grammar-valid tokens can be emitted. What does it guarantee?

A) Both the shape and the factual correctness of the output, because a token that would produce a false value also violates the schema and gets masked
B) That the prompt no longer matters, since the grammar does all the work
C) The shape of the output (valid JSON, required keys, valid enum values) — but not the correctness of the content inside that shape
D) That the model will never hallucinate, because invalid tokens are masked

Show Answer

Correct Answer: C

At each step the engine zeros the probability of any token that would break the grammar (a compiled finite state machine / pushdown automaton), giving hard guarantees on shape — no missing keys, no malformed JSON. But a schema-valid object can still contain hallucinated values: the shape is enforced, the truth is not. A is the inverse trap for anyone who over-trusts structured outputs — enforcing the container says nothing about the values placed in it. You still need a good prompt (B wrong), and content hallucination is unaffected (D wrong).

Question 20: The Security Boundary

A malicious instruction hidden in a web page the agent reads makes it ignore the application’s instructions (prompt injection — OWASP LLM01). How does this differ from CLAUDE.md non-compliance, and where must the mitigation live?

A) Both are reliability problems fixed by writing a stronger, clearer prompt
B) Prompt injection is adversarial; the influence layer is effectively defeated, so safety must live in the enforcement layer (permissions, sandboxing, output validation) and the human layer (review of destructive actions)
C) Prompt injection is fully solved by lowering the model’s temperature to 0, which removes the randomness an attacker exploits
D) It is the same as CLAUDE.md drift — both are instructions losing influence — so the fix is to raise the system prompt’s priority and pin it at the top of the context

Show Answer

Correct Answer: B

CLAUDE.md drift is accidental; prompt injection is adversarial. The model is not “tricked” — it follows the most contextually plausible continuation, which the attacker has arranged to be their instruction. You cannot out-prompt an adversary: for adversarial inputs the influence layer is assumed defeated, so safety must live in the enforcement layer (tool permissions, sandboxes, validation, egress controls) and the human layer (review of anything touching money, secrets, or production). D is the seductive wrong answer — re-weighting or pinning prompts is still the influence layer, which is defeated. A makes the same mistake, and C confuses temperature (a sampling knob) with a security control.

Scoring Guide

18-20 correct: Excellent! You can both diagnose why the model fails and engineer the influence layer deliberately
14-17 correct: Good understanding! Review the areas you missed
10-13 correct: Fair understanding. Revisit the lecture materials for key concepts
Below 10: Please review the lecture carefully and reach out if you need clarification

Key Takeaways to Remember

Next-token prediction is the only training objective — code generation, translation, and instruction following are all emergent; nothing else was ever optimised directly
The model has no built-in concept of truth — correct and incorrect text live in the same statistical universe, which is why hallucination is structural, not a bug
Post-training makes the assistant — SFT shows what a good response looks like; RLHF ranks which form humans prefer. RLHF rewards form, not truth
Hallucination is structural in fluent next-token prediction — the model has no built-in concept of truth and is never allowed to abstain, so when it lacks the fact it still emits a plausible-looking continuation; it can be confidently wrong, not merely uncertain
Three hallucination categories, three mitigations — factual → retrieval, structural → grounding/runtime, confident-nonsense → external verification
CLAUDE.md influences, it does not command — it biases the distribution but generation still ends in a sample, and a rule competing with a long, recent context can be attended to less (a tendency, not a fixed proportional loss); for a guarantee you need the enforcement layer, not louder wording
AGENTS.md is the cross-tool standard, but Claude Code reads CLAUDE.md — same mechanism, different file name and placement
Lost in the middle — front-load the most important rule (primacy) and repeat critical instructions at the end (recency); the centre is the worst place to hide a rule
The three-layer model — influence (probabilistic) + enforcement (deterministic) + human (semantic); no layer can replace another
Temperature re-weights the dice — it cannot move the argmax, add facts, or enforce constraints; at T = 0 hallucinations just become more repeatable
Few-shot is induction heads — find [A][B], see [A], predict [B]; format transfers more than content (Min et al.)
For codebase style, examples beat instructions — show, don’t tell
Chain-of-thought trades depth for length — same N, L× the effective computation — but the visible chain is not a faithful audit trail
Reasoning models are not architecturally different — prompt for goal, constraints, and verification criteria, not “let’s think step by step”
Self-consistency is variance reduction — it fails when the wrong answer is the mode
Enforce shape, not truth; you cannot out-prompt an attacker — constrained decoding guarantees structure only, and prompt injection must be stopped in the enforcement and human layers