06 Lecture 6 Quiz: Steering the Distribution

Pre-training is one loss function — next-token cross-entropy — applied over trillions of tokens, and everything that looks like a separate capability (code generation, translation, question answering) is emergent from it. The most tempting wrong answer is B: instruction-following feels fundamental, but the lecture singles it out as the one behaviour that does not emerge from pre-training — it needs the separate post-training step. A describes the RLHF reward signal and C describes SFT — both belong to post-training, not pre-training.

The training corpus contains both correct and incorrect text with no flag marking which is which, so the model has no concept of truth — a wrong answer is just text in the same statistical universe as a right one. The most tempting wrong choice is A: emitting a probability distribution is the model’s one operational behaviour — the very thing it does have by construction — so a reader who skims past the capitalised NOT may pick it. A, B, and D are all things the model genuinely possesses.

Base models exist and labs such as Meta, Mistral, and DeepSeek release them alongside their post-trained versions — but the products you use (Claude Code, ChatGPT, Copilot) ship the post-trained model. The most tempting wrong answer is D: post-training does not merely “polish” an already-assistant-like base — a base model has no preference for being helpful and might answer “What is the capital of France?” with more capital-of questions. C is false (base versions are published), and B is a wrong reason.

SFT (instruction tuning) trains on curated example conversations with the same next-token loss, so the model imitates the shape of assistant replies; RLHF then ranks candidate responses by human preference and optimises toward the preferred ones. Crucially, RLHF rewards form, not truth. B is the tempting trap and is doubly wrong: post-training is not the main vehicle for new facts (those come from pre-training), and RLHF aligns the model with preferred forms, not truth. C swaps the two methods, and D is false — refusal is learned in both stages.

The endpoint was never in training, so there is nothing to look up — and the pipeline cannot abstain. A fluent next-token predictor with no concept of truth fills the gap with the most plausible-looking continuation, here a grammatically valid but invented endpoint. Note it can be confidently wrong: the distribution may be sharply pointed at a plausible falsehood, not flat. The most tempting wrong answer is D, but a hallucinated endpoint is “not a retrieval of a known fact” — the endpoint was never in training, so there is nothing to look up. C is wrong because the model has no goals or intent (it does not “deliberately” do anything), and A is wrong because this is the normal mechanism, not a code bug.

A long prose explanation that reads like a senior engineer’s blog post but is invented end-to-end is confident-nonsense. Unlike factual hallucinations (catchable by checking a source) or structural ones (catchable by running the code), it requires domain expertise or external verification to detect. The most attractive distractor is C: it names the right category but the wrong mitigation — a linter parses code style and syntax and cannot catch a wrong prose claim. B is the wrong category, since the scenario is invented prose, not code the runtime could reject.

The file is still in the context (the sequence only grows). For each head and query position the softmax weights sum to one, so an early instruction now has to compete with everything added since, and the model may attend less to it — but this is a tendency, not a fixed proportional law: a well-trained long-context model often still follows an early rule, and attention weight is not the same as causal influence on the output. CLAUDE.md influences the distribution; it does not command. The most seductive wrong answer is A — “ignored” intuitively reads as “never loaded” — but the file has not been overwritten; the sequence only grows. It was not a refusal (B: there is no safety conflict), and tokens are not deleted (D).

AGENTS.md began as an OpenAI initiative, was donated to the Agentic AI Foundation (a Linux Foundation fund), and is read natively by many tools — but Claude Code reads CLAUDE.md, not AGENTS.md (the workaround is a one-line @AGENTS.md indirection). The most attractive distractor is A, the clean inversion of the truth: AGENTS.md is not Anthropic-authored, and Claude Code does not read it natively. The same attention mechanism applies to every instruction file (C wrong), and none are enforced as configuration — they are loaded as context (D wrong).

Retrieval accuracy is U-shaped — high at the beginning (primacy) and end (recency), worst in the middle. So front-load the most important rule and repeat a critical instruction at the bottom for recency. A is the precise inverse of the result: the middle is the ~20-point accuracy trough, the worst place to bury a rule. C’s premise is right (attention can in principle reach any position) but the empirical U-shaped positional bias is real, so order matters; B is wrong because padding a file with repetition mostly adds distractors and pushes the important rule toward the weak middle — it does not reliably help.

The influence layer (prompts, AGENTS.md, few-shot, CoT triggers) only biases the distribution — it is position-dependent, attention-dilutable, and RLHF-bounded. Enforcement (linters, types, tests, CI) is deterministic and catches hard violations; the human layer catches wrong-but-plausible output. You cannot substitute one layer with more of another. The most attractive distractor is C, but strict enforcement still cannot tell whether a function does the right thing — only whether it parses and passes types. B inverts the influence layer’s property, and D mistakes the enforcement layer for a reason the influence layer is sufficient.

Temperature divides the logits before softmax, sharpening (low T) or flattening (high T) the distribution. The argmax never moves — only the logits decide the top token (A wrong: flattening shrinks the gaps but preserves their order). The most tempting trap is B: a sharper distribution feels safer, but on a flat distribution T = 0 just commits harder to the modal-but-wrong token, so hallucination becomes more repeatable, not gone. Temperature only redistributes mass; it cannot enforce a grammar — that is constrained decoding (D wrong).

For project-specific conventions, examples beat instructions — show, don’t tell. Few-shot lets the induction heads route over genuine instances of the style, producing output far more consistent with the existing code. The most attractive distractor is A, the instruction-style prompt the lecture explicitly stages against few-shot: each rule becomes a separate piece of conditioning the model can misinterpret, so the output drifts. D fails for the same reason as A — a referenced style guide is documentation the model must interpret, not concrete examples for the induction heads to route over; C changes variety, not fidelity.

Olsson et al. (2022) identified induction heads doing exactly this copy-and-continue routing — tightly correlated with in-context-learning ability and emerging at a visible phase transition during training. The most tempting wrong answer is C, but in-context learning involves no weight updates; the behaviour change comes purely from the longer context. A describes mean-pooling and B describes an explicit example cache/lookup — neither is what an attention head does.

Most of the few-shot lift survived randomising the labels. Demonstrations primarily transfer the format, input distribution, label space, and task identity; the induction heads route to the demonstration positions regardless of whether the labels there are correct. B is the naive expectation Min et al. overturned. D is the over-correction: correct labels still do slightly better, so labels are not entirely irrelevant. C is a category error — there are no weight updates in in-context learning, so “overfitting” does not apply.

A single forward pass gives N rounds of computation; CoT emits intermediate tokens — each itself the output of N layers — and later passes attend back over them, stretching effective depth to ~N × L. There is no special module (A), and N itself never changes (B) — it is the model’s fixed depth, not something the orchestration layer can grow. D confuses CoT (a depth/sequence mechanism) with temperature/top-p, which only reshape sampling, not computation.

A CoT trace catches some cascading errors and reveals the model’s framing, but the model can generate a clean chain that justifies the answer post-hoc rather than deriving it (Lyu et al., 2023) — a signal, not a guarantee. A is the most tempting wrong answer and doubly wrong: the visible chain is not a faithful record (it justifies post-hoc), and its premise is false too — each reasoning token is a separate forward pass, not the same one as the final token. C and D both mistake internal consistency or plausible-looking steps for correctness; a CoT chain can be a longer wrong answer.

Reasoning models are trained (often with RL against verifiable-correctness signals, e.g. DeepSeek-R1’s GRPO) to reason at length internally. They are not architecturally different — same attention, FFN, layers, and auto-regressive loop. So you prompt for the goal, constraints, context, tools, output format, and verification criteria, not a visible chain; vendors warn that “Let’s think step by step” can degrade quality (A wrong). C is wrong — there is no separate planning module (no new mechanism). B is wrong — temperature is a sampling lever, not the relevant control for reasoning quality.

Self-consistency is a variance-reduction / law-of-large-numbers argument: it works only when correct paths are more likely than any individual wrong path. If the model is systematically miseducated so that a wrong answer is the mode, the vote concentrates on it — a more confident wrong answer. C is backwards (you need T > 0 so samples differ), D describes exactly when it works, and B is wrong — vote-splitting across many answers narrows the margin but is not the documented failure; a wrong mode is.

At each step the engine zeros the probability of any token that would break the grammar (a compiled finite state machine / pushdown automaton), giving hard guarantees on shape — no missing keys, no malformed JSON. But a schema-valid object can still contain hallucinated values: the shape is enforced, the truth is not. A is the inverse trap for anyone who over-trusts structured outputs — enforcing the container says nothing about the values placed in it. You still need a good prompt (B wrong), and content hallucination is unaffected (D wrong).

CLAUDE.md drift is accidental; prompt injection is adversarial. The model is not “tricked” — it follows the most contextually plausible continuation, which the attacker has arranged to be their instruction. You cannot out-prompt an adversary: for adversarial inputs the influence layer is assumed defeated, so safety must live in the enforcement layer (tool permissions, sandboxes, validation, egress controls) and the human layer (review of anything touching money, secrets, or production). D is the seductive wrong answer — re-weighting or pinning prompts is still the influence layer, which is defeated. A makes the same mistake, and C confuses temperature (a sampling knob) with a security control.