Home

06 Steering the Distribution: Hallucination, Compliance, and Prompting

May 2026 (29687 Words, 165 Minutes)

ai llm hallucination prompting in-context-learning chain-of-thought rlhf act-2

1. Introduction

Welcome to Week 6. Last week we opened the box. You followed an embedding through positional encoding, multi-head attention, the feed-forward network, the residual stream, all \(N\) layers, the unembedding to vocabulary scores, the softmax-temperature-top-p sampling pipeline, and back into the input as the next forward pass began. Every arrow had a name. Every shape had a reason. By the end, the mechanism was no longer a black box.

Today we ask two questions that the mechanism alone cannot answer.

First: given everything we now understand about how this system works, why does it ever go wrong? Why does the same machinery that produces flawless validate_email for one prompt produce a confidently-wrong Stack Exchange API endpoint for the next? Why does CLAUDE.md sometimes feel like it might as well not exist? Why does a vague prompt drift into elegantly-formatted nonsense?

Second — and this is the half of the lecture that, in a course called AI-supported software development, ultimately matters most — once we understand the failure modes, what do we deliberately do to steer the system? When you write “Let’s think step by step”, what is actually happening inside the model? When you paste three examples of your codebase’s error-handling style before asking for a new function, what mechanism makes the model match them? When a paper claims a new prompting technique improves accuracy by 12 points, what is it manipulating, and what is the simplest explanation in last week’s vocabulary?

The structural pay-off, the one you should carry from the first sentence, is this: every prompting pattern that appears in this lecture — zero-shot, few-shot, chain-of-thought, self-consistency, role priming — is a deliberate, named use of the mechanism we built last week. There are no new ingredients. You already have everything. What you do not yet have is a way to talk about what you are doing when you write a prompt. By the end of this lecture you will.

The lecture is in three parts. Block A (chapters 2–8) is the diagnostic side: what the model actually is after pre-training and post-training (chapters 2–3), and how it fails. Hallucination as a structural property of next-token prediction, the CLAUDE.md compliance anecdote, the lost-in-the-middle effect, and the three-layer model that production systems use to wrap a probabilistic component in deterministic guarantees. We will explain every failure mode in last week’s vocabulary and refuse the temptation to call any of it a bug. Block B (chapters 9–14) is the engineering side. Temperature, in-context learning, induction heads, few-shot, chain-of-thought, self-consistency. Real theory, with citations. Chapter 15 then steps outside the influence layer entirely: constrained decoding, retrieval, validation, model-class choices, and the security boundary between accidental and adversarial failures. The pattern library closes in chapter 16. The mystery board closes for the second-to-last time in chapter 17. The lab and deliverables sit in chapter 18.

Before we start, one cultural note. The prompting literature is full of folklore: “always say please,” “tell the model it is an expert,” “promise it a tip and it will work harder.” Some of this is true under some conditions. Most of it is the kind of advice that sounds satisfying and decays the moment the underlying model changes. We will not be doing that. Every pattern in Block B has a one-line mechanistic justification grounded in Lecture 5, and a citation. If a pattern in this lecture survives the next model generation, it is because the mechanism it exploits survives. Folklore does not survive. Mechanism does.

Let us start where Lecture 5 ended: with the single thing the model was ever trained to do.

2. Next-Token Prediction Is the Only Thing the Model Was Trained To Do

Pre-training, in its entirety, fits in one sentence. Take a colossal corpus of text — code, books, web pages, transcripts, Wikipedia, Stack Overflow, GitHub, Reddit, transcripts of YouTube videos, scientific papers, everything anyone can scrape, license, or buy. For every sequence in that corpus and every position in that sequence, ask the model: given everything to the left of position \(t\), what is the next token at position \(t\)? Compare the model’s predicted distribution over the vocabulary to the actual next token. Backpropagate the cross-entropy loss. Repeat trillions of times.

That is the entire training objective. One loss function. Trillions of tokens. Billions of parameters. Tens of thousands of GPUs. Months of wall-clock time. The result, at the end of all that compute, is a model with one operational behavior: given a sequence, emit a probability distribution over the next token. Nothing else. The next chapter will show you why “nothing else” is not a complaint but a structural claim.

Everything that looks like a different capability is emergent.

A useful way to internalise this: the model does not know it is writing code. It does not know it is answering a question. It is completing a sequence that statistically looks like the right continuation. Calling that “intelligence” is a vocabulary choice, not a description of mechanism.

What the model does not have, by direct construction:

The implication. The model is a sophisticated pattern completion engine. It is spectacularly good at patterns it has seen many times in training. It fails — predictably, in ways we can now describe — wherever patterns are rare, absent, or misleading. Everything in Block A is a consequence of this single fact. Everything in Block B is a way to give the model the patterns it needs.

Hold that frame: the model is a probability distribution over the next token, computed by the machinery you saw last week, conditioned on whatever tokens are currently in its context. Every “why does it do X?” question in the rest of this lecture has the same shape of answer: because that completion was the most plausible continuation under the distribution it just computed. Block A asks why the distribution is sometimes wrong. Block B asks how to make the distribution more often right.

Before we get to either, one chapter of background that turns out to matter for both. Chapter 2 described pre-training and the base model it produces. The model you actually interact with has been through a second training phase — post-training — that you have never seen the absence of. The next chapter explains what that phase does, because every failure mode in the rest of Block A, and every prompting pattern in Block B, is a behaviour of a post-trained model.

3. Post-Training: From Base Model to Assistant

You have never used a base model. The model you talk to in Claude Code, ChatGPT, or Copilot is not the model that comes out of pre-training. Between pre-training and the assistant you interact with, there is a second, smaller, much more carefully managed training phase called post-training. It is what turns a sophisticated pattern completion engine into something that behaves like an assistant.

Two reasons this chapter lives in Block A, not later in the course. First, every failure mode we are about to describe — hallucination, the compliance anecdote, lost-in-the-middle — looks the way it does because it is happening inside a post-trained model. The base model fails too, and it hallucinates too — fabrication is a property of next-token prediction itself, not of the post-training stack on top of it. What changes with post-training is the shape of the failure: pattern-continuation gives way to the confident-assistant style that gives modern hallucinations their characteristic gloss. (This is not a hypothetical claim — open-weights labs like Meta, Mistral, and DeepSeek publish base versions alongside their post-trained ones, so the difference is observable directly.) Calling out post-training before we diagnose failures is the difference between accurate diagnosis and folklore. Second, every prompting pattern in Block B relies on post-trained behaviour. Few-shot, chain-of-thought, role priming — all of them exploit instruction-following preferences the base model does not have. Naming the source of those preferences here means we can refer to it without parentheticals later.

This chapter explains what post-training is, what it achieves, and what it does not. It does not explain how it works at the mathematical level — the algorithms (PPO, DPO, RLAIF) belong in a machine-learning course. For our purposes the operational summary is what matters.

3.1 The base model, and why you have never met one

The base model is the artefact at the end of pre-training, exactly as chapter 2 described it. A model whose only learned behaviour is “produce a probability distribution over the next token that reflects the statistics of the training corpus.” Its weights are optimised against next-token cross-entropy and nothing else.

Base models are real artefacts; the major labs typically release a base version alongside the post-trained version. They are also genuinely strange to interact with, and the reason is exactly what chapter 2 said: the base model does not “know” it is talking to a user, does not know it is being asked a question, does not prefer helpful responses over unhelpful ones. It completes patterns.

A concrete example. Give a base GPT-style model the prompt "What is the capital of France?" Two of the most likely continuations, in raw next-token probability, are:

  1. "Paris." — because Wikipedia, textbooks, and educational sites contain many sentences of exactly that form.
  2. "What is the capital of Germany? What is the capital of Italy?" — because lists of capital-of questions also appear in the training corpus, often without answers, in places like quiz preparation sheets or trivia game outlines.

The base model has no preference between these continuations. Both have non-trivial probability. Sometimes you get an answer; sometimes you get more questions; sometimes you get a wandering essay about French geography. The base model is not deciding between “helpful” and “unhelpful” — it is sampling from a distribution that has both, because both exist in its training data.

You have never seen this behaviour. The Claude or GPT-4 or Llama model you have used through any consumer or developer product has gone through a second training phase — post-training — that shifted its preferences toward responding like an assistant. Every behaviour we are about to diagnose — instruction following, the confident style, refusal patterns, the [system]-versus-[user] hierarchy — is a behaviour of a post-trained model. None of it was present in the base.

3.2 Supervised Fine-Tuning (SFT)

The first stage of post-training is Supervised Fine-Tuning (SFT), sometimes called instruction tuning. The recipe is procedurally simple:

  1. Hire humans — typically large teams of contractors with relevant expertise — to write thousands or tens of thousands of example conversations. Each example shows a user prompt and an ideal assistant response.
  2. Take the base model. Train it on this curated corpus, using the same next-token cross-entropy loss as pre-training. No new loss function; only a new, much smaller, much more carefully constructed dataset.
  3. The model that comes out is one that has seen many examples of how a well-behaved assistant responds to user prompts. By the pattern-completion mechanism from Lecture 5, the model now imitates those patterns when given similar prompts.

SFT alone is a lot. After SFT, the model:

Many open-source assistant models stop at SFT. Vicuna, Alpaca, the original LLaMA Instruct, and many smaller domain-specific assistants are SFT-only. They are recognisably helpful, recognisably instruction-following, and visibly different from a base model. The further refinement of RLHF (next section) is a quality bump on top of SFT, not the only path to instruction following.

The mechanism, mechanically, is unchanged. The architecture is identical to pre-training: same attention, same FFN, same \(N\) layers, same forward pass. What changed is the distribution of contexts the model has been optimised on. After SFT, the prompt "What is the capital of France?" has a much sharper peak on "Paris"-style continuations and a much smaller mass on more-questions-style continuations. The base model’s pluralism has been replaced by an assistant’s directness — just by training on examples that demonstrate directness.

Where to read more. Instruction-tuning predecessors include Wei et al. (2021), Finetuned Language Models Are Zero-Shot Learners (the FLAN paper) and Sanh et al. (2022), Multitask Prompted Training Enables Zero-Shot Task Generalization (the T0 paper). Both showed that even basic SFT on diverse instruction-formatted data dramatically improved zero-shot task generalisation. The canonical reference for SFT as part of the modern assistant recipe is Ouyang et al. (2022), Training Language Models to Follow Instructions with Human Feedback — the InstructGPT paper, which is the recipe ChatGPT was built on. We will reference it again in the next section because the same paper introduces the RLHF step.

3.3 Reinforcement Learning from Human Feedback (RLHF)

The second stage of post-training is Reinforcement Learning from Human Feedback (RLHF). It is a refinement on top of SFT, not a replacement for it. SFT teaches the model what an assistant looks like. RLHF teaches the model which of several plausible assistant-style responses humans prefer.

The recipe has two sub-stages.

Stage A: train a reward model. Take the SFT model. For a given prompt, have it produce multiple candidate responses (typically 2–4) — easy to do by sampling at non-zero temperature. Then have humans rank the candidates from best to worst. Repeat this for many thousands of prompts. Train a separate, smaller model — the reward model — to predict the human ranking from the response text. The reward model takes a (prompt, response) pair and outputs a scalar indicating how much a human would prefer that response. By the end of this stage you have a function that can score any response with a numerical estimate of human preference.

Stage B: optimise against the reward model. Take the SFT model. Have it generate responses to prompts. Score those responses with the reward model. Use a reinforcement learning algorithm — Proximal Policy Optimisation (PPO) in the original recipe; Direct Preference Optimisation (DPO) and others in modern variants — to update the model’s weights so that future responses score higher under the reward model.

The mathematical details (PPO’s clipped surrogate objective, the KL penalty that keeps the model close to the SFT starting point, the reward-hacking failure modes that researchers worry about) are not what an engineering audience needs. The operational summary is: after RLHF, the model’s behaviour shifts decisively toward the kind of response humans prefer. Helpful, well-formatted, confident, refusing-when-appropriate, hedging-when-uncertain. The shifts can be subtle (a slightly more authoritative tone) or dramatic (refusing categories of requests the SFT model was happy to fulfil).

Where to read more. The underlying idea — using a learned reward model from human preference rankings — predates large language models entirely: Christiano et al. (2017), Deep Reinforcement Learning from Human Preferences, applied it to RL agents in classic Atari and robotics tasks. The first application to language modelling was Stiennon et al. (2020), Learning to Summarize from Human Feedback (OpenAI), which used RLHF to fine-tune a model for summarisation. The full SFT+RLHF recipe as we now know it was codified in Ouyang et al. (2022) — the InstructGPT paper, again — and is the most-cited reference for “what was done to turn GPT-3 into ChatGPT.” Anthropic’s parallel work is Bai et al. (2022), Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. The Constitutional AI extension, where AI judges replace human raters using a written set of principles, is Bai et al. (2022), Constitutional AI: Harmlessness from AI Feedback. The DPO simplification — collapsing the reward model and RL stages into one — is Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

Modern variants of post-training compress or change these stages. DPO collapses the reward-model and RL stages into a single optimisation. RLAIF replaces human rankings with AI rankings — an existing model judges what humans would prefer, at much greater scale than human labellers allow. Constitutional AI (Anthropic’s approach) extends RLAIF with a written set of principles the AI judge uses to score responses. From an engineering standpoint, all of these end at the same operational place — a model whose distribution favours assistant-shaped, human-preferred responses — by different paths.

The frontier has moved. The canonical recipe assumes the reward signal comes from human preferences over response forms. For reasoning-heavy tasks, recent work has shown that the reward signal can be replaced by something stronger: automatic verification of the answer. DeepSeek-R1 (Guo et al., 2025), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, demonstrated that pure reinforcement learning against a correctness signal — does the model’s final answer match the known-correct answer? — can elicit complex reasoning behaviour, and that the SFT stage can be skipped entirely (their “R1-Zero” variant). They also introduced a new RL algorithm, GRPO (Group Relative Policy Optimization), which replaced PPO in this setting. We will return to DeepSeek-R1 and the broader story of reasoning models in chapter 13.7, because R1 is more directly relevant to chain-of-thought than to the general post-training story. The point to register here: post-training is no longer a single canonical recipe but a growing family of techniques, and the frontier is moving fast.

3.4 What post-training buys you, and what it does not

For an engineering audience, four operational consequences are the ones to remember.

Instruction following. Before post-training, the base model continues patterns. After SFT + RLHF, the model’s distribution, when conditioned on an instruction, peaks much more sharply on tokens that obey the instruction. This is why asking the model to “write a Python function that validates an email address” produces a Python function rather than five more questions. It is also the entire mechanism that the prompting patterns in Block B will exploit.

Helpfulness has a style. Confident tone. Bullet points. Structured responses. “Here is a solution:” framings. Caveats and trade-offs. Code blocks for code. Brief refusals with explanation. This is the style human rankers preferred and the reward model learned to score highly. The style is durable across topics — the model uses the same shape of response whether you ask about web servers or world capitals.

The role hierarchy. Training conversations distinguish turns by special tokens: [system] (instructions from the application developer), [user] (messages from the end user), and [assistant] (the model’s own past statements). Human raters consistently preferred responses that followed [system] instructions even when the [user] asked for something contrary. By the end of post-training, the model has internalised this hierarchy as a learned bias: [system] instructions are weighted more heavily than [user] instructions, and both more heavily than the assistant’s own past statements. None of this is enforced architecturally — it is a learned preference in the weights, durable but not absolute. This fact is the hinge of the CLAUDE.md anecdote in chapter 6.

Refusal of harmful or off-topic requests. SFT examples and human rankings both consistently rewarded responses that decline harmful or out-of-scope queries. The model learns to refuse, in the same statistical way it learns to answer. Refusal is a learned pattern, not a hardcoded rule, which is why “jailbreaks” exist — prompts that confuse the role hierarchy or invoke unusual patterns can sometimes get the model to override its refusal training.

And for honesty’s sake, four things post-training does not buy you.

3.5 What this means for the rest of the lecture

Three points to carry forward.

  1. Every Block A failure mode is a failure mode of a post-trained model. When chapter 4 describes hallucination, the confident style is a post-training contribution. When chapter 6 describes the CLAUDE.md anecdote, the [system]-versus-[user] hierarchy is a post-training fact. When chapter 7 describes lost-in-the-middle, the model’s tendency to answer at all under any prompt is a post-training pattern. The base model would do something else, usually weirder.
  2. Every Block B prompting pattern exploits post-training. Few-shot demonstrations work partly because RLHF taught the model to take demonstrations seriously. Chain-of-thought triggers work partly because reasoning-style responses were rewarded by rankers. Role priming works partly because the model learned during RLHF that personas shape responses. The induction heads we will name in chapter 11 are a pre-training phenomenon; the willingness to follow few-shot patterns is a post-training phenomenon. Both have to be present for the patterns to work.
  3. The mechanism is the same; the distributions are different. Same attention, same FFN, same forward pass, same auto-regressive loop — everything you built in Lecture 5 is intact. What post-training changed is which conditional distributions the mechanism emits. Whenever this lecture says “the model is biased toward X” or “the model tends to do Y,” the precise statement is: post-training reshaped the conditional distribution so that, given a prompt that triggers the right pattern, tokens that produce X are now more probable than they would be from the base model alone.

This is the bridge to the rest of Block A. The remaining chapters describe what fails about that bias and that distribution.

4. Why AI Hallucinates: Confident Wrong Answers Without a Truth Check

The first thing to settle is the word. In the literature, hallucination is used in two slightly different senses: a factual error (the model states something false) and a fabricated structure (the model invents a function, an endpoint, a citation, a syntax that does not exist). For our purposes the distinction does not matter — both reduce to the same mechanism. We will use hallucination for any output that is fluently generated and confidently presented but materially wrong.

The central claim of this chapter. Hallucination is not a bug; it is a structural property of fluent next-token prediction. The model is trained to continue patterns, and nothing in the objective checks the continuation against the truth. So when the prompt activates a pattern whose faithful continuation happens to be false — whether the distribution is flat (no clear winner) or sharply peaked on a plausible-but-wrong token — the model produces it just as fluently as it produces a correct answer. Insufficient or misleading context is the common trigger; the missing truth check is the root cause. The mechanism that produces a perfect validate_email and the mechanism that produces a non-existent stackexchange.api.get_monthly_questions() are the same mechanism. The difference is the shape of the conditional distribution.

4.1 The shape of the distribution

Recall the picture from Lecture 5 §14. After the full forward pass on a context of \(T\) tokens, the model produces a vector of logits \(\boldsymbol{\ell} \in \mathbb{R}^V\), one real number per vocabulary entry. Softmax turns it into a probability distribution

\[p_i = \frac{e^{\ell_i}}{\sum_{j=1}^{V} e^{\ell_j}}\]

over the next token. We sample from that distribution (chapter 14 of Lecture 5) and the chosen token is appended to the sequence. The whole pipeline runs again. That is generation.

The crucial property is that the shape of the distribution varies with the context.

When the context provides strong, specific, unambiguous signal — when the prompt is “the capital of France is” and the training corpus contains many instances of exactly that pattern followed by Paris — the distribution is peaked. One token captures most of the probability mass. The other 50,000 entries in the vocabulary share a small remainder. Sampling that distribution at any reasonable temperature picks Paris almost surely.

When the context provides weak, vague, or absent signal — when the prompt is “the API endpoint for Stack Exchange monthly question counts is” and the training corpus contains no exact match for that specific endpoint — the distribution can go flat. Many tokens have similar, modest probability. Sampling picks one of them. The one it picks is grammatically and syntactically appropriate (the model has seen a million sentences of the form “the API endpoint for X is /api/v2/...”), but the specific URL it generates is a guess at the pattern, not a retrieval of a known fact.

Flat is one shape an under-specified prompt can produce — many near-equal options, no clear winner. It is not the only one. The model can also be confidently wrong: a sharp distribution pointed at a token that is plausible-given-the-pattern but false (this is the confident-nonsense case of §5.3). The unifying claim is not “the distribution is always flat when the model errs” — it is that next-token prediction has no truth check, so when the prompt activates the wrong pattern the model commits to it, flat distribution or sharp.

A useful image. A peaked distribution is a knife edge: one tall spike, near-zero everywhere else. A flat distribution is a low plateau: many tokens at roughly the same height, no clear winner. The same model, the same parameters, the same forward pass — only the prompt’s specificity is different.

P(next | "the capital of France is …") Peaked
One token (Paris) captures almost all the mass.
P(next | "the Stack Exchange API endpoint is …") Flat
Many tokens share the mass — no clear winner.
Same softmax. Same forward pass. Only the prompt's specificity decides the distribution's shape.

4.2 The model always picks a token

This is the part that does the real damage.

There is no “I do not know” token in the vocabulary. Or rather, there is — the model can in principle emit the tokens that spell I do not know — but those tokens have to win the softmax competition, and they only win when the prompt context primes them strongly enough. In a flat distribution where many continuations all have modest probability, “I do not know” is one option among many, and not generally the most likely one.

When the distribution is flat and the context is ambiguous, the model still emits a token. It has to: the forward pass produces logits, softmax produces probabilities, sampling returns a token. The pipeline is not allowed to abstain. The token it produces is whichever the sampling happened to pick — and the resulting continuation is a pattern-completion artefact — the model continued what the context looked like, not a fact it retrieved. (When the distribution is flat, the specific pick is also somewhat arbitrary; when it is sharp on a wrong token, the model is confidently wrong. Either way nothing checked it against the truth.)

That artefact is what we call a hallucination. A confidently-stated continuation that is grammatically valid, syntactically plausible, contextually appropriate — and, with respect to ground truth, an invention.

4.3 The first mystery, mechanistically resolved

In Lecture 1’s Monday Morning Story, Alex’s code looked plausible and contained a subtle bug. In Lecture 2 you saw that vague prompts produced hallucinated outputs that crisper prompts avoided. In Lecture 4 you learned about token budgets (the finite context window), and in Lecture 5 you learned how relevant context competes with irrelevant context for the model’s attention — what we call “attention dilution” as a course-level shorthand. We can now state the answer cleanly.

Why do vague prompts hallucinate? Less context leaves task dimensions unspecified, so the model fills the gaps from learned priors rather than from the facts of your situation — and it can do so confidently, committing to a wrong continuation, not just hesitantly. Sometimes that shows up as a genuinely flat distribution with many near-equal options; just as often the distribution is sharp but pointed at the wrong token, because the prior the prompt activated was the wrong one. Either way, the longer the generation, the more such gap-filling picks compound: a short answer might survive one or two, a paragraph rarely survives ten.

Why do more structured prompts work? More context shifts the distribution toward “peaked.” Each additional relevant token in the prompt activates attention heads that pull on related representations in the embedding space, narrowing the set of plausible continuations. The mechanism is the one you built in chapter 7 of Lecture 5: attention reads from context, and more useful context narrows the conditional distribution.

This is the mechanistic reason Approach 3 from Lecture 2 outperformed Approach 1. It is also the reason curated context (Lecture 5 mystery 3) outperforms raw context. The distribution does not care that you copied half a repository into the prompt; it cares about which tokens, in what positions, are relevant to the next token decision. Irrelevant context adds distractors the model can be pulled toward without improving the conditioning, so it rarely sharpens the next-token decision and can muddy it. Curated context concentrates the relevant signal, so the tokens the model attends to are the ones that actually narrow the answer. (As a course-level intuition we sometimes say irrelevant tokens “compete for attention”; keep in mind attention is normalised separately in every head and at every position, so this is a useful abstraction, not a single global budget being drained.)

A forward link to Block B. Temperature is the deliberate version of this same phenomenon — a knob you turn at sampling time to make the distribution more or less peaked. Block B opens with temperature as engineering: now that you know what the distribution looks like and why it sometimes goes flat, what knob do you turn, and when?

5. The Hallucination Taxonomy

Not all hallucinations are alike. They share a mechanism, but the way they manifest — and the way you should engineer against them — depends on what kind of structure the model is fabricating. Three categories are worth naming. They are not exhaustive, but they are the three you will see most often in code-generation tasks, and the three your Act 1 work has already produced.

5.1 Factual hallucinations

The model states a specific fact with confidence, and the fact is wrong.

“The Stack Exchange API endpoint for monthly question counts is /api/v2/questions/monthly. It accepts a from_date and to_date parameter and returns a count grouped by month.”

The endpoint does not exist. The Stack Exchange API exposes /questions with filtering, not a separate monthly summary. The model has hallucinated a URL, a parameter list, and a return shape. All of it sounds professional. None of it is real.

Mechanism. Training data contains many sentences of the form “the API endpoint for X is /api/v2/...“. The model has learned the shape of an API documentation paragraph. When you ask for an endpoint it has not specifically seen, the local distribution favours any plausible-looking URL. It picks one. The pattern continuation is good. The factual grounding is absent because grounding was never part of the training objective.

How to recognise it. Factual hallucinations cluster around proper nouns, specific numbers, version strings, API surfaces, library functions, and external entities. They sound like the model is recalling a specific piece of information. They are usually catchable by checking against an authoritative source: read the actual docs, run pip show, hit the URL.

5.2 Structural hallucinations

The model produces code that has the shape of correct code, with method names, type signatures, imports, and call chains — but the methods, signatures, or import paths do not exist in the library.

import locust
from locust.stats import get_monthly_report

stats = get_monthly_report(start_date="2026-01-01", end_date="2026-03-31")
print(stats.percentile(99))

This compiles in nobody’s head and breaks instantly on import. locust.stats.get_monthly_report does not exist. The Locust statistics API exposes RequestStats and StatsEntry and a few aggregation helpers, but no monthly-report function. The model has fabricated a function that should exist by the logic of the pattern — it knows that statistics libraries often have report-generating functions, it knows that monthly aggregation is a common need — and emitted code that exercises the fabricated function.

Mechanism. This is the most insidious category. The shape of the code is correct because the model has seen many statistics libraries and many monthly-report functions. The attention heads that route over locust, stats, monthly, report collectively pull on representations that say “yes, a function with this signature is plausible.” The peak of the distribution lands on a function that is plausibly-shaped but not present in this specific library. Structural hallucination is the cost of pattern completion not being constrained to the actual library surface.

How to recognise it. Structural hallucinations look like normal code. They use real imports, real-sounding identifiers, plausible argument lists, and reasonable return shapes. They are caught by the runtime (AttributeError, ImportError, TypeError) or by static analysis if the library has type stubs. They are not caught by reading the code — the code reads as if it should work.

5.3 Confident-nonsense hallucinations

The model emits a long, grammatically perfect, contextually appropriate explanation that is, end to end, wrong.

“The requests library uses an internal connection pool of size 10 by default. To increase it, you set requests.DEFAULT_POOL_SIZE = N before making your first call. This is thread-safe because the pool uses a threading.RLock internally. Be aware that the pool is per-process, not per-thread, so subprocesses inherit the parent’s pool size.”

There is no DEFAULT_POOL_SIZE attribute. The thread-safety claim is half right and half wrong. The “per-process not per-thread” sentence is grammatically lovely and semantically meaningless. The whole paragraph reads as if it came from a senior engineer’s blog post. It is invented.

Mechanism. Confident-nonsense hallucinations are exactly the failure mode chapter 3 warned about: post-training rewarded confident, well-structured explanations, so when you ask a question whose specific answer the model does not have, it still produces the form of an authoritative answer — often as a fairly sharp distribution over fluent, plausible-shaped tokens. The confidence is real; the grounding is absent. RLHF teaches style — sound authoritative, structure the answer, provide examples, mention caveats — and it does not teach truth, so the well-formed answer can be wholly invented.

This is the category that does the most reputational damage. Factual hallucinations are catchable by checking sources. Structural hallucinations are catchable by running the code. Confident-nonsense hallucinations require domain expertise to detect, because they look exactly like good answers to people who do not know the domain.

How to recognise it. The smell test: an answer is suspiciously well-organised given the obscurity of the question. The model “knows” things in too much detail. The paragraph cites specific implementation details without naming a source. The structure is the structure of a great answer; the content has not been independently verified.

5.4 What the three categories share

All three reduce to the same mechanism. All three are predictable from chapter 4: with no grounded answer to give, the model continued the pattern of a good answer — sometimes from a flat distribution with no clear winner, sometimes from a sharp distribution confidently pointed at a plausible falsehood — its shape matching what training data calls a good answer while its content was fabricated, and nothing in the pipeline checked it against the truth. Telling the categories apart helps you decide how to engineer against each — different categories respond to different mitigations.

Mystery board update. This chapter resolves three Act-1 mysteries simultaneously: invented API endpoints (factual), hallucinated function signatures (structural), and confidently-wrong explanations (confident nonsense). All three are pattern-completion artefacts of an objective with no truth check: the model continued what the context looked like — sometimes from a flat distribution with no clear winner, sometimes from a sharp distribution confidently pointed at a plausible falsehood — and nothing in the pipeline compared the result against ground truth. The full mystery board update comes in chapter 17.

The next chapter takes the same diagnostic apparatus and applies it to a different failure: not “the model said something wrong” but “the model did not follow the instructions I gave it.” That is the CLAUDE.md compliance question, and it has a mechanistic answer too.

6. Instruction Files in Practice: The CLAUDE.md Compliance Anecdote

The setup is one you have seen before in Lecture 3. A Claude Code session, a 615-line CLAUDE.md loaded from the project root, two explicit project conventions stated in that file. The first: run ruff formatting on any code before committing. The second: create a diary entry under diary/NNN-*.md for every meaningful AI interaction, in the format specified by the file. The model, in that session, generated a benchmark script with PEP 8 violations. When asked to fix the violations, it fixed the code and did not create a diary entry.

Neither failure was a refusal. The model was not declining to follow the conventions. It simply did not, even though the conventions were sitting in plain text in the prompt. If you have done any AI-assisted work this is a familiar texture: the system you instructed extensively just did not do the thing, with no warning and no apology, and the next prompt fixed it without effort.

After Lecture 5 you have the vocabulary to explain why. Before we get to the mechanism, a short framing note (§6.1), because CLAUDE.md is not a Claude-specific phenomenon; it is one instance of a broader pattern that every AI coding tool now follows. Then a three-part mechanistic explanation (§§6.2–6.4).

6.1 Instruction files: AGENTS.md as the emerging standard

Before we explain why CLAUDE.md compliance fails, we should be honest about what kind of object CLAUDE.md is. It is one instance of a broader pattern, and the pattern has, in the last year, acquired a cross-vendor name.

The pattern. Every modern AI coding tool reads at least one instruction file from the project root before answering. The file documents project conventions, code-style rules, testing commands, architectural notes — things you want the AI assistant to know without repeating them in every prompt. Across vendors the convention has converged on a single file name: AGENTS.md.

Governance. AGENTS.md started as an OpenAI initiative in August 2025. In December 2025 it was donated to the Agentic AI Foundation — a directed fund of the Linux Foundation, with OpenAI, Anthropic, Google, AWS, Bloomberg, and Cloudflare among the contributing members. The same foundation received MCP (Lecture 7’s orchestration story) from Anthropic at the same time. AGENTS.md is no longer a single vendor’s preference; it is a multi-vendor, foundation-governed spec. Over 60,000 open-source GitHub repositories already include an AGENTS.md.

Tool support, April 2026 snapshot. This snapshot will date; the pattern will not.

There is a current wrinkle worth naming directly: Anthropic’s Claude Code does not natively read AGENTS.md. If you do your hands-on coursework in Claude Code and put your project rules only in AGENTS.md, Claude will not load them. The community workaround — a one-line indirection in CLAUDE.md pointing at AGENTS.md — works because Claude Code follows @-referenced files transitively. This is a portability detail worth getting right early.

Why this lecture uses CLAUDE.md as the running example. The mechanism we are about to explain — attention dilution, multi-head competition, the [system]-versus-[user] hierarchy from §3.4, and the lost-in-the-middle effect coming up in chapter 7 — is identical across every instruction-file convention. AGENTS.md, CLAUDE.md, .github/copilot-instructions.md, .cursorrules, .clinerules are all the same kind of object: a chunk of project-root text that the tool’s orchestration layer (Lecture 7) reads from disk and injects into the model’s context as tokens. You will use Claude Code in the lab exercises, so CLAUDE.md is the concrete artefact you can touch. Everywhere this chapter says “CLAUDE.md,” you should mentally substitute “your tool’s instruction file” — the analysis carries over unchanged. The tool-specific differences live one layer up, in which file the orchestration layer reads and where in the token sequence it injects the content. That is Lecture 7’s territory.

For specifications and adoption details, the canonical reference is the AGENTS.md spec; OpenAI’s Codex AGENTS.md guide is the cleanest worked example of a vendor adopting the standard.

With the framing in place, we return to the mechanism. The explanation has three parts.

6.2 CLAUDE.md tokens compete for attention with task tokens

CLAUDE.md is a file on disk. The orchestration layer (Lecture 7) reads it and injects its contents as tokens near the start of the context — typically in positions a few hundred to a few thousand from the beginning, depending on how long the system prompt is and where exactly the loader chooses to splice. For our purposes the position is early, but not first, and certainly not unique.

Once the conversation starts, more tokens accumulate. The user’s first prompt. The model’s first response. A tool call to read a file. The file’s contents. Another response. Another tool call. Within a few minutes of interactive work, the conversation is at 5,000–20,000 tokens; a serious coding session that opens several files, runs tests, and iterates over diffs comfortably reaches 50,000–150,000 tokens, well within current frontier context windows — which by 2026 routinely sit at 400K and beyond, with 1M-token windows (across multiple frontier models, both proprietary and open-source) increasingly the new standard rather than the exception. CLAUDE.md is still in there — it has not been overwritten, the sequence only grows — but it now sits at position ~500 of a 30,000-token sequence, surrounded by far more recent, far more task-specific content.

1,000 tokens
turn 1 — opening
CLAUDE.md (500)
task (500)
CLAUDE.md = 50%
30,000 tokens
mid-session — a few files open
task (29,500)
CLAUDE.md = 1.67%
150,000 tokens
extended session — full repo loaded
task (149,500)
CLAUDE.md = 0.33%
Same 500-token CLAUDE.md across all three rows — same file, same instructions, same disk content. Only the surrounding task tokens have grown. The shrinking indigo sliver shows how small a fraction of the window CLAUDE.md now occupies — but be careful: that fraction is not the share of attention it receives. Softmax weighting is winner-take-most, so a rule the model has learned to treat as important can still draw strong attention from a tiny slice of the context (this is exactly why models retrieve a single relevant sentence out of hundreds of thousands of tokens). The honest reading of the figure is positional and competitive, not proportional: CLAUDE.md is now one early region competing with far more, far more recent, far more locally-relevant task tokens for the heads that would route to it — how much attention it keeps depends on how strongly the model has learned to weight rules like it, not on what percentage of the window it fills.

Now apply the mechanism from chapter 7 of Lecture 5. Attention at every generation step is row-wise softmax over the full sequence: for each query position, the weights over all preceding positions sum to 1. Crucially this is not a uniform split — softmax is exponential in the scores, so a position the model has learned to score highly can keep most of the weight even when thousands of other positions are present (this is what makes long-context retrieval possible at all). What growing the sequence does is add competitors: more recent, more locally-relevant task tokens that the heads near the generation point tend to score highly. The CLAUDE.md instruction “create a diary entry for every interaction” is still reachable — the attention weight to that position is non-zero, the head that would route over it still exists — but it does not lose attention because the window grew; it loses attention when louder, closer, more task-relevant tokens — the benchmark script you are currently writing — out-score it for the heads that would otherwise route to it.

This is the “context drift” tendency we named in Lecture 5 §16.1. It is not a failure of the file or of the loader; it is the predictable direction of how attention behaves as a long, busy context accumulates — early, general instructions tend to compete less effectively against recent, specific task tokens. How strong the effect is depends on the model, the training, and how the rule is written; it is a reliable tendency to design around, not a fixed proportional law.

6.3 Multi-head dilution: not every head listens to CLAUDE.md

The story has a second layer. From Lecture 5 chapter 9, attention is multi-head. A modern model has dozens of heads per layer (Claude-class models run roughly 64–128 heads in some layers), and each head specialises during training for a different pattern of routing. Some heads route over syntactic dependencies. Some over coreference. Some over the start of the sequence. Some, by happy accident, over earlier instruction-like tokens. Some over none of the above.

We do not need to know exactly which heads do what inside a production model to make the engineering point. The mechanistic intuition is enough: the model’s final representation at each generation step is a mixture of many heads’ outputs, combined by \(W_O\). Heads that happen to route over CLAUDE.md tokens contribute their values to that mixture; heads that do not route there contribute other values. As the conversation grows, the heads that anchor on early-position instruction tokens compete with heads that anchor on local task tokens. The CLAUDE.md instruction is mathematically present in the output mixture but with a variable effective weight, and that weight tends to shrink as the local context grows louder.

Multi-head attention dilution Eight attention heads at a single layer. Three route to CLAUDE.md tokens (indigo arrows); five route to local task tokens (blue arrows). The output is a mixture of all eight heads' values. CLAUDE.md task tokens (more recent, more local) H1 H2 H3 H4 H5 H6 H7 H8 output representation = Wₒ mixture of all 8 head values 3 heads contributed CLAUDE.md content · 5 contributed task content
Each of the model's attention heads independently routes to wherever its query/key match is strongest. Only some of them happen to route to the CLAUDE.md tokens at any given layer — 3 of 8 in this caricature; the real ratio depends on the head, the layer, and the local context. The output representation is the WO-weighted combination of all head values, so CLAUDE.md's effective influence is whatever weight that subset of heads carries in the mix.

Treat this as a qualitative mechanism rather than a precise prediction. We cannot say which heads route to which CLAUDE.md rule in a model whose internals are not public. What we can say is the operational fact, and it is the one Anthropic’s own Claude Code documentation on memory makes plain: CLAUDE.md is loaded as context, not enforced as configuration. Concise, specific, recent instructions are followed more reliably than long, vague, distant ones. The Claude Code best-practices documentation reinforces the same point in operational language: the context window fills quickly, and performance degrades as it fills. The attention story explains why — but you do not need the explanation to act on the engineering rule.

The actionable framing: CLAUDE.md is context, and context competes with other context. Long, stale, or vague instructions are less reliable than short, recent, specific ones combined with external enforcement (hooks, tests, CI). A rule that aligns with how the model already wants to behave — say, “use snake_case for function names,” which echoes a strong prior from training data — sticks more reliably than a rule that is project-specific and idiosyncratic — say, “create a diary entry for every interaction in this specific format” — even though both are syntactically present in the file.

6.4 The model is influenced, not commanded

Both effects compound. Context drift reduces the positional weight of CLAUDE.md instructions as the sequence grows. Multi-head dilution reduces the per-head weight of any individual instruction. By the time the model is choosing its next token in the middle of a complex coding task, the CLAUDE.md instruction “create a diary entry” is a faint signal competing against a loud one.

A useful way to internalise this: CLAUDE.md does not command the model. It influences the distribution. When CLAUDE.md is short, recent, and aligned with the local task, the influence is strong — the relevant tokens contribute meaningfully to the attention pattern, and the heads that route over them shift the distribution toward compliance. When CLAUDE.md is long, distant, and orthogonal to the local task, the influence is weak. The model is doing what its mechanism makes likely: in a busy context, the heads near the generation point tend to weight the most recent, most task-relevant tokens most heavily, so an early, general rule has to compete hard to stay in the mix. It can still be attended to — that is why repeating a critical rule near the query helps (§7.3) — but by default it is easy to out-shout.

This is also the mechanistic explanation for why different AI coding tools — Claude Code, Copilot, Cursor — get different mileage from the same model and the same instruction file. The orchestration layer (Lecture 7) decides where in the token sequence CLAUDE.md lands, how often it is repeated, whether it is in [system] or [user], and what additional context surrounds it. Same instruction, different placement, different effective influence.

6.5 Why large instruction files don’t break the bank: KV and prompt caching

A natural objection at this point. If AGENTS.md, CLAUDE.md, and the surrounding repository context routinely add up to tens of thousands of tokens, and those tokens are present in every turn of a conversation, the per-turn compute and latency must be devastating. They would be — without caching.

Modern inference relies on two related but distinct mechanisms, and it is worth keeping them apart. KV caching is intrinsic to how an engine generates a single response. Recall from Lecture 5 chapter 7 that attention projects each token into a key \(\mathbf{k}_t\) and a value \(\mathbf{v}_t\) at every layer; once those vectors have been computed for a token, they do not need to be recomputed within the same response, and each subsequent generation step extends the cache by one column rather than re-projecting the entire sequence. This is per-request, automatic, and nothing the application developer opts into.

Prompt caching (also called context caching) extends the same idea across requests. For a stable prefix — system instructions, tool definitions, repo files, AGENTS.md, CLAUDE.md, accumulated chat history through the previous turn — the serving stack may persist the prefix’s \(\mathbf{k}, \mathbf{v}\) state after a response finishes and reuse it on subsequent turns, rather than fully prefilling the prefix again. The catch: the prefix must be byte-identical, still cache-resident (vendors evict on idle or capacity pressure), and in some implementations explicitly marked for caching. The new turn’s tokens still run a fresh forward pass and still attend over the cached \(\mathbf{k}, \mathbf{v}\); the prefix tokens themselves are not re-embedded, re-projected, or re-FFN’d. Across vendors, Anthropic prompt caching, OpenAI prompt caching, and Google context caching all implement the same cross-request idea with slightly different APIs and pricing. For the rest of this section, when we talk about the economic properties of long instruction files, we are talking about prompt caching specifically — KV caching is in the picture too, but it is not the lever that makes a 30K-token AGENTS.md cheap to keep in context turn after turn.

Turn 2 without cache
~30K × forward-pass cost
recompute k,v for every prefix token
AGENTS.md @ pos 500
Every prefix token gets a full forward pass, every turn.
Turn 2 with cache
~100 × forward-pass cost
cached k,v (prefix reused)
Prefix k,v is read from cache; only the new turn's tokens run a fresh forward pass.
Attention weight on AGENTS.md @ pos 500
identical in both rows
same softmax over the same key sequence
The dot lines up vertically across all three rows. The model's view of AGENTS.md hasn't moved.
Prompt caching changes the cost of running the prefix, not the attention the prefix receives. The dot is at the same position in all three rows — cheaper to serve, just as easy to miss.

Two consequences matter for the rest of the lecture.

First, prompt caching changes the economics of large instruction files, not the influence story. A 50K-token AGENTS.md is cheap to keep in context on every turn if the engine caches it. But every word from §6.2 and §6.3 still applies: those tokens still occupy positions in the sequence, still sit among the keys every generation step attends over, and still compete with newer, more local tokens for the heads’ routing. Prompt caching does not change which tokens win that competition; it only changes the cost of having them in the context. (We keep calling this competition “attention dilution” as a course-level shorthand — just remember the mechanism is per-head, per-position softmax competition, not a single shared budget being spent down.) The instruction sitting at position 500 of a cached 30K-token prefix is just as easy to miss as one sitting at position 500 of an uncached one. The reason to keep AGENTS.md and CLAUDE.md concise is unchanged.

Second, prompt caching is one piece of the orchestration layer, not the model. The model does not know which tokens are cached. The orchestration layer (Lecture 7) decides what to cache, when to invalidate, and how to lay out the prefix so the cache hit rate stays high. Putting the volatile parts of the conversation (the user’s latest message, fresh tool output) after the stable parts (system prompt, AGENTS.md, accumulated chat history) is what makes caching practical in a long-running session. Reorganising your prompt to slot a fresh detail in the middle of a previously stable block evicts the cache for everything after it. Lecture 7 will return to this when it discusses skill loading and tool-result placement.

Mystery board update. This chapter resolves Act 1 mystery 7 (why is CLAUDE.md inconsistent?) and finishes resolving mystery 2 (why does CLAUDE.md improve results in some sessions and not others?). Mystery 8 (why do different tools behave differently from the same instructions?) is partly resolved here and finishes resolving in Lecture 7.

The next chapter takes one more step into the positional story and looks at a related empirical phenomenon: lost in the middle, the U-shaped attention curve over long contexts.

7. Lost in the Middle: The Positional Story

CLAUDE.md compliance has a position component. Whether the model follows an instruction depends, partly, on where that instruction lives in the token sequence. The literature has a specific name for the empirical curve that describes this phenomenon: lost in the middle.

7.1 The U-shape

The headline result comes from Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts. The experimental setup is conceptually simple. Take a question whose answer is in some long document. Vary the position of the relevant passage within the document — beginning, middle, end. Measure the model’s accuracy as a function of where the passage sits.

Lost in the Middle — U-shaped retrieval accuracy Retrieval accuracy is high at the beginning of a document (primacy advantage), drops sharply in the middle, and rises again near the end (recency advantage). 100% 50% 0% Beginning Middle End Position of the relevant passage Retrieval accuracy primacy advantage recency advantage ~20 pp drop
Liu et al. (2023): retrieval accuracy is high at the start (primacy) and end (recency), and drops sharply in the middle — a U-shape rather than the flat line naive attention theory would predict.

The accuracy curve is U-shaped. The model performs best when the relevant information sits near the beginning of the context (primacy advantage) or near the end (recency advantage), and worst when it sits in the middle. The dip in the middle can be 20 percentage points or more on retrieval-style tasks, and it gets sharper as the context grows. A passage that is 80 % retrievable when it sits at position 500 of 5,000 might be only 55 % retrievable when it sits at position 2,500 of 5,000.

This is striking because it is not what naive attention theory would predict. Attention is global — every token can attend to every other token. There should be no positional preference in principle. The positional preference is an empirical fact of trained models, and it has multiple contributing causes — positional encoding inductive biases, the distribution of training data (the most informative content tends to cluster at the start and end of documents), and likely some interaction with the residual stream’s accumulated-information geometry.

7.2 The Chroma “Context Rot” finding

A more recent result, Chroma Research’s Context Rot report (Hong, Troynikov & Huber, 2025), calls the same phenomenon context rot: as the input grows, the model’s effective accuracy on any specific piece of buried information decays. They evaluate 18 current frontier models — GPT-4.1, Claude 4, Gemini 2.5, Qwen3, and others — and find the effect is universal but its shape varies by model. Three compounding effects: the U-shape from Liu et al., growing competition as a long sequence adds more positions (and more recent, locally-relevant ones) for the heads to weight — what we have been loosely calling “attention dilution” — and distractor interference, where semantically similar but irrelevant content actively misleads the model. Note these are partly overlapping rather than three clean independent dials. The honest synthesis: positional effects on long-context retrieval are real, they are model-dependent, they are hard to predict exactly, and a well-trained long-context model loses much less to them than a naive length argument would suggest — but they do generally grow with context length.

7.3 What this means for instruction files

A practical reading.

Lecture 7 will show how this informs the orchestration layer’s design choices: when to inject CLAUDE.md, whether to repeat it, how to fold it into the chat history. For now: position matters, instructions decay, the centre is the worst place to hide a rule.

The next chapter is the last in Block A. We have spent six chapters describing what fails about the model and why. Chapter 8 is the engineering framework that wraps the probabilistic core in deterministic guarantees — the three-layer model that production systems use to live with the failure modes we have just diagnosed.

8. The Three-Layer Model for Production Compliance

Influence layer
Probabilistic — biases the distribution
AGENTS.md prompts few-shot CoT triggers
Catches
soft signal drift
Enforcement layer
Deterministic — refuses bad output
linters type checkers tests CI schemas
Catches
hard violations
Human layer
Semantic — judgment and review
code review architecture domain expertise
Catches
wrong-but-plausible
Three parallel safety nets. Each layer catches what the other two miss — remove any one and the gaps line up.

We close Block A with a framework. The framework is small — three layers — and the value is not in the layers themselves but in how they fit together. You will use this framework for the rest of the course, and you will use it for the rest of your career working with probabilistic components.

The motivation comes from everything in Block A. The model is influenced, not commanded. Instructions decay with position. Hallucinations are predictable but not eliminable. RLHF buys you instruction-following at a probability, not a guarantee. None of these problems can be fixed inside the model. They can only be wrapped, supplemented, and caught from outside.

The three layers are:

  1. Influence layer. What you write to make the right behaviour likely. Project instruction files — AGENTS.md as the cross-tool standard (§6.1), with tool-specific cousins like CLAUDE.md and .github/copilot-instructions.md — together with the prompts you craft, the few-shot examples you supply, the system prompts your application defines, and the chain-of-thought triggers you include. Everything that biases the distribution the model emits, without enforcing any particular outcome.
  2. Enforcement layer. What catches the model when the influence layer was not enough. Linters, type checkers, pre-commit hooks, CI pipelines, schema validators, unit tests, integration tests. Everything that is deterministic — ruff check either passes or fails, pytest either green or red — and that can refuse to merge, deploy, or accept output that violates a hard constraint.
  3. Human layer. What catches everything the first two layers miss. Code review, architectural judgement, design decisions, should-we-even-do-this judgement. Everything that requires semantic understanding, taste, or domain expertise that no automated check can encode.

8.1 Why all three matter

The interesting claim is not that any one layer is good or bad. The claim is that you cannot replace one layer with more of another. Each layer covers a different category of failure, and the gaps line up exactly with the gaps in the others.

Production-grade AI-assisted development uses all three. CLAUDE.md (influence) teaches the model your conventions. CI (enforcement) refuses to merge code that violates them. Code review (human) catches architectural mistakes that neither convention nor automation will. Remove any one layer and the system breaks in a predictable, named way.

8.2 How the influence layer becomes engineering

Block A’s diagnostic claim was that the influence layer is probabilistic. Block B’s engineering claim is that the influence layer is probabilistically engineerable. Temperature, prompting patterns, few-shot examples, chain-of-thought triggers — these are not magic. They are mechanisms for shaping the distribution the model emits, and we will study each of them with the same rigour we used for attention.

The right way to read Block A: here is why the influence layer alone is not enough. The right way to read Block B: here is how to engineer the influence layer so it carries as much load as it can. The enforcement and human layers are still required. But a well-engineered influence layer means the enforcement layer catches less, the human reviews fewer mistakes, and the overall system runs faster and with less friction.

8.3 The teaching-repo asymmetry

A note for the rest of the course. The repository you are working in for the Locust exercises is, deliberately, thin on the enforcement layer. There is a ruff configuration but no pre-commit hook by default. Type checks exist but are not gated on CI. The diary format is documented but not enforced. This is a pedagogical choice, not a professional one. The cost of a missing enforcement layer is visible when it is missing — you feel exactly which failures the layer would have caught, because they reach you.

In a professional project the enforcement layer is dense. Every commit runs hooks, every PR runs CI, every PR comment is its own enforcement of code-review standards. The three-layer model still applies; the layers are just much more developed. Lecture 7 will revisit this picture once we know what the orchestration layer adds on top.

The next chapter is the transition into Block B. We turn from why the mechanism fails to how we deliberately steer it.

9. Transition: From Consequences to Controls

Block A was the negative space. Block B is the positive space.

You now have, in mechanistic terms, the full diagnosis. The model is a probability distribution conditioned on context. When the conditioning context is strong, the distribution peaks on a correct answer. When the conditioning context is weak — vague, short, position-decayed, off-distribution from training — the distribution flattens, the model samples a plausibly-shaped token, and the generation drifts into invention. RLHF shifted the model’s preferences toward helpfulness, instruction-following, and a confident style, but did not give the model a separate truth-checking faculty. The three-layer model wraps the probabilistic core in deterministic guarantees where they are required.

The question Block B asks, given all of that, is: what can you actually do at the prompt-construction stage to make the distribution sharper, more peaked, more aligned with correct answers?

The answer is a small library of named patterns. Zero-shot. Few-shot. Chain-of-thought. Self-consistency. Role priming. None of them are new ideas; all of them have been studied empirically for years, and several have specific peer-reviewed papers that we will cite by name. None of them are folklore; all of them have a one-line mechanistic explanation in Lecture 5’s vocabulary, and the rest of Block B’s job is to give you both the citation and the mechanism for each.

We open with the simplest one. Temperature was introduced in Lecture 5 §14.2 as a knob you turn at sampling time. Chapter 10 promotes it from “knob defined” to “engineering decision,” with rules for when to use which value and an honest discussion of what temperature does not solve.

10. Temperature, Revisited as Engineering

You met temperature in Lecture 5 §14.2 as a scalar that divides the logits before softmax:

\[p_i = \frac{e^{\ell_i / T}}{\sum_{j} e^{\ell_j / T}}\]

The mechanism is straightforward. \(T \to 0\) sharpens the distribution toward a single winner; \(T = 1\) leaves it as it came out of the model; \(T > 1\) flattens it, giving the long tail meaningful probability mass. The chapter that introduced it ended with a one-line forward link to Lecture 6: temperature is the primary knob that controls the hallucination-vs-reliability tradeoff. Now we cash that in.

T = 0.1 — binary correctness Sharp
Argmax dominates; effectively deterministic.
T = 1.0 — graded correctness Default
Top tokens dominate, but the seconds occasionally win.
T = 2.0 — variation desired Flat
Sampling explores the long tail; correctness traded for diversity.
Temperature is the dial that turns the same logits into different distributions. Low T concentrates mass on the argmax; high T flattens the mass across the vocabulary. The model and its parameters are unchanged — only how its softmax is shaped at sampling time.

10.1 The decision tree

Three regimes, three uses.

The cleanest way to internalise this: temperature is the knob you turn to match the sharpness of the model’s output to the sharpness of the task’s correctness criterion. Binary-correct tasks want sharp distributions. Graded-correct tasks want moderate sharpness. Diverse-output tasks want flat distributions. There is no universal “right temperature” — there is only “right temperature for this task.”

10.2 A worked example: same logits, three temperatures

The decision tree in §10.1 names the regimes; let us watch the numbers move so the regimes feel less abstract. Recall from Lecture 5 §14.2 that temperature cannot move the argmax — it only changes how concentrated the distribution is around it. The table below makes that concrete.

Suppose at some step of generating Lecture 5’s running example, the model produces logits over a small 10-token vocabulary. Running softmax-with-temperature at three settings gives:

Token Logit \(\ell\) \(p\) at \(T = 0.1\) \(p\) at \(T = 1.0\) \(p\) at \(T = 2.0\)
<BOS>-1.2\(\approx 0\)0.0030.024
wizard0.5\(\approx 0\)0.0160.056
mixes-0.3\(\approx 0\)0.0070.037
glowing1.8\(\approx 0\)0.0580.107
potion4.2\(\approx 1.000\)0.6370.354
Dragon0.1\(\approx 0\)0.0110.046
Drinks-0.8\(\approx 0\)0.0040.029
fire3.1\(1.7 \times 10^{-5}\)0.2120.204
troll0.9\(\approx 0\)0.0240.068
smashes1.1\(\approx 0\)0.0290.075

Three things are worth pointing out.

  1. The argmax never moves. potion is the most probable token at every temperature, because its logit was the largest to begin with. \(T\) cannot promote fire (or anything else) to the top of the distribution — only the model’s logits can do that.

  2. At \(T = 0.1\) the model is functionally deterministic. potion carries essentially all the mass; fire gets a probability of about \(1.7 \times 10^{-5}\); every other token rounds to zero. Sampling from this distribution is, for practical purposes, the same as taking the argmax. This is the “binary correctness” regime — the model commits.

  3. At \(T = 2.0\) the runner-ups become real candidates. potion drops to 35 %; fire to 20 %; glowing, smashes, troll, wizard each carry a few percent. The distribution is flat enough that across 100 sampled completions you would expect roughly 35 potion, 20 fire, 11 glowing, 8 smashes, 7 troll — and the occasional <BOS> or Drinks. This is where the “variation desired” regime gets its variation from.

The mechanism, in one sentence: temperature re-weights the dice; it does not change which face has the most dots. The engineering decision in §10.1 is, in effect, picking which column of this table you want to be sampling from.

10.3 The first-token cascade

A subtlety from Lecture 5 §14.5 that is worth its own attention. Greedy decoding is deterministic, but not necessarily correct. The reason is the auto-regressive loop.

At \(T = 0\) the model picks the argmax token at every step. In the ideal decoding algorithm — the textbook version of greedy sampling on a fixed model — this removes sampling randomness, and the same prompt produces the same output. This feels like correctness; it is actually only consistency. The token the model commits to at step 1 conditions every subsequent token. If the step-1 argmax was wrong — if the genuinely correct answer was the second most likely token, and the second-most-likely token was a confident wrong answer — then every subsequent step is conditioned on a wrong first token. The cascade compounds. A 500-token response with one early wrong token can read as fluently incorrect from start to finish.

A second caveat is worth naming because students will hit it the first time they try to reproduce a result. Exact bit-for-bit reproducibility from a hosted API is not guaranteed even at \(T = 0\) — and the same caveats apply at any \(T\), with or without a seed. Real-world inference stacks involve numerical kernels that may run in different orders on different GPUs, batching strategies that change adjacent computations, periodic model-version updates that the API does not announce, and routing layers that can send your request to different replicas. OpenAI’s own advanced-usage documentation describes their API as “mostly” deterministic with seed and system_fingerprint, and is explicit that completion is not strictly reproducible. The teaching version of \(T = 0\) is deterministic; the production version is mostly deterministic, and the difference matters whenever you are debugging a regression in a long-running pipeline. The seed parameter fixes the sampling step but not the forward-pass step, so the same drift sources apply across all temperatures — sampling at \(T = 0.7\) with seed=42 is just as susceptible to backend-side variation as greedy decoding is.

A practical consequence: \(T = 0\) does not give you the most-probable sequence — it gives you the per-step argmax. Greedy decoding picks the highest-probability token at step 1, then conditions on that pick for step 2, and so on. The mode of the joint distribution over sequences would require search across all paths (intractable at vocabulary size ~50K); greedy is a fast approximation that is vulnerable to local maxima. A first-step token at probability 0.4 whose continuations are tightly peaked on a coherent answer can lose the step-1 race to a different token at probability 0.6 whose continuations are flat — and the sequence-level mode never gets a chance to win.

The classical NLP alternative is beam search, which keeps the top-\(k\) partial sequences alive at every step and so escapes the most obvious greedy traps. Modern LLM inference rarely uses it, for three reasons: it is computationally expensive at frontier vocabulary sizes; it tends to produce repetitive, generic text (a well-documented empirical failure known as the beam search curse); and chain-of-thought (chapter 13) addresses the same cascade problem more cheaply by giving the model intermediate tokens to recover on.

If the greedy commit is a confident wrong token, you lose — repeatably, with no warning. This is part of why some engineers run code generation at \(T = 0.3\) instead of \(T = 0\): a small amount of stochasticity lets you, across a few samples, occasionally escape a wrong cascade.

The first-token cascade is also one mechanistic reason chain-of-thought (chapter 13) helps so much. CoT spreads the decision across many tokens, so the harm of any single wrong commit is locally contained — the model can recover in later steps. Greedy decoding without CoT is the most cascade-vulnerable configuration.

10.4 Temperature and top-p, together

Modern APIs let you set temperature and top-p simultaneously. Recall from Lecture 5 §14.3 that top-p sampling keeps the smallest set of tokens whose probabilities sum to at least \(p\), throws away the rest, and renormalises. The standard production default is \(T = 0.7\) with top-p = 0.95, and the combination is doing real work.

Temperature shapes the distribution globally. Top-p crops the long tail adaptively — when the model is confident (a few tokens carry most of the mass), top-p keeps few; when the model is uncertain (many tokens share mass), top-p keeps many. The combination temperature warms or cools the distribution, top-p chops off the pathological tail. You almost always want both. \(T = 0.7\) alone occasionally samples a junk token from the deep tail; top-p = 0.95 alone leaves the distribution unscaled and the top of the distribution maybe too dominant. Together they give you a reasonable sharpening with a hard cap on weirdness.

For software-engineering tasks, the production defaults are good defaults. If you want more reliability, push T toward 0.3 and keep top-p at 0.9. If you want more diversity, push T toward 1.0 and keep top-p at 0.95. If you want pure determinism, set T to 0; top-p becomes irrelevant in that regime.

10.5 What temperature does not solve

Honesty: temperature is the simplest distribution-shaping tool, and it has limits.

In other words: temperature is the easiest knob, with the smallest effect. The rest of Block B is about knobs that act earlier in the pipeline — at prompt construction time — and that can shape the distribution far more substantively than any sampling-time scaling. Same goal (sharpen the distribution around correct answers), different mechanism, much more leverage.

10.6 Where the name temperature comes from

Worth a stepping-back moment before we leave the chapter. Softmax-with-temperature is not a piece of ML jargon — it is a 150-year-old object from statistical mechanics called the Boltzmann distribution (or Gibbs distribution). For a physical system with possible states \(i\), each with energy \(E_i\), in thermal equilibrium at temperature \(T\), the probability of finding the system in state \(i\) is:

\[p_i \;=\; \frac{e^{-E_i / kT}}{Z}, \qquad Z \;=\; \sum_j e^{-E_j / kT}\]

where \(k\) is Boltzmann’s constant and \(Z\) is the partition function (the denominator). That formula is literally softmax-with-temperature: identify “logit \(\ell_i\)” with “\(-E_i / k\)” and the two expressions coincide term-for-term. The \(T\) here is physical temperature, measured in kelvin. The intuition transfers without modification:

ML borrowed the name because the math is identical, the intuition transfers without translation, and the limits are exactly the ones a physicist would expect. “Sampling at temperature \(T\) from a language model” is structurally the same activity as “observing a physical system at temperature \(T\).” This is also why the engineering regimes in §10.1 have such clean limits: they are the same limits the Boltzmann distribution has had since the 1870s, just applied to a discrete vocabulary instead of an energy spectrum.

A small subtlety worth naming: the same scalar \(T\) is applied uniformly across the entire vocabulary — isotropic scaling, in the language of Lecture 5 §14.2. You could in principle define an anisotropic version with a vector of temperatures \((T_1, \ldots, T_V)\), one per token, but nobody does. The whole point of temperature is to be the single thermal-equilibrium knob the Boltzmann analogy gives you. Per-dimension temperatures break the physical interpretation, and anything you would want them to do is already covered more flexibly by logit biases, top-k, and top-p.

The next chapter is the centrepiece of Block B. It introduces in-context learning as a real theoretical object and shows you the attention-head mechanism that, mechanically, makes few-shot prompting work.

11. In-Context Learning and the Induction Heads That Make It Work

This is the chapter Block B is built on. The next three chapters (few-shot, chain-of-thought, self-consistency) are applications of what we establish here. The goal is to turn in-context learning from a folklore phrase into a real mechanistic object — a phenomenon you can name, point at in the model’s attention pattern, and reason about in last week’s vocabulary.

11.1 What in-context learning is

Definition. In-context learning (ICL) is the phenomenon by which a large language model, with no weight updates, changes its behaviour in response to examples placed in its prompt. You write three input-output pairs into the prompt. The model, on the fourth input, emits an output that follows the pattern of the first three. No gradient descent. No fine-tuning. No training. Just a forward pass over a longer context that includes examples.

This is, on its face, a strange thing for a model to do. The model has not been taught to learn from examples in any explicit way. There is no piece of the training objective that says “if the prompt contains demonstrations, follow the demonstrations.” ICL is emergent — it appears at scale, in models trained purely on next-token prediction, and the discovery that it works was one of the central findings of the GPT-3 paper.

11.2 The 2020 surprise: GPT-3 and the headline result

Brown et al. (2020), Language Models are Few-Shot Learners, was the paper that introduced GPT-3 and named in-context learning as its headline capability. The setup, in the simplest framing the paper uses:

The paper’s empirical claim is that few-shot performance scales sharply with model size, in a way that one-shot and zero-shot performance do not. Small models do not benefit much from in-context examples — adding three demonstrations to GPT-2 does not meaningfully change its output. Large models (the GPT-3 scale, ~175B parameters) do benefit, often dramatically: on some benchmarks the gap between zero-shot and few-shot is 10–30 percentage points.

The headline of the GPT-3 paper was not “we made a bigger language model.” The headline was “we trained a language model and an unexpected new capability emerged at scale: it can learn from examples in its prompt, with no parameter updates.” That capability is what every prompting pattern in this lecture exploits.

There is some debate in the literature about whether this is a sharp phase transition or a smooth scaling, and recent work (Wei et al. 2022 on emergent abilities; later critiques) has refined the picture. For our purposes the operational summary holds: large models do ICL, small models do not, and Block B’s prompting patterns work because the model you are interacting with is large enough to do it.

11.3 The mechanism, in last week’s vocabulary

Now the part that matters for engineers. ICL is not a separate algorithm bolted onto the model. It is the same attention machinery from Lecture 5, doing what attention does, on a longer context.

When you put a few-shot example into a prompt, you are giving the model additional context tokens to attend to. The model’s forward pass — unchanged from Lecture 5 — processes those tokens, builds Q, K, V projections, computes attention patterns, mixes values, runs through the FFN, layer by layer, all the way to the unembedding. At the final position, where the model is choosing the next token of its response, attention heads at various layers route back over the example tokens. The values they pull in shift the final representation, which shifts the unembedding logits, which shifts the output distribution.

The example, mechanically, is additional context. It is not “training data.” The model’s weights are unchanged. The shift in behaviour is entirely a consequence of changing the conditioning context for the same fixed function. This is what we mean when we say ICL is attention-based conditional pattern matching.

The remarkable part is that, for some patterns, the conditioning is strong enough to look like learning. If you give the model three (English, French) translation pairs and then an English sentence, it will emit French — even on language pairs the model has never explicitly been trained to translate, even on token sequences the model has never seen. It is not learning French in the prompt. It is recognising the translation-pair pattern and continuing it. The attention heads have to do that recognition, and that is where the next subsection comes in.

11.4 Induction heads: the mechanistic substrate

Olsson et al. (2022), In-context Learning and Induction Heads (Anthropic), is the paper that gave ICL a specific mechanistic name. The headline finding: there are attention heads in trained transformers that implement a very specific pattern of routing, which the authors call induction heads, and these heads are tightly correlated with the model’s ability to do in-context learning.

An induction head implements the following routine. Given a current token \(A\), find the previous occurrence of \(A\) in the context, look at the token \(B\) that immediately followed that previous \(A\), and route attention from the current position back to \(B\). In one sentence: if you saw [A][B] earlier, and now you see [A], predict [B].

Induction head routing pattern A sequence of 7 tokens. Current token "cherry" at position 7 matches the earlier "cherry" at position 5. The induction head routes attention from position 7 back to position 6 — the token immediately following the previous match, which is "red". induction head routes here apple red banana yellow cherry red cherry pos 1 pos 2 pos 3 pos 4 pos 5 pos 6 pos 7 previous match ↑ attention target current (query)
Olsson et al.'s induction head made literal. The current token cherry (position 7) appears earlier at position 5; the token immediately following that previous match is red at position 6. The induction head routes attention from position 7 back to position 6, so the value vector at position 6 mixes into the current position's representation — biasing the next-token prediction toward red. The mechanism in one sentence: find previous match → look at the next token → route there.

This sounds trivially simple. It is. The remarkable empirical fact is:

  1. It is exactly the algorithm that few-shot needs. Few-shot prompts are sequences of (input, output) pairs followed by a new input. The pattern is [in1][out1][in2][out2]...[in_k][out_k][in_query]. To produce the right \([out_query]\), the model has to look at how previous inputs were followed by outputs. That is induction-head routing, on the granularity of multi-token inputs and outputs.
  2. It emerges at a specific point in training. Tracking the loss during pre-training, Olsson et al. find a small but visible phase transition — a “bump” — where the loss curve briefly flattens and then resumes descending. They show that this bump coincides, mechanistically, with the formation of induction heads in the model. Before the bump, the model cannot do few-shot. After the bump, it can.
  3. It is empirically traceable. Olsson et al. and follow-up work can identify specific attention heads in specific layers of specific real models that implement induction. You can visualise the attention pattern of these heads and see the routing: from the current token back to the previous occurrence’s neighbour. They are not theoretical objects; they are observed heads in observed models.

This is the kind of mechanistic anchor that makes prompting engineering rather than folklore. Few-shot is not “the model magically learns from examples.” For copy-and-continue patterns at least, few-shot is “the model has induction heads that route from a query input back to the corresponding output in earlier examples, the values mixed in via that routing shift the final representation toward the example output’s pattern, and the unembedding lands on tokens that continue that pattern.” Every word of that sentence is a Lecture 5 mechanism.

A calibration note before we go on. Induction heads are one of the best-understood mechanistic substrates of in-context learning, especially for copy-and-continue patterns. They are not the complete explanation. More recent work — for example, Musat (2025), On the Emergence of Induction Heads for In-Context Learning and related lines — shows that some forms of ICL persist even when induction-style copying is suppressed in attention heads, suggesting that abstractive ICL (where the model has to compose or transform examples, not just copy) involves additional mechanisms we still understand less well. For our engineering purposes the picture remains useful: when you write few-shot prompts whose payload is largely “match this pattern,” you are leaning on induction-head routing, and the Min et al. result in §11.5 confirms that this is where most of the lift comes from. When you write few-shot prompts that require the model to abstract across examples rather than copy from them, you are relying on mechanisms that are less mechanistically transparent, and the engineering rule “more examples does not always mean more accuracy” is one of the empirical signs of that.

11.5 The Min et al. surprise: format matters, content matters less

If induction heads are the substrate, the question of what the example content needs to be becomes interesting in its own right. The counterintuitive answer comes from Min et al. (2022), Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?

The experiment is simple. Take a classification task, like sentiment analysis. Set up few-shot prompts with \(k\) demonstrations. In the baseline condition, each demonstration is a (review, correct-label) pair. In the experimental condition, replace the labels with random labels — pair the review with a label drawn uniformly at random from the label set, without regard to the actual sentiment of the review.

The expected result, if few-shot worked by “showing the model the answer,” would be a collapse in accuracy. The actual result: the random-label condition’s accuracy is only slightly lower than the correct-label condition’s accuracy, and much higher than zero-shot. Most of the lift from few-shot — most of the gap between zero-shot accuracy and few-shot accuracy — survives randomising the labels.

What does this mean? The labels in the demonstrations are not primarily teaching the model what answer to give. What is transferring across the demonstrations is:

In induction-head terms: the heads route from the current input back to the position immediately following the previous similar input. If the labels in those positions are random, the heads still route there, and the model still treats the demonstration positions as what comes next. But because the labels are random, the conditioning signal is “the format is label-shaped” without “the answer is X-shaped.”

The model is using the demonstrations primarily as format and task-identity cues, and only secondarily as answer guides. This is one of the most counterintuitive empirical results in the prompting literature, and it has direct engineering consequences for chapter 12: when you write few-shot examples, invest your effort in matching the format and the task shape, not in writing perfect example answers.

11.6 The pedagogical anchor: a logit trace

To make this concrete, imagine the simplest possible case. A sentiment classification task with two labels, positive and negative. A query review: “The plot dragged for the first hour but the ending saved it.”

Zero-shot: prompt is just the review and “Sentiment:”. The logits over the next token are spread over many candidates — positive, negative, mixed, neutral, the, and a long tail of tokens that are plausible English continuations of “Sentiment:”. The distribution is fairly flat between the legitimately competing classifications, and the top-1 token might not even be one of the canonical labels — it might be mixed or neutral if those appear in training data after the word “Sentiment:”.

Few-shot with two correctly-labelled demonstrations: prompt now contains two prior (review, label) pairs before the query. The induction heads at the relevant layers route from the final position back over the demonstration pattern. The values they mix in carry information about how prior reviews were classified. The final-position representation now lives in a region of activation space that is much more concentrated on the label vocabulary. The logits at the unembedding sharpen dramatically on positive and negative. The long tail drops away. The distribution is peaked on the legitimate label set, and the model picks one of them — likely the one the demonstrations’ pattern suggests is more probable for a review of this shape.

Few-shot with two randomly-labelled demonstrations: induction heads still route. The final-position representation is still in the label-concentrated region of activation space — the heads still pulled values from positions that were followed by something label-shaped. The label vocabulary still dominates the logits. The model still picks positive or negative. The which of those two it picks may be slightly worse than in the correctly-labelled case (the random labels are mildly misleading), but the that it picks a label at all — the format collapse — is preserved.

Zero-shot — no demonstrations Flat
Mass spread across many continuations — the top-1 may not even be a label token.
Few-shot — correctly-labelled demonstrations Peaked ✓
Peaked on positive — the right label for this review.
Few-shot — randomly-relabelled demonstrations Peaked, but wrong
Still peaked, still on a label token — but on negative. Format transferred; correctness didn't.
Three logit distributions over the same vocabulary. Demonstrations shape the peak; their content shapes which peak wins; the peak itself is format-driven, not content-driven. The "wrong" peak in row 3 is the load-bearing observation behind Min et al.'s result.

This is what the Min et al. result is observing, mechanistically. The induction heads care about the positions of the demonstration outputs; they do not care that the outputs were the right outputs. The model is using the structure of the demonstrations as a prompt-engineering signal, and the structure transfers even when the content is randomised.

You should treat this picture as your default mental model for few-shot prompting. Chapter 12 turns it into engineering rules.

12. Few-Shot in Practice

Chapter 11 established that few-shot prompting is induction-head-driven pattern continuation, and that the format of the demonstrations transfers more than the content. This chapter is the engineering follow-up: given the mechanism, how do you actually write few-shot prompts that work?

12.1 How many examples?

Empirically, the marginal value of additional examples decays fast.

The right \(k\) is a budget-versus-quality tradeoff, not a fixed number. A short task with a clear format might need only one example. A nuanced task with a non-obvious output shape might benefit from four or five. Past that, the law of diminishing returns is unforgiving, and you should consider whether you really want few-shot or whether you want fine-tuning.

12.2 Order matters: the recency-bias caveat

If your examples conflict — if two demonstrations show different output styles for similar inputs — the model is disproportionately influenced by the last example. This is the recency effect we saw in chapter 7 (lost in the middle, end advantage) applied to demonstrations specifically.

Two practical consequences.

Both rules are predictable from the induction-head story. Routing favours nearby positions; conflicting examples send the heads to incompatible patterns simultaneously, which makes the mixture’s signal noisier rather than sharper.

12.3 Format beats content

Min et al. (2022), chapter 11.5, is the empirical anchor. The practical reading:

This is not licence to use random labels in production. The correctly-labelled condition does outperform the randomly-labelled condition; the gap is just smaller than naive intuition predicts. The actionable framing: prioritise format quality, accept reasonable-quality answers, and do not spend an hour perfecting the third decimal place of a demonstration answer when a quick plausible answer would do.

12.4 When few-shot does not help — or actively hurts

Few-shot has failure modes worth naming.

Tasks the model has been heavily RLHF’d for. Chat-assistant tasks like “summarise this article,” “explain this concept,” or “write a polite email” are already in the high-confidence region of the RLHF’d model’s distribution. Adding few-shot examples consumes budget without sharpening behaviour. The induction heads are still doing their work, but the unconditional distribution was already peaked correctly. There is little distance left to travel.

Tasks where the demonstration distribution drifts from the query. If your examples are short and your query is long, or your examples are in one domain and your query in another, the induction heads’ routing pulls in less-relevant patterns. You can produce worse output than zero-shot in this regime — the model is being pulled toward the demonstrations’ shape when the query needed a different shape.

Tasks where examples are misleading. If your demonstrations contain bugs, hallucinations, or stylistic choices you would not want repeated, the model will faithfully repeat them. Few-shot is exactly the wrong technique for tasks where you do not have known-good examples.

Tasks where the input is itself enormous. Few-shot examples compete with the input for token budget. For long-document tasks (summarise this 50-page PDF), few-shot is often crowded out by the input itself; in that regime, zero-shot or zero-shot CoT (chapter 13) is usually a better choice.

12.5 Code-specific few-shot: examples beat instructions for style

For software-engineering use cases, few-shot has a specific superpower: examples are how you communicate codebase-specific style.

Compare two ways of asking the model to write a new error-handling function for your codebase.

Instruction-style:

“Write a function that handles HTTP errors. Use our project’s standard logging. Raise our ApiError exception with the status code. Always include the request ID in the error message. Use snake_case. Keep functions under 30 lines.”

Few-shot style:

“Here are two examples of how we handle HTTP errors in this codebase:

[example 1: existing handle_http_error function from your repo] [example 2: another existing error handler from your repo]

Now write a similar handler for the new endpoint [X].”

The instruction-style prompt requires the model to construct your codebase’s pattern from a verbal description. Each constraint is a separate piece of conditioning, and many can be misinterpreted. The few-shot prompt shows the model the pattern. The induction heads can route over genuine examples of the codebase’s style. The result is much more reliably consistent with the existing codebase.

This is the engineering principle that makes few-shot indispensable for AI coding tools: for project-specific conventions, examples beat instructions. Show, do not tell. The mechanism is induction heads doing exactly what they do, applied to your codebase’s idioms instead of a paper’s experimental tasks.

It is also the bridge to Lecture 7. A skill, in Claude Code, is a codified few-shot pattern. A .claude/skills/error-handling/SKILL.md file that includes two existing handlers from your repo as examples is, mechanically, a saved few-shot prompt. Lecture 7 will show you how the orchestration layer loads these on demand and threads them into the right point in the conversation. For now: every time you reach for a skill, you are using the mechanism we just built.

12.6 A worked example, anchored in Act 1

To make this concrete, take a task you have already done. In the Stack Overflow Challenge from Lecture 2, several students asked the model to query the Stack Exchange API and produce monthly question counts. The model frequently hallucinated endpoints (chapter 5, factual hallucination).

Imagine running the same task three ways.

Zero-shot:

“Write a Python script using requests that queries the Stack Exchange API to count the number of questions tagged python per month over the last year.”

The induction heads have no examples to route over. The distribution at “the endpoint URL is” is wide. The model emits a plausible-shaped URL, sampled from training-data statistics about how Stack Exchange URLs usually look. The probability that the specific URL exists is the probability that the training data happened to mention this specific endpoint many times. Often it does not; the model hallucinates.

Few-shot with two examples of correct API calls:

“Here are two examples of how we call the Stack Exchange API in this project:

requests.get('https://api.stackexchange.com/2.3/questions', params={'order': 'desc', 'sort': 'creation', 'site': 'stackoverflow', 'tagged': 'python'})

requests.get('https://api.stackexchange.com/2.3/questions', params={'order': 'desc', 'sort': 'creation', 'site': 'stackoverflow', 'fromdate': 1735689600, 'todate': 1738368000})

Now write a script that gets monthly counts of python-tagged questions over the last year.”

The induction heads can now route over genuine endpoint references. The distribution at “the endpoint URL is” is heavily concentrated on https://api.stackexchange.com/2.3/questions — the exact endpoint the examples used. The structural hallucination space has collapsed. The model may still get the parameter logic wrong, but the endpoint hallucination is solved.

Few-shot plus zero-shot CoT (a teaser for chapter 13):

“[Same two examples] Now think step by step about which API endpoint and parameters you need, then write the script.”

The induction heads still route over the examples. The CoT trigger adds an additional reasoning pass that, in many runs, names the endpoint and parameters explicitly before committing to the code. This makes the first-token cascade (chapter 10) less risky: the model has reasoned about the endpoint before it generates it.

This is what “engineering the influence layer” looks like. Each pattern is a different mechanism for shifting the distribution: zero-shot relies entirely on pre-training and RLHF; few-shot adds in-context examples that route through induction heads; CoT adds intermediate reasoning that uses the auto-regressive loop as a scratchpad. The next chapter formalises the third pattern.

13. Chain-of-Thought: The Auto-Regressive Scratchpad

Chapter 13 is the chapter where Lecture 5 §15 — the auto-regressive loop — pays the cleanest engineering dividend. The connection is so direct that, once you see it, chain-of-thought stops feeling like a separate technique and starts feeling like a deliberate use of mechanism you already understand.

13.1 The empirical headline

Wei et al. (2022), Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, established the headline result. On reasoning-heavy benchmarks — multi-step arithmetic, common-sense inference, multi-step word problems — prompting the model to show its working before producing the final answer produced dramatic accuracy gains. On some benchmarks, accuracy doubled. The improvements concentrated at scale: small models did not benefit, but at the GPT-3 / PaLM scale (hundreds of billions of parameters) the effect was unambiguous and large.

The pattern Wei et al. studied was a few-shot variant. The demonstrations in the prompt included not just (question, answer) pairs but (question, reasoning chain, answer) triples. The model, given a new question, would emit its own reasoning chain and then the answer. The reasoning was visible, the answer was correct more often, and on benchmarks like GSM8K (grade-school math) the gap between direct prompting and chain-of-thought prompting was 10–30 percentage points.

A follow-up paper, Kojima et al. (2022), Large Language Models are Zero-Shot Reasoners, showed something even more striking: most of the chain-of-thought benefit was available without any example demonstrations. The phrase “Let’s think step by step,” appended to a prompt as a zero-shot trigger, recovered most of the gain. This is zero-shot CoT, and it is the form most users invoke without realising it: every time you ask the model to “explain your reasoning” or “think it through,” you are deploying Kojima et al.’s result.

The trigger phrase itself is doing work. It is a pattern from training data — billions of “Let’s think step by step” continuations in math tutorials, in textbooks, in homework solutions, in Stack Overflow answers — and invoking it shifts the model’s distribution toward the style of writing that follows that pattern, which happens to be the style of explicit, step-by-step reasoning.

13.2 The mechanism: Lecture 5 §15, deliberately

Now the part that makes this satisfying. Chain-of-thought is the auto-regressive loop being used on purpose.

Recall the loop. Each forward pass takes the entire prior sequence as input and produces logits for one more token. The token is sampled, appended, and the loop runs again. Every generated token conditions every subsequent token. The model is free to attend back to its own intermediate outputs at every step.

In direct prompting — “What is 47 × 83?” answered with “3901” — the model produces the answer in one or two tokens. Whatever computation needed to multiply happened inside a single forward pass. The transformer has a fixed depth (\(N\) layers); the answer is a function of \(N\) rounds of attention and FFN computation over the prompt tokens. If the right answer requires more rounds than the model has layers, direct prompting cannot produce it. The model will emit something — softmax always gives you a distribution — but it will not be the correct answer; it will be whatever the model can compute in its available depth.

In chain-of-thought — “What is 47 × 83? Let’s think step by step. 47 × 83 = 47 × 80 + 47 × 3 = 3760 + 141 = 3901.” — the model produces intermediate tokens. Each intermediate token is generated by a full forward pass. The next forward pass sees the intermediate token in its context. Attention heads in subsequent forward passes can route back over the intermediate reasoning, building on it.

In other words, chain-of-thought trades depth for length. A single forward pass gives you \(N\) rounds of computation. CoT, by generating \(L\) reasoning tokens before the answer, gives you effectively \(N \times L\) rounds — the answer’s forward pass attends over the prompt and over the \(L\) intermediate tokens, each of which was itself the output of \(N\) rounds of computation. The model has stretched its effective depth.

Direct generation — one forward pass
pass 1
answer
↑
Layer N
⋮
Layer 2
Layer 1
↑
prompt
depth = N
Chain-of-thought — L+1 forward passes; each reads the growing context
pass 1
t1
↑
N
⋮
1
↑
prompt
pass 2
t2
↑
N
⋮
1
↑
prompt t1
pass 3
t3
↑
N
⋮
1
↑
prompt t1 t2
…
pass L+1
answer
↑
N
⋮
1
↑
prompt t1 t2 t3 … tL
effective depth = N × L
Each pass is one full N-layer forward pass that emits a single token. That token is appended to the input of the next pass — the growing strip at the bottom of each stack shows the cumulative context that pass actually reads. By the time the answer-producing pass (right-most, indigo) runs, its input contains every prior intermediate token, each itself the output of N layers of computation. That is the precise sense in which CoT trades depth for length: same architecture, same N — L-times the effective depth.

This is not a metaphor. It is a real expressivity result. Feng et al. (2023) and Merrill & Sabharwal (2023) have formal results showing that there are problem classes which are provably hard for direct generation in fixed-depth transformers, but which become tractable when the model is allowed to generate intermediate reasoning tokens. The proof techniques use circuit-complexity arguments — depth-bounded transformers can express only certain classes of functions; CoT moves the model into a strictly more expressive regime. We will not work the proofs; the result is mentioned to anchor the intuition. The chain-of-thought gain is not an empirical accident. It is the model accessing more computation by spending more tokens.

13.3 The scratchpad analogy

A useful image. The model in direct-generation mode is doing mental arithmetic — it has to produce the answer in one quick step, using only the working memory of one forward pass. The model in chain-of-thought mode is using a scratchpad. It writes intermediate steps. It can look back at what it wrote. It can use the intermediate values in subsequent steps. The same model, the same parameters; the difference is whether it gets to use scratch paper.

For tasks that need scratch paper — arithmetic, multi-step reasoning, plan-then-act — CoT is the difference between a model that pretends to reason and a model that reasons. For tasks that do not — a lookup, a single-step classification, a simple lookup of stored knowledge — CoT is overhead. The model already had the answer; making it write its working merely consumes tokens.

13.4 Zero-shot CoT versus few-shot CoT

Kojima et al.’s result is operationally important: most CoT benefit is available without demonstrations.

Operational rule: try zero-shot CoT first. If the model’s style of reasoning is appropriate, you are done at no extra token cost. If the reasoning is the wrong shape — too long, too short, in the wrong format — add one or two demonstrations to fix the shape. Few-shot CoT is rarely justified beyond two or three demonstrations; the format-transfer effect (chapter 11.5) does its work quickly.

13.5 When chain-of-thought hurts

Four failure modes are worth naming.

Simple lookups. “What is the capital of France?” Direct: Paris. CoT: “France is a country in Europe. Its capital city is the one where the President of France lives. The President of France lives in the Élysée Palace. The Élysée Palace is in Paris. Therefore, the capital of France is Paris.” Same answer, ten times the tokens. The model already had the answer in a single forward pass; CoT adds opportunity for the auto-regressive loop to drift without adding any depth the model needed.

The first-token cascade, on reasoning steps. Chain-of-thought reduces the first-token-of-the-answer cascade because the model has reasoned before committing. But it introduces a first-token-of-the-reasoning cascade: if the model commits to a wrong intermediate step early, every subsequent step is conditioned on it. The auto-regressive loop is a feature, not a guarantee. A CoT chain that turns the wrong way at step one is a longer wrong answer, not a more correct one.

Tasks where the model does not know the answer. CoT extends the model’s effective depth, but it does not add knowledge. If the model genuinely does not know what API endpoint exists, asking it to think step by step will produce a step-by-step explanation of how it is making up the endpoint, with the same end result. The fake reasoning sometimes makes the hallucination more convincing — it now comes with elaborate justification — which is its own failure mode.

Trusting the chain as an audit trail. This is the failure mode that catches engineers most often, and it deserves its own warning. The reasoning chain you read is useful — it spreads computation across tokens, catches some intermediate errors, gives you a debugging surface — but it is not necessarily a faithful record of how the model arrived at its answer. The model can generate a clean, plausible-sounding chain that bears little relationship to the actual computation that produced the final token. The result is a CoT that justifies the answer post-hoc rather than deriving it. Lyu et al. (2023), Faithful Chain-of-Thought Reasoning, and a substantial follow-on literature, are explicit on this point: visible CoT is not an interpretability tool. Ask for a plan, trace, or checklist when it is useful for the task. Do not treat visible reasoning as proof of how the conclusion was reached, and do not rely on it for high-stakes verification — that is what tests, hooks, and human review are for (chapter 15.5).

13.6 Software-engineering CoT: plan, then code

For code generation, the most reliable CoT pattern is plan before coding. Concretely:

“First, list the API endpoints, library functions, and data structures you intend to use. Then write the code.”

The model emits a plan as tokens. The plan’s tokens become context for the code generation that follows. Two payoffs.

For debugging, the dual pattern is trace, then conclude. Ask the model to walk through what each relevant line does before stating where the bug is. The reasoning tokens force the model’s attention through every step, and the conclusion is conditioned on the trace.

These are not folklore prompts. They are mechanism-aware uses of the auto-regressive loop. You are deliberately spending tokens to buy the model more effective depth, on the kinds of tasks where depth is what was missing.

13.7 Connection to modern reasoning models

The newest generation of “reasoning models” — OpenAI’s o1 family, DeepSeek-R1, Anthropic’s extended thinking, Google’s Gemini thinking models, Qwen’s QwQ — internalise chain-of-thought. They are trained to generate long internal reasoning traces, often hidden from the user, before producing a final answer. The reasoning trace can run to tens of thousands of tokens for a single response. Several misconceptions about these models are worth pre-empting because they bear directly on how you should think about prompting them.

Reasoning models are not architecturally different. Same attention. Same FFN. Same \(N\) layers. Same auto-regressive loop. Same softmax sampling at the output. Everything you built in Lecture 5 is intact. There is no new mechanism. What is different is the behavioural distribution the model has been trained to emit. When you ask a reasoning model for an answer to a hard math problem, it generates a long sequence of reasoning tokens, and that long sequence is what produces the better answer. The auto-regressive scratchpad from chapter 13.2 is the entire story; the model has just been trained to use it aggressively.

The training is different. A vanilla post-trained model is RLHF’d against human preferences over response forms (chapter 3.3). A reasoning model is additionally trained with reinforcement learning against verifiable correctness signals — does the final answer match the known-correct answer for a math problem? Does the generated code compile and pass tests? Does the proof check? The reward signal is no longer “do humans prefer this?” but “is this actually right?” This changes what the model learns to do during reasoning, because the gradient now flows from a much stronger truth signal. DeepSeek-R1 (Guo et al., 2025) is the cleanest public account of this; their R1-Zero variant skipped SFT entirely and showed that pure RL against correctness — using their GRPO algorithm rather than PPO — was enough to elicit behaviours like step-by-step decomposition, backtracking on dead ends, self-verification, and reconsidering wrong intermediate steps. Some of these behaviours emerged spontaneously without being explicitly trained for. OpenAI’s o1 (announced September 2024; no detailed paper) used a similar though undisclosed recipe. The pattern is reproducible enough that, by mid-2025, every major lab had a reasoning model in production.

Reasoning models are the productised form of test-time-compute scaling — the conceptual move chapter 14 will introduce explicitly. Wang et al. (2022) showed that more inference-time compute on a fixed model improves accuracy. Self-consistency was the early empirical demonstration. Reasoning models are that idea internalised: the model decides how much compute to spend on a query (typically much more than non-reasoning models), generates that compute as reasoning tokens, and emits a final answer. The pricing model has shifted accordingly — reasoning-model APIs cost noticeably more per request than non-reasoning ones, because the cost is dominated by inference-time tokens.

What this means for prompting. The operational rule, as the vendors themselves now phrase it: with a reasoning model, prompt for the goal, constraints, context, allowed tools, output format, and verification criteria — not for a visible chain-of-thought. The model is already going to reason internally; the question is whether your prompt gives it the information and the success criteria it needs. OpenAI’s reasoning best-practices documentation is explicit that step-by-step prompts are not needed for these models and can degrade quality. OpenAI’s reasoning guide describes a reasoning.effort control and frames reasoning models as the natural choice for coding, scientific reasoning, and multi-step workflows. Google’s Gemini thinking documentation gives a parallel account for the Gemini family. Across vendors the engineering picture has converged.

For the standard non-reasoning models — the day-to-day workhorses for most code generation, summarisation, and chat tasks — explicit CoT triggers and few-shot reasoning examples are still useful when the task needs intermediate steps. Knowing which class of model you are calling is now part of choosing the right prompt; the side-by-side comparison table lives in chapter 15.4. And the faithfulness caveat from §13.5 carries over: even a reasoning model’s visible reasoning trace, when surfaced, is not a guaranteed audit trail.

The next chapter takes the auto-regressive scratchpad one step further with a technique that was the conceptual seed for the reasoning-model paradigm. If one CoT path can be wrong, what about many?

14. Self-Consistency and the First Test-Time Compute Scaling Result

Wang et al. (2022), Self-Consistency Improves Chain of Thought Reasoning in Language Models, is a paper that, at first glance, looks like a small empirical improvement on chain-of-thought. Sample several CoT responses at moderate temperature, take the majority answer, accept it. The accuracy gains are real but modest — a few percentage points on most benchmarks. The reason the paper matters is not its numerical gain. It is the conceptual move.

14.1 The recipe

The technique is procedurally simple.

  1. Construct a chain-of-thought prompt (zero-shot or few-shot, chapter 13).
  2. Sample \(N\) responses at temperature \(T > 0\). Each sample produces a complete reasoning chain and a final answer.
  3. Aggregate the final answers across the \(N\) samples. Take the majority vote.
  4. Return the majority answer (and, optionally, the reasoning chain associated with one of the samples that produced it).

The temperature has to be non-zero for the samples to differ from each other; the original paper uses values around \(T = 0.5\). Typical \(N\) values range from 5 to 40. The compute cost is linear in \(N\) — five times as many forward passes for an \(N = 5\) self-consistency call.

14.2 Why it works: the distribution-over-paths argument

The mechanism is genuinely interesting and worth pausing on. It is a variance-reduction argument from elementary probability.

Consider the model’s behaviour at temperature \(T = 0\) on a CoT prompt. The auto-regressive loop produces one specific reasoning chain. That chain ends in one specific answer. If the chain happens to be a good one, the answer is correct; if not, not. The first-token cascade from chapter 10 is fully in effect — one wrong commit early in the chain produces a wrong answer with high confidence.

Now consider \(T > 0\). Each sample produces a different reasoning chain — different word choices, different intermediate steps, sometimes radically different approaches to the problem. The distribution over reasoning chains is broad. Many chains lead to the right answer. Some lead to wrong answers. The probability that any single chain is correct is some number \(p\); the probability that the majority of \(N\) independent samples are correct is much higher than \(p\), provided \(p > 0.5\). This is just the law of large numbers in action: independent noisy votes for the truth concentrate on the truth, faster than each individual vote does.

More precisely: the distribution over reasoning paths is wide — many ways to think about the problem. The distribution over final answers, after marginalising over paths, is narrower — different paths often converge on the same answer. Self-consistency exploits the marginalisation: instead of trusting one path’s answer, you sample many paths and trust where they agree.

There is a subtlety. Self-consistency works only when correct paths are more likely than any individual wrong path. If the model’s most common reasoning failure mode is consistently wrong in the same way — if “the wrong answer” is more popular than any single right answer — self-consistency reinforces the bias. The N-vote concentrates on the modal answer, and if the mode is wrong, you get a more confident wrong answer. This is not a hypothetical: tasks where the model has been systematically miseducated on a misconception will fail self-consistency exactly because the misconception is the mode.

14.3 The conceptual move: test-time compute scaling

The reason self-consistency matters beyond its modest accuracy gains is what it proves about inference-time computation. Until 2022, the story of LLM capability gains was: train bigger models on more data, and accuracy goes up. Self-consistency was one of the first results showing that spending more compute at inference time, on a fixed model, also makes accuracy go up.

This is a structural claim about how to think about model capability. The same model, with the same weights, has different capabilities depending on how much compute you spend per query. \(N = 1\) self-consistency is just CoT. \(N = 5\) is meaningfully better. \(N = 40\) is meaningfully better still, on tasks where the technique applies. There is a compute-quality curve per task, and you can choose where on the curve to sit by choosing \(N\).

This conceptual move is the seed of the modern reasoning model paradigm. The o1 family, DeepSeek-R1, and their successors are explicit elaborations of “spend compute at inference time to do better reasoning.” They internalise the spend — the user does not pick \(N\); the model decides how much reasoning to do for each query — but the underlying argument is the same one Wang et al. made. Test-time compute is a lever. Self-consistency was the first paper to pull it visibly.

14.4 When self-consistency is and is not worth it

For practising engineers the cost-benefit is usually unfavourable for code generation, and usually favourable for evaluation and difficult one-off problems.

Production code generation. Rarely worth it. Code generation usually wants determinism (so you can debug what was generated) and is highly sensitive to compute cost (you make many calls). \(N = 5\) means five times the latency and cost for a modest accuracy bump. Better to invest in better few-shot examples or better chain-of-thought triggers.

Evaluation pipelines. Often worth it. If you are running benchmarks where you want to know the best-effort accuracy of the model on a task — exactly the SWE-Bench-style situation that motivates Lecture 7’s bridge to evaluation — self-consistency gives a cleaner signal than single-sample CoT. The compute cost is amortised across many evaluation samples and matters less than the noise reduction.

Difficult one-off problems. Sometimes worth it. If you are trying to solve a hard problem and you have time for \(N\) attempts, self-consistency is a way to extract more signal from the model than one shot would give you. You may also want to inspect the different reasoning chains; the disagreement among samples often points to the part of the problem the model is uncertain about, which is itself informative.

Tasks where you can verify correctness cheaply. Almost always not worth it. If you can run a unit test, hit the API, or compile the code, you should use the verification step rather than self-consistency. Self-consistency aggregates noisy answers; verification eliminates wrong answers. The enforcement layer (chapter 8) is a stronger tool than self-consistency whenever it is available.

14.5 What “consensus” buys you

Self-consistency, when used, also produces a small but real confidence signal you would not otherwise have. If 38 out of 40 samples agree, the model is consistent on that question. If 12 out of 40 samples agree (and the other 28 are split among many alternatives), the model is uncertain. The agreement rate is not a calibrated probability — the model can be confidently wrong — but it is a useful diagnostic, especially when you are trying to decide whether to escalate a difficult problem to a human reviewer (the human layer of chapter 8).

For software engineers, this is most often what self-consistency is actually used for in practice: not to win an extra percent of accuracy on a benchmark, but to detect when the model is hedging. A single-sample CoT answer always looks confident; a 12-of-40 majority vote signals “the model is not sure,” and that signal lets the orchestration layer decide whether to ask the user, call out for human review, or run additional verification.

The next chapter pulls all five patterns — zero-shot, few-shot, chain-of-thought, self-consistency, and temperature — into one decision framework.

15. Beyond Prompting: Constraints, Validation, and Limits

Block B’s claim was that every prompting pattern is a way to shape the conditional distribution at prompt-construction time. That claim is true. It is also incomplete. The influence layer can sharpen the distribution around correct answers, but it cannot enforce the answer’s shape, ground it in external truth, or stop it from being adversarially redirected. For production software-engineering work, prompting is a starting point, not a finishing line. This short chapter names the four most important things you do in addition to prompting — constrained decoding, retrieval and tool use, validation, and security boundaries — and gives you a decision rule for which of them to reach for first.

The chapter is also a setup for what comes next. Chapter 16’s pattern library now closes Block B with the explicit caveat that the patterns are the influence layer, and chapter 17’s bridge to Lecture 7 explains how the orchestration layer wires up the other three layers around them.

15.1 Structured outputs and constrained decoding

When the output has a formal grammar — valid JSON, a typed tool call, a value from a fixed enum, a function signature, a Pydantic schema — do not rely on prompting alone. Prompting can describe the schema in words and ask the model to follow it. Modern inference stacks can do something stronger: they mask the sampling distribution at every step so that only tokens consistent with the grammar can be emitted. This is constrained decoding, also marketed as structured outputs, JSON mode, or function/tool calling.

The mechanism is straightforward and worth understanding because it tells you what constrained decoding can and cannot do. At each generation step, the inference engine consults the grammar (a JSON schema, an enum, a regular expression, a context-free grammar) and zeros out the probability of any token that would make the partial output ungrammatical. Under the hood, modern engines do this efficiently by compiling the grammar ahead of time: a regex or JSON schema becomes a finite state machine (FSM), a context-free grammar becomes a pushdown automaton (PDA). At each step the engine advances the FSM/PDA by one token and, in roughly \(O(1)\) per token (a lookup over the current state’s allowed transitions), masks the logits of any token not in the allowed set. The remaining probability mass is renormalised. Sampling picks from what is left. The model never has the option of emitting an invalid character, an unclosed quote, or a missing required key. OpenAI’s structured-outputs documentation is explicit: text responses adhere to the supplied JSON schema; required keys are present; enum values are valid. The same family of techniques is available across vendors under different names (function calling, JSON mode, response format schemas, grammar-constrained decoding in open-source inference engines like vLLM and llama.cpp); the open-source Outlines library is the canonical reference implementation of the FSM-based approach, and Willard & Louf (2023) is the academic paper that introduced it.

What constrained decoding gives you:

What constrained decoding does not give you:

Operational rule. If the output has a grammar, use constrained decoding plus a parser-plus-retry wrapper. Use prompting to explain the task. Use decoding and validation to enforce the shape. This is the difference between hoping the model produces JSON and making it produce JSON.

15.2 When prompting is the wrong tool

Once constrained decoding is on the table, the larger pattern is visible. Several common AI-engineering failures cannot be fixed by writing a better prompt. They require a different kind of tool. The compact form:

What you need Why prompting alone fails Right tool
Output adheres to a schema Prompting can describe but not enforce structure; under load, the model occasionally violates it. Structured outputs / constrained decoding / function calling (§15.1).
Answer must be grounded in current facts Pre-training is frozen; the model cannot consult external sources. Hallucinations from chapter 4 happen exactly where context is missing. Retrieval (RAG), tool/function calls to real APIs, documentation injection. The orchestration layer (Lecture 7) is where this is wired up.
Code must be correct, not plausible The model can write fluent-looking code that compiles but fails on the actual inputs you care about. Tests, static analysis, type checkers, linters, code review. The enforcement and human layers from chapter 8.
Action must respect policy or trust boundaries Prompted instructions are influence, not enforcement; they can be misread, drifted past, or adversarially overridden (§15.5). Permission design, tool sandboxing, secrets isolation, output validation, audit logs. See chapter 8 (three-layer model) and Lecture 7.

The pattern across all four rows: the influence layer is one of four layers. Block B taught you how to make the influence layer as good as it can be. Production-grade AI engineering treats the influence layer as the starting point and wraps it in mechanisms that enforce the parts prompting cannot.

A useful framing for software-engineering practice specifically: your prompts and instruction files live in the IDE; your enforcement layer lives in CI. The split is not architectural fashion — it is the operational consequence of chapter 8’s three-layer model in a continuous-integration workflow. The prompt biases what the model emits; the GitHub Actions pipeline (or whatever CI you use) decides whether what was emitted is allowed to land. AGENTS.md tells the agent how you want code written; the pre-commit hook, the type checker, and the test suite decide whether what the agent wrote is shippable. Lecture 7 will return to this when it covers how the orchestration layer sits between the two.

15.3 Faithfulness: visible reasoning is not an audit trail

We touched on this in §13.5 and it is important enough to restate as a standalone principle in this chapter.

A chain-of-thought trace — whether you elicited it with “Let’s think step by step” or whether a reasoning model produced it internally — is useful for several engineering purposes. It catches some cascading errors before they reach the final answer. It gives you a debugging surface when the answer is wrong. It surfaces the model’s framing of the problem, which sometimes reveals a misunderstanding earlier than the answer would.

It is not an audit trail. The reasoning chain you read is not a faithful record of how the model arrived at its conclusion. The model can — and empirically does — generate a plausible-sounding chain that bears little relationship to the actual computation that produced the final token. The chain justifies the answer post-hoc more often than it derives it. Lyu et al. (2023), Faithful Chain-of-Thought Reasoning, and a substantial literature since, document the gap.

The engineering rule. Ask for a plan, a trace, a checklist, or a concise rationale when it is useful for the task. Do not treat visible reasoning as proof. For high-stakes verification, lean on outcome tests, schema validation, source checks, and human review. The reasoning chain is a signal, not a guarantee.

15.4 Model class matters

By 2026 the model landscape has stratified. The prompting pattern that is right for a vanilla chat model is wrong for a reasoning model is wrong for a coding agent. The compact decision matrix:

Model / tool class Best prompting style Common mistake Validation mechanism
Standard chat model Clear task, concrete examples, explicit output format, optional CoT for reasoning-heavy steps Asking for hidden or "faithful" reasoning as proof; trusting the visible chain as audit trail Review, tests, source checks, the enforcement layer of chapter 8
Reasoning model Goal, constraints, context, allowed tools, output format, verification criteria; let the model reason internally Over-prompting visible CoT; using "Let's think step by step" when the model already plans to reason at length Outcome tests, concise rationale, structured-output schemas for the final answer
Coding agent Repository context, task boundaries, acceptance tests, what files are in scope, what tools are allowed Treating CLAUDE.md instructions as enforcement; assuming agent loops self-correct without external checks CI, pre-commit hooks, sandboxed execution, test suites, code review
Tool-using agent Clear tool permissions, typed schemas for tool inputs and outputs, scoped credentials Letting the model invent tool outputs in absence of real results; trusting unsanitised tool returns Tool logs, schema validation on every tool call, dry-run modes, human-in-the-loop on destructive actions

The framing to carry forward: the right prompting pattern is a function of the model class, not a universal recipe. The pattern library in chapter 16 names the prompting techniques; this table names which ones apply where.

15.5 Security boundary: accidental vs adversarial

The CLAUDE.md compliance anecdote in chapter 6 described a reliability failure: a model that did not follow stated instructions because attention drifted, multi-head dilution kicked in, or position-of-rule mattered. The model was not adversarial. Nobody attacked it. It simply did not comply.

A different class of failure exists, and it deserves its own line in the engineering picture: prompt injection. A malicious input — embedded in a document the model reads, a web page it browses, a tool result it consumes — overrides the application’s instructions and gets the model to do something the application did not authorise. The model is not “tricked” in any human sense; it is doing what next-token prediction always does, following the most contextually plausible continuation. When the most contextually plausible continuation is “execute the attacker’s instruction,” the model follows it.

This is not a hypothetical risk. The OWASP Top 10 for Large Language Model Applications lists prompt injection as LLM01 — the first and most prevalent risk — and insecure output handling as LLM02. Both bear directly on AI-assisted software development. They both connect to your three-layer model in a specific way:

Not all instruction failures are accidental. CLAUDE.md drift is a reliability problem. Prompt injection is a security problem. The mitigation is not “write a stronger prompt”; it is permission design, tool sandboxing, output validation, secrets isolation, and human review of destructive actions.

The three-layer model from chapter 8 already named what to do. The security boundary just makes the stakes explicit: for adversarial inputs, the influence layer can be assumed defeated. Your safety properties have to live in the enforcement layer (tool permissions, sandboxes, schema validation, network egress controls) and the human layer (review of any action that touches money, secrets, or production).

Lecture 7 — the orchestration layer — is where most of these mitigations are implemented. The prompt itself is a small part of the surface; the configuration of the agent, the scope of its tools, the trust level of its inputs, and the human gates on its outputs are most of the picture.

16. The Pattern Library: Synthesis

You now have five named patterns and five mechanisms. Chapter 15 collects them, names the engineering rules, and explicitly catalogues how each fails, because knowing the failure modes is the engineering payoff. There is one central claim that everything in this chapter elaborates.

Every prompting pattern is a way to make the conditional distribution sharper around correct answers. Temperature shapes the distribution at sampling time. Zero-shot, few-shot, chain-of-thought, and self-consistency shape it at prompt-construction time. Each pattern is a deliberate use of one of the mechanisms we built in Lecture 5.

That is the spine. Now the catalogue.

16.1 What each pattern is, mechanistically

A compressed version of Block B’s content, in one table.

Pattern What you do Mechanism (in L5 vocabulary) Where it lives in this lecture
Zero-shot Ask directly. No examples, no reasoning trigger. Relies on pre-training + RLHF having seen this kind of task so often that the right answer is in the peak of the distribution. Implicit baseline; chapter 9 framing.
Temperature Set T at sampling time to sharpen or flatten the distribution. Scales logits before softmax (L5 §14.2). The sampling-time knob. Chapter 10.
Few-shot Provide k input-output examples before the query. In-context learning via induction heads (Olsson et al., 2022). The examples shift the attention pattern; format transfers more than content (Min et al., 2022). Chapters 11, 12.
Chain-of-thought Either prompt visible step-by-step reasoning (chat models) or rely on the reasoning model's internal scratchpad — see §13.7, §15.4. Exploits the auto-regressive loop (L5 §15) as a scratchpad. Trades depth for length — buys effective layers at the cost of tokens. Theoretical expressivity gain (Feng et al., 2023; Merrill & Sabharwal, 2023). Chapter 13.
Self-consistency Sample N CoT chains at T > 0; take the majority answer. Marginalises over reasoning paths. Variance reduction on the answer distribution. The first inference-time-compute scaling result (Wang et al., 2022). Chapter 14.
Role / persona priming "Act as a senior engineer," "You are a math tutor," etc. Biases attention toward subspaces of training data that match the persona's writing style — a deliberate use of the multi-head dilution from L5 §16.1, with the persona token re-weighting the heads that route over similar prior contexts. Mentioned in chapter 1; brief — the mechanism is the same as the others.

Hold on to the central column. Every pattern reduces to a Lecture 5 mechanism. There is nothing in Block B that is not, ultimately, an application of attention, the auto-regressive loop, or sampling-time logit manipulation. Folklore patterns that do not reduce to one of these mechanisms are usually wrong (they exploit specific RLHF artefacts of one model generation, then break) or they are restating one of the patterns above under a different name.

16.2 The decision tree

A practical workflow when you are about to write a prompt.

  1. Is the task well-known and standard? (A validate_email function, a summary of a paragraph, a translation between major languages.) Zero-shot is fine. Run at \(T = 0.3\) to 0.7 depending on whether you want determinism or graded quality. Move on.
  2. Is the task project-specific in its style or format? (Match your codebase’s error-handling pattern; produce a JSON matching your schema; write a test in your house style.) Few-shot with 2–4 examples. Match the format strictly. Place the best example last. Move on.
  3. Does the task require intermediate reasoning? (Multi-step arithmetic; planning before code; tracing a bug; reasoning over multiple constraints.) Start with zero-shot CoT: “Let’s think step by step.” If the reasoning style is wrong, escalate to few-shot CoT with 1–2 demonstrations of the desired reasoning shape.
  4. Is the task accuracy-critical, and can you afford \(N\)× compute? (Evaluation; difficult one-off problems; high-stakes decisions where the model is the only judge available.) Self-consistency on top of CoT. Otherwise: skip.
  5. Combine where useful. Few-shot CoT is the standard for evaluation-style work. Few-shot CoT + self-consistency is the heavyweight option for difficult problems. Temperature is set independently per the table in 15.1, modulo the requirement that self-consistency needs \(T > 0\).

That is the entire library. There are no other named single-prompt patterns you need to know for software engineering. There are many named patterns in the literature (Tree-of-Thoughts, ReAct, Self-Refine, Reflexion, …) — they are all orchestration patterns built on top of these five, where multiple model calls are composed by an outer loop. Those belong in Lecture 7 and Act 3, not in this lecture.

16.3 How each pattern fails

The companion table. Every pattern has a predictable failure mode, and every failure mode is predictable from a Lecture 5 mechanism.

Pattern Failure mode Mechanism of failure
Zero-shot No grounded answer — pattern completion emits a plausible-but-wrong continuation (flat or confidently wrong). Insufficient context. Pattern completion picks a confident wrong continuation (chapter 4).
Temperature At T=0, first-token cascade — model commits to a confidently wrong opening and never recovers. Auto-regressive conditioning: every wrong commit shapes everything that follows (chapter 10.2).
Few-shot Examples are misleading, conflicting, or off-distribution; model faithfully imitates the wrong thing. Induction heads route to the examples regardless of whether the examples are correct. Garbage demonstrations produce garbage continuations.
Chain-of-thought Model commits to a wrong intermediate step; subsequent steps build on the error. The auto-regressive scratchpad amplifies early commits. First-token cascade on the reasoning chain.
Self-consistency The wrong answer is more popular than any individual correct answer; consensus on the misconception. Self-consistency aggregates the modal answer, not the truth. If training data systematically misleads, the mode is wrong, and aggregation reinforces it.
Role priming Persona's style does not match the task; the model writes confidently in the wrong register. Persona shifts the attention-head weighting toward a subspace; if that subspace is wrong for the task, the wrong tokens win.

The key engineering takeaway from this table: no pattern is uniformly safe. Every pattern has tasks where it shines and tasks where it actively hurts. Choosing the pattern is the engineering decision. There is no setting where “always use few-shot CoT with self-consistency” is the right answer — that combination is overkill for routine tasks and the latency and cost would be prohibitive.

16.4 The bridge to the enforcement layer

Notice that all of the failure modes in the table above are catchable by the enforcement layer of chapter 8. Run the code; the hallucinated function name fails on import. Compile-check the JSON; the malformed output is rejected. Hit the API; the wrong endpoint returns 404. Test the logic; the cascade-induced bug surfaces.

This is the production framing. Block B is about making the influence layer carry as much load as possible. The enforcement layer catches what the influence layer misses. The human layer catches what the enforcement layer cannot encode. Better prompting reduces the failure rate from “frequent” to “occasional”; better enforcement reduces it from “occasional” to “caught”; better human review reduces it from “caught” to “addressed.” All three layers work together. None replaces the others.

The next chapter closes the mystery board for Act 2 and sets up Lecture 7’s orchestration story.

17. Mystery Board Closure and the Bridge to Lecture 7

We close two threads in this chapter. The mystery board from Lecture 1 onwards, started before students had any of the vocabulary we have now built. And the forward link to Lecture 7, where the orchestration layer turns the patterns of Block B into infrastructure.

17.1 The mystery board, almost fully resolved

A version-controlled accounting of the mysteries posed across Act 1 and the early Act 2 lectures, with their resolution status as of the end of Lecture 6.

# Mystery Status after L6
1 Why can't AI count letters? Solved (L4). Tokenisation hides characters inside multi-character tokens.
2 Why does CLAUDE.md improve results in some sessions and not others? Solved (L5 §16.1 + L6 §6). Token injection via the orchestration layer + attention with context drift + multi-head dilution. The full orchestration mechanics are in Lecture 7.
3 Why were English prompts often better than German? Solved (L4 + L5). Token efficiency in tokenisation: German prompts spend more tokens for the same content, pushing relevant signal into more, longer, more spread-out positions that compete more and land less reliably (the same positional/competition effects as §7), not a fixed "budget" being drained.
4 Why was curated context better than pasting everything? Solved (L5). Irrelevant context adds distractors and competes for the heads' routing without sharpening the next-token decision; curated context concentrates the relevant signal. (Course-level shorthand: "tokens compete for attention" — the precise mechanism is per-head, per-position softmax competition, not a single shared budget.)
5 Why did the AI hallucinate API endpoints? Solved (L6 §4, §5). Pattern completion with no truth check — a fluent but ungrounded continuation, flat or confidently wrong. Factual hallucination.
6 Why did vague prompts create plausible nonsense? Solved (L6 §4). A vague prompt under-specifies the task, so the model fills the gaps from learned priors — sometimes as a flat distribution with no clear winner, sometimes confidently committing to a plausible-but-wrong continuation. The pipeline always emits a token, and nothing checks it against the truth.
7 Why is CLAUDE.md compliance inconsistent? Solved (L6 §6, §7). Attention competition between CLAUDE.md tokens and task tokens, plus positional effects (lost-in-the-middle).
8 Why do different tools behave differently from the same instructions? Solved (L6 §3). [system] vs [user] placement and orchestration choices. Post-training establishes the hierarchy; Lecture 7 covers how the orchestration layer assigns tokens to each bucket.
9 Why does few-shot prompting work? Solved (L6 §11). In-context learning via induction heads. Format transfers more than content.
10 Why does "Let's think step by step" help reasoning tasks? Solved (L6 §13). Chain-of-thought uses the auto-regressive loop as a scratchpad, trading depth for length.
11 Why does sampling many times and voting improve accuracy? Solved (L6 §14). Self-consistency: variance reduction over the answer distribution by marginalising over reasoning paths.

Ten solved. One — mystery 8 — partly solved, awaiting Lecture 7. Every “why does X behave the way it does?” question that Act 1 raised has a mechanistic answer now, expressible in last week’s vocabulary. The framing of Act 2 was: learn the mechanism, and the surprises stop being surprising. That promise has been kept.

17.2 What Lecture 7 will add

Lecture 7 closes Act 2 with the orchestration layer — the system outside the model that decides what tokens the model sees, when, and in what order.

The transition from “understand the mechanism” to “engineer with the mechanism” is the entire arc from Lecture 4 through Lecture 7. By the end of Lecture 7 you will have both halves: the why (Lectures 4–6) and the how (Lecture 7’s orchestration layer). Act 3 takes the engineering further into production: validation, agentic workflows, real codebases, and the capstone.

18. Lab and Deliverables

The exercise slots for Lecture 6 are extensive — four exercises covering temperature, hallucination taxonomy, context-vs-hallucination, and a prompting-strategy lab — and they live in the exercise sessions rather than inside the lecture. The deliverables for the lecture itself, alongside the exercise outputs, are documented below.

18.1 Exercises (separate slots)

You will work through four exercises in the exercise sessions following the lecture. Each is described in detail in the corresponding exercise page; the headline expectations are:

  1. Temperature Experiments. Run the same prompt at \(T = 0\), \(0.7\), \(1.0\), \(1.5\) on a code task, a creative task, and a factual task. Build a comparison table with a “best T for this task type” conclusion. Connect back to chapter 10’s decision tree.
  2. Hallucination Taxonomy. Return to your Act 1 work. Find 3–5 cases where the AI generated incorrect output. Classify each as factual, structural, or confident-nonsense (chapter 5). Identify the missing context that would have prevented each. Rate your confidence before and after classification.
  3. Context vs Hallucination, extended. Pick one Act 1 task that hallucinated. Run it four ways: zero-shot raw context, zero-shot curated context, few-shot with two examples, and zero-shot CoT. Measure hallucination rate, token cost, and time-to-correct-answer. Compute the per-pattern ROI.

  4. Prompt Strategy Diagnosis. Twelve real-world software-development prompt cards (code generation, code review, debugging, test generation, requirements clarification, external-API usage, structured-output extraction, architecture decisions, security middleware, agentic repo tasks, and over-applied chain-of-thought). For each card you classify the prompting strategy, identify the textual evidence, name the likely failure modes, decide what cannot be handled inside the prompt at all, and rewrite a stronger version. Operationalises both chapter 8’s three-layer model and chapter 16’s pattern library on tasks you will actually face. Two formats:

    • Interactive self-study quiz — 12 multi-select questions; you classify the prompting strategies per card before the instructor key reveals. Firebase-tracked progress with GitHub login.
    • Full exercise & workshop guide — all 12 cards with the complete analysis (evidence, failure modes, what lives outside the prompt, improved prompt, teaching point) as raw readable text. Also includes the instructor scaffolding for running the cards as a class: duration options (25 min / 60 min / 90 min lab), group activity formats, rubric, grading scale, and debrief questions.

18.2 Reflection log

In addition to the exercise outputs, write a reflection log capturing your lecture-level understanding. The expected sections:

  1. The two halves, in your own words. A paragraph each on Block A and Block B. What is the core claim of each block? How do they connect? If a colleague asked you “why does AI sometimes lie and what do I do about it?” — what would you say after this lecture?

  2. The pattern library, in your own words. A short decision tree of your own making for when you would use which prompting pattern. Do not copy chapter 16.2 — adapt it to your specific software-engineering practice. What patterns will you actually use this week, and which will you keep in reserve?

  3. The CLAUDE.md redesign. If you have an active CLAUDE.md on a real project, do a quick audit through the lens of chapters 6 and 7. Where are the rules positioned? Are they front-loaded? Are any buried in the middle, the worst place? Note one concrete change you will make (or have made) as a result.

  4. The most counterintuitive finding. Across Block A and Block B, which result most surprised you? The randomly-labelled few-shot examples (Min et al.)? The depth-vs-length tradeoff of CoT? The first-token cascade at \(T = 0\)? The induction-head story? Write a paragraph on why it surprised you and what it changes about your mental model.

  5. Forward link. One question you want Lecture 7 to answer. The orchestration layer is the topic; what specifically about how the patterns of this lecture become infrastructure do you want to see?

18.3 Optional reading

For students who want to read the originals — all are open-access and the figures alone are worth the time. Grouped by topic. Every entry links to the canonical open-access version, normally arXiv.

Instruction tuning and post-training (chapter 3):

Prompting and in-context learning (chapters 11–14):

Theoretical results on CoT expressivity (chapter 13.2, mentioned in passing):

Industry documentation (chapters 6, 10, 13.7, 15):

These are not academic papers; they are the vendor docs and standards that the lecture cites for current operational guidance. Treat them as primary sources for “how to actually call these models in 2026.”

Inference and serving (chapters 6.5, 15.1):

Security and adversarial robustness (chapter 15.5):

These are the references you need. Anything else on prompting that does not cite one of these — or that contradicts them — should be read with the scepticism Block B was designed to give you.

The lecture ends here. The pattern library is closed. Lecture 7 takes everything we have built and zooms out one more time: from the model, to the system around the model. See you in a week.

© 2026 Dominik Müller   •  Powered by Soopr   •  Theme  Moonwalk