08 Measuring the Machine: LLM Evaluation for Code

June 2026 (47758 Words, 266 Minutes)

1. Introduction

Welcome to Week 8, and to Act 3. The course so far is two completed movements. In Act 1 you experienced the shift: you used the tools on real tasks, kept the diary, and watched the same assistant be brilliant on Monday and useless on Wednesday without being able to say why. In Act 2 — Lectures 4 through 7 — you understood it: tokenization (Lecture 4), the transformer forward pass (Lecture 5), the steered distribution and the prompting patterns that steer it (Lecture 6), and finally Lecture 7’s orchestration layer, the system outside the model that decides which tokens the model sees at all. The Mystery Board that opened in Act 1 closed in Lecture 7 with every “why does this happen?” question answered in mechanistic vocabulary.

Act 3 asks a different kind of question. Not why does the system behave this way? — you know that now — but how do we engineer with it? And engineering, in any discipline that deserves the name, begins with measurement. You cannot improve what you cannot measure. More immediately: you cannot choose between models, tools, and vendors without trusting numbers that somebody else measured, under conditions you did not control and mostly cannot see. Every tool decision you will ever make in this field starts with a number someone else measured. Act 3 starts by teaching you what those numbers are made of.

The starting point is already on the record. Lecture 6’s closing chapter promised it explicitly: after the orchestration layer, Act 3 begins, “and the first stop is SWE-Bench.” Today we keep that promise — though not in the way a product tour would. We are going to take the most-cited benchmark in AI-assisted software engineering apart, piece by piece, and reassemble it with all of its flaws visible.

Here is the scene the whole lecture is built around. It is June 2026. A frontier lab publishes a launch post for its newest model, and the headline benchmark number reads “~95% on SWE-bench Verified.” As of June 2026, two frontier systems are reported at essentially this number — roughly 95% and 95.5% — both measured by the vendors themselves, on their own infrastructure, with their own tooling. The launch post does not dwell on any of that. It presents the number the way a spec sheet presents a clock speed: as a property of the model.

Now make it concrete. Your course project is Locust, a real open-source load-testing tool with a real issue tracker, real legacy decisions, and real tests. Suppose your team takes the launch post at face value and adopts the model for the project. The implied promise of “95% on real GitHub issues” is roughly this: hand the system a closed Locust issue — just the issue text and the repository — and nineteen times out of twenty it produces a correct fix. Would you bet €50 on that? Write your answer down now, before the lecture tells you anything. We will collect these predictions, and we will come back to them — the gap between what you expect and what you find is where the teaching happens.

And to be clear about the spirit of the exercise: the bet is not a setup for a cynical reveal. You will spend the rest of the course earning the right to answer it properly — the homework in chapter 14 takes the first step of a thread in which the class builds a small benchmark of its own out of closed Locust issues, and in the course finale you will audit that benchmark exactly as ruthlessly as we audit SWE-bench today.

If your instinct says no, of course it won’t transfer — good instinct. But press yourself on why not, and notice how quickly the vocabulary runs out. Ninety-five percent of what, exactly — which tasks, chosen how? Measured with the model alone, or with the model wrapped in retries, verifiers, and best-of-N selection? Graded by what — and would a wrong answer have passed the grader? Were the tasks public on GitHub before the model’s training cutoff? None of these are rhetorical questions. Each one has a precise, researched answer for SWE-bench specifically, and the answers are the content of this lecture.

Part of the difficulty is not even specific to SWE-bench. Chapter 3 treats it properly; the preview is three sentences. Model outputs are sampled, so a single run is one draw from a distribution and every score is a statistical estimate — usually published without error bars. Scores swing by whole points on harness details as mundane as prompt formatting and answer extraction. And “can code” is not one thing: writing a function from a docstring, fixing a bug across a 100,000-line repository, and working autonomously for an afternoon are different constructs that happen to share a verb.

That gives you the structural pay-off, the sentence to carry from here to the exam: a benchmark score is a measurement of a system — model + scaffold + grader + data — under specific, gameable conditions; by the end of this lecture you can decompose any score into those parts and know which questions to ask before believing it. Nothing in that sentence is cynicism. It is the same move you already know from measurement in any engineering field: a number is meaningless without its measurement procedure, and the procedure has parts.

The four parts deserve names now, because every chapter of this lecture is about one or more of them:

The lecture builds this decomposition in four moves. Chapters 3–6 build the measurement toolkit on the small, clean world of function-level benchmarks: why evaluating a stochastic system is hard at all (chapter 3), HumanEval and MBPP as the founding template (chapter 4), and then chapter 5 — pass@k in depth, deliberately the most mathematical chapter of Act 3. Fair warning and reassurance in one breath: the mathematics is an estimator-bias argument built from binomial coefficients, not new transformer machinery, and it exists in this lecture because underdefined “Pass@1” claims are appearing in real manuscripts right now. Chapter 6 closes the toolkit with EvalPlus and the lesson that a score is only as trustworthy as its grader.

Chapters 7–10 take the toolkit to SWE-bench: how the benchmark is constructed (chapter 7), the family of variants — each of which quietly fixes a flaw in the original (chapter 8), the leaderboard’s climb from under 2% in the original paper to vendor-reported ~95% as of June 2026, in under three years (chapter 9), and the scaffold problem (chapter 10) — the reason the same models, on the same benchmark, in the same month, can post two leaderboard numbers nearly twenty points apart, both of them correct.

Chapters 11–13 are the critical literature and what we distill from it: the SWE-Bench+ manual audit that re-examined “resolved” instances by hand (chapter 11), the contamination and reward-hacking evidence (chapter 12), and a nine-question checklist (chapter 13) — the one-page artifact you keep, and the bridge to Lecture 9, where the question shifts from is the number real? to what does the capability cost? Chapter 14 is the lab: you will hold one real SWE-bench instance in your hands, dissect a real leaderboard artifact, and start the Locust thread that runs through the rest of the course. In the room, this is again two 90-minute blocks: Block A runs through the leaderboard curve’s cliffhanger at the end of chapter 9; Block B opens with its resolution.

Nothing in this lecture requires new machinery — it leans entirely on vocabulary you already own. Sampling and temperature from Lecture 5 are why a metric like pass@k exists at all. Few-shot, chain-of-thought, and self-consistency from Lecture 6 will reappear, frozen into software, as the anatomy of a scaffold. And Lecture 7’s orchestration layer is precisely the thing a leaderboard entry wraps around the model before anyone measures anything.

The course’s running diagram grows a ring today as well: LLM-in-One-Picture reaches v1.4, with an evaluation-harness box drawn around the orchestration layer of v1.3. The geometry is the argument. A benchmark is not a probe that touches the model directly; it is a system built to measure another system from the outside, and everything between the two — scaffold, grader, data — leaves fingerprints on the number.

One cultural note before we start, in the same spirit as Lecture 6’s warning about prompt folklore. The discourse around benchmarks has two well-worn failure modes:

• Benchmark worship. Take the headline number at face value, rank the models by it, pick the winner, move on. This quietly treats a measurement of one system under specific conditions as a property of a model under all conditions — a category error you will be able to name precisely by chapter 10.
• Benchmark nihilism. “All benchmarks are contaminated and gamed, the numbers mean nothing, go by vibes.” This sounds like sophistication and is exactly as lazy — it replaces analysis with a different way of not looking.

We will not do benchmark worship, and we will not do benchmark nihilism — both are ways of not thinking. A benchmark is an instrument. Instruments can be miscalibrated, misread, and gamed, and still be the best tool available — if you know exactly what they measure and where they fail. This lecture is the calibration manual.

There is one more piece of infrastructure to set up before the toolkit, and it is the piece that keeps this lecture honest for the rest of the semester. Act 2 had the Mystery Board. Act 3 gets its successor.

2. The Claim Board

This chapter exists because Act 3 needs what Act 2 had: a continuity device that accumulates the course’s open questions in public and refuses to let them quietly disappear. The Mystery Board carried Act 2. It collected “why does this happen?” questions from Act 1 — why can’t AI count letters, why does CLAUDE.md sometimes not work — and it closed in Lecture 7 with every mystery resolved in mechanistic vocabulary. Those answers stay answered: tokenization will still hide characters inside tokens next year, and attention will still dilute. Mechanism, once understood, does not expire.

Act 3’s board has the opposite polarity. The Claim Board asks not “why does this happen?” but “should I believe this?” — and unlike the Mystery Board’s answers, its verdicts come with expiry dates. A claim about a model is a claim about a moving market: the number may be superseded, the model retired, the benchmark itself withdrawn. Every rating we issue carries an “as of” stamp, and part of what the board teaches is precisely that the stamp matters.

2.1 The mechanic

Today, real claims go up — quoted from actual launch posts, leaderboards, and headlines, uncommented. Before anything is taught about any of them, you write down your gut answer to the same question you answered in chapter 1: what does this number actually claim, and would you bet €50 that it holds for your Locust work? The predictions are collected. This is the predict–observe–explain pattern the course has used since Act 1: the surprise, when it comes, is the teaching moment, and it only works if your prediction is on paper first.

Over the coming weeks, each claim gets run through the rubric — the nine questions of chapter 13 — and rated by the class. A claim only comes down rated. No claim is allowed to fade away unexamined, and no claim is allowed to be dismissed by eye-roll either; “obviously marketing” is not a rating, it is an excuse not to do the analysis. Lecture 9 will add a second wave of claims about pricing and “cheapest model” assertions, and the final lecture of the course makes the last pass: which claims survived? The board lives in the same dual format as the Mystery Board did — the physical board in the seminar room and a digital page in the course repository that records each claim, its source, its date, and eventually its verdict.

2.2 The first five claims

Here is what goes up today. All five are real, current, and quoted as found, as of June 2026 — and some of them will already be stale by the exam, which is itself part of the lesson.

Notice that none of these claims is an obvious lie, and none is self-evidently true. That is exactly why they are on the board. Invented strawman claims would teach you to spot strawmen; these teach you to interrogate the real thing.

2.3 The rating rubric

Each claim ends in one of three verdicts, assigned by the class using the nine questions of chapter 13 as the rubric:

• Verified — the claim survives the nine questions: the measurement conditions are disclosed, the number has been independently reproduced or is checkable in principle, and it means what the headline implies.
• Plausible-but-unproven — nothing refutes it, but the disclosures needed to check it (scaffold, attempt count, sampling settings, error bars, subset) are missing. The claim may well be true; it is currently unfalsifiable as published.
• Refuted-as-stated — the number may even be real, but the claim as phrased fails: it means something materially different from what the headline implies, or it does not survive contact with the evidence.

Expect the middle verdict to do most of the work. The uncomfortable, realistic finding of this lecture is not that vendors fabricate numbers — it is that the conditions that give a number its meaning are routinely omitted, and an unomitted version would often look considerably less impressive.

You cannot apply the rubric yet, because the rubric is built from everything between here and chapter 13. But here is the commitment this lecture makes: by the end of today you will already be able to rate two of these — the headline Verified claim and the “best coding model” slogan, numbers 1 and 2: the first because the next eleven chapters take that benchmark apart down to the individual test case, the second because chapter 3 alone dismantles it. The place to start is the awkward fact underneath all five claims: measuring an LLM is hard in ways that measuring a sorting algorithm is not. That is chapter 3.

3. Why Evaluating LLMs Is Hard

The Claim Board (chapter 2) is now full of numbers that someone, somewhere, measured. Before you can rate a single one of them, you need to understand why producing an honest benchmark number is hard in the first place — not hard because vendors cheat (some of that comes later), but hard for reasons built into the system being measured. This chapter exists to name those reasons. There are three, and none of them is sloppiness: the output you are grading is a random sample; the program doing the grading shapes the score; and “can code” is not one measurable thing. Every benchmark you will meet in this lecture is somebody’s attempt to engineer around one or more of these three problems, so you need them in hand before the history starts.

Each of the three maps directly onto machinery you already own from Lectures 4 through 7. Nothing in this chapter is new mechanism — it is the old mechanism, viewed from the perspective of someone trying to attach a number to it.

The three problems also give you the lecture’s table of contents in miniature. The sampling problem is answered by a metric (chapter 5). The harness problem returns as the grader question (chapter 6), the scaffold question (chapter 10), and the audit (chapter 11). The construct problem is the entire reason SWE-bench exists (chapter 7). Keep the triple in view and the next eleven chapters organize themselves.

3.1 The output is a sample, not a property

Recall the sampling step from Lecture 5: the forward pass does not end in an answer, it ends in a probability distribution over the next token, and a sampler draws from that distribution. Lecture 6’s temperature chapter turned that draw into an engineering parameter you set per task. The consequence for measurement is immediate: the same model, the same prompt, the same benchmark task can produce a passing solution on the first run and a failing one on the second. Nothing changed except the dice.

So what does the sentence “the model solved task 17” actually mean? Formally, very little. For a fixed task, a fixed prompt, and fixed sampling settings, the model has some per-task success probability \(p\) — the fraction of all possible completions, weighted by their probability under the distribution, that would pass the tests. A single benchmark run does not reveal \(p\). It reveals one draw from a Bernoulli distribution parameterized by it:

A benchmark run does not observe capability; it samples it. The quantity you care about — how reliably this model solves this kind of task — is \(p\), averaged over tasks. The quantity one run hands you is a single weighted coin flip per task. “Did it pass” is a sample, not a property, and confusing the two is the original sin of benchmark reading.

Two consequences follow, both named here and both quantified properly in chapter 5. First, single-run scores carry sampling noise, and that noise is large enough to matter: two models reported a few points apart can be statistically indistinguishable — yet error bars are rarely published at all. Second, the score depends on the sampling configuration. Temperature, top-p, and the number of attempts are all part of the experiment, and the Lecture 6 lesson that there is no universal right temperature has a measurement-side mirror: a score reported without its sampling settings is an underdefined quantity. Even pinning everything to \(T = 0\) does not dissolve the problem — it just measures the model in one particular configuration, and chapter 5 will show that the choice of temperature and the choice of metric are coupled in a way most reported numbers silently ignore.

You have already brushed against this problem with your own hands. Lecture 6’s lab asked you to measure hallucination rate across prompting strategies — and if you ran any condition twice, you got two different rates. That was not an error in your lab work. That was the Bernoulli structure above, experienced before it had a name. This lecture turns that ad-hoc instinct into a discipline.

3.2 The harness is part of the experiment

Now suppose you fix the sampling configuration completely. The score is still not a property of the model alone, because between the model and the reported number sits a piece of software — the evaluation harness — that formats the prompt, calls the model, extracts an answer from whatever free text comes back, and grades it. Every one of those steps involves choices, and the choices move the score:

• Prompt format. Whether the task is phrased as a raw completion or wrapped in a chat template; whether few-shot examples are included, how many, and in what layout. You know from Lecture 6’s few-shot chapter that format transfer is real leverage on the output distribution — which means it is also real leverage on the measured score.
• Answer extraction. The model replies in prose, with the answer somewhere inside it. The harness must locate that answer — a regex, a code-block parser, a “final answer:” convention. Brittle extraction grades correct answers as wrong; generous extraction grades wrong answers as right.
• Execution environment. For code benchmarks the grader runs code: dependency versions, timeouts, and container configuration all sit between a generated solution and its verdict.

None of this is hypothetical. EleutherAI’s evaluation team — the people who maintain one of the field’s most widely used open evaluation harnesses — report in “Lessons from the Trenches on Reproducible Evaluation of Language Models” that scores swing by whole points on prompt formatting and answer-extraction choices alone. Same model, same tasks, different harness: different number.

What makes harness bugs uniquely dangerous is that they fail silently. A bug in production software crashes, throws, or corrupts data — something downstream complains. A bug in an evaluation harness just shifts a number, and the shifted number looks exactly like an unshifted one. There is no stack trace for “your extraction regex rejected fifty correct answers”; there is only a score that is quietly three points too low, published, cited, and pinned to a Claim Board.

Every benchmark number is the output of a program, and programs have bugs. It helps to see the harness for what it is: an orchestration layer in exactly the sense of Lecture 7 — software that decides which tokens the model sees and what happens to the tokens that come back — with a grader bolted on the end. A benchmark score is therefore a property of the model–harness pair, not of the model. Hold onto that framing, because the rest of the lecture escalates it three times: chapter 6 shows what happens when the grader is too weak, chapter 10 shows what happens when the orchestration layer around the model becomes the dominant term, and chapter 11 shows what a manual audit finds when it checks the grader’s verdicts by hand.

3.3 “Can code” is not one construct

The third problem has a name from psychometrics: construct validity — does the test measure the thing the claim is about? A vendor claim says “Model X can code.” But “can code” decomposes, on even casual inspection, into distinct skills:

These are genuinely different skills, and a measurement of one tells you remarkably little about another. The cleanest demonstration sits right at the seam between this lecture’s two founding benchmarks: by 2023, frontier models were scoring near 90% on function synthesis while the best system resolved under 2% of real repository issues — chapter 7 puts exact names and dates on that gap. Same models, same year, two constructs, a 45-fold difference in apparent ability. No benchmark measures “coding ability”; each benchmark measures one operationalization of it, and rankings do not transfer between constructs. A model that tops the function-synthesis tables can sit mid-pack on repository tasks, and a vendor is free to quote whichever construct flatters the product.

For the Claim Board, this gives you the first concrete habit: before you rate a claim, annotate it with its construct. “Best coding model in the world” backed by a function-synthesis score and “best coding model in the world” backed by a repository-issue score are two different claims wearing the same headline — and your Locust project, which is repository work through and through, cares about exactly one of them.

This is also where a question you have already heard gets its formal treatment. Lecture 6’s introduction asked: when a paper claims a new prompting technique improves accuracy by 12 points, what is it manipulating? Back then the answer was mechanistic — it is reshaping the output distribution. Now look at the same claim through this chapter’s lens. A 12-point improvement is a difference between two measurements, and each side of that difference inherits all three problems above: is 12 points larger than the sampling noise (§3.1)? Was the same harness, the same extraction code, the same sampling configuration used on both sides (§3.2)? And 12 points on which construct (§3.3)? Evaluation is where every claim about model capability either becomes a measurement or stays marketing — and the difference between the two is whether these three questions have answers.

None of this is a counsel of despair. The three problems are better read as a design space: every benchmark in the rest of this lecture is an engineering response to one or more of them, and every flaw we will dissect is a place where the response fell short. They are also the skeleton of the artifact you will carry out of this lecture: the nine-question checklist of chapter 13 is, at heart, these three problems unfolded into a form you can apply to a vendor table in five minutes.

3.4 The anatomy of an evaluation run

The three problems are easier to hold — and the eleven chapters ahead easier to navigate — once you can see the machine they are problems with. Strip every benchmark in this lecture down to its plumbing and the same six stages appear, in the same order, turning a dataset into a headline number:

Data → Prompt → Sample → Extract → Execute → Aggregate.

• Data — load the task instances: 164 functions, 500 repository issues, whatever the benchmark ships. One HumanEval instance is a single self-contained task like the common_prefix_length function the next chapter walks through; one SWE-bench instance is a real bug report bundled with the entire repository it lives in. Where they came from, and whether the model has already seen them, is the data question that chapter 4 opens at function level, chapter 7 at repository level, and chapter 12 turns into the contamination crisis.
• Prompt — assemble each instance into the model’s input. For a function that input is literally the signature and docstring with nothing added; for a repository issue it is the bug report plus whichever files the harness retrieves plus an example of the answer format. The prompt, and the loop that may wrap it, are the scaffold — opened in chapter 7 and dissected in chapter 10 — and already a score-moving choice, as the harness section warned.
• Sample — draw one or more completions at a fixed configuration: temperature, top-p, and the number of samples \(n\). At its simplest this is a single greedy draw — the one function body the model writes below the docstring. Each such draw is the Bernoulli event this chapter opened with; turning a pile of draws into an honest score is the entire job of chapter 5.
• Extract — parse a usable answer out of the model’s free text. The model rarely emits exactly the answer and nothing else, so you trim the completion down to one function body, or pull the unified diff out of an agent’s surrounding chatter. This is the brittle, silent step that the harness section flagged as answer extraction.
• Execute — run the extracted answer against the grader, in a sandbox, because you are about to run code a machine wrote and nobody read. For common_prefix_length that means appending its hidden asserts and running the file; for a repository patch, applying the diff and running the project’s own test suite in a container. How far to trust that grader’s verdict is chapter 6, chapter 11, and chapter 12.
• Aggregate — collapse the per-task pass/fail verdicts into one number. One task passes or fails; do that across all 164 (or 500) instances and the fraction becomes pass@k for function benchmarks, % resolved for repository ones.

The same six stages run in two very different bodies, and the rest of the lecture is mostly a tour of the difference. At the function level the whole pipeline is essentially one forward pass — the prompt fits in the context, the model emits a body, the tests run, done — which is HumanEval, and the next chapter walks one all the way through. At the repository level the Sample and Execute stages swell into a loop with tools: an agent reads files, edits, runs tests, reads the failures, and edits again for dozens of turns before any patch exists — which is SWE-bench, walked through at repository scale. Two shapes of the same machine; what changes between them is not the six stages but how much machinery hides inside Sample and Execute.

Read the rest of the lecture, then, as a tour of this pipeline: every remaining chapter zooms into one or more of these boxes.

So we start where the field started — with the simplest possible design, built in 2021, that answered the third problem by picking the smallest construct on the table and answered the first with a metric you will meet in chapter 5.

4. The First Generation: HumanEval and Function-Level Benchmarks

This chapter exists because every benchmark on the Claim Board descends from a single template, and you cannot understand what the later benchmarks fixed — or what their critics exposed — without first seeing the original design and why, in 2021, it was the right one. The template has a birthplace: the Codex paper (2021), which introduced both the first widely adopted code benchmark and the metric that still dominates reporting today.

4.1 HumanEval: 164 functions and a unit-test grader

HumanEval is a set of 164 hand-written Python function-completion problems. Each problem gives the model a function signature and a docstring; the model writes the body; a suite of unit tests decides pass or fail. That is the entire design. No repository, no issue tracker, no conversation — one self-contained function per task, graded by execution.

To make the shape concrete, here is a schematic task in the HumanEval mould (written for this lecture, not taken from the dataset):

def common_prefix_length(a: str, b: str) -> int:
    """Return the length of the longest common
    prefix of the strings a and b."""
    # ← the model writes everything below this line

and the grader is nothing more than assertions executed against whatever comes back:

assert common_prefix_length("flag", "flake") == 3
assert common_prefix_length("", "anything") == 0
assert common_prefix_length("same", "same") == 4

The prompt is the signature plus the docstring; the completion is the body; the verdict is binary. Notice how much this design quietly assumes: the specification is complete and unambiguous, the function needs no context beyond itself, and the handful of assertions captures everything “correct” means. Each of those assumptions becomes a named problem later in the lecture.

And look at the real thing, not just this schematic: all 164 problems are public on the Hugging Face Hub at openai/openai_humaneval — open the dataset viewer to read each problem’s prompt, canonical solution, and tests in your browser, or pull them in three lines with datasets.load_dataset("openai/openai_humaneval"). The original execution harness — the code that actually computes pass@k — is openai/human-eval.

Two design choices deserve your attention, because both were deliberate. First, the problems were hand-written — composed fresh for the benchmark rather than scraped from GitHub — precisely so that they could not already sit in the training corpus. That is a quiet, early acknowledgement of the contamination problem that chapter 12 will escalate into a crisis. Second, correctness is decided by running tests, not by comparing the generated text against a reference solution. Prose has no executable ground truth; code does, and HumanEval was built on the insight that you should use it. A generated function that looks nothing like the reference but passes every test is correct, full stop.

MBPP — roughly 974 crowd-sourced, entry-level Python problems — followed the same shape at larger scale and lower difficulty: short specification in, function out, tests decide — and it is equally browsable on the Hub at google-research-datasets/mbpp (the full config is all 974 problems, sanitized the 427 hand-checked ones). Between them, these two datasets defined what “can code” meant, in numbers, for the first years of the LLM era.

4.2 One HumanEval run, end to end

You have seen the task and where to read it (§4.1); now watch one model produce one answer to one task, byte by byte, before we let the count run to all 164. We will keep the run as simple as a run can be — one attempt per task, with the randomness switched off — and follow the same common_prefix_length example you just saw through each step of §3.4’s pipeline.

The prompt is not something you write — it is a field in the dataset. This is the first thing to slow down on, because it is nothing like chatting with an assistant. The instance’s prompt field is the model’s input, verbatim: the signature and docstring, and not one character more.

def common_prefix_length(a: str, b: str) -> int:
    """Return the length of the longest common
    prefix of the strings a and b."""

That whole string is fed to the model — no “please implement the following”, no list of rules, just an unfinished Python file. Why does handing over an unfinished file work as a way to ask for code? Because of what Lecture 5 showed a base model is: a next-token predictor trained on enormous amounts of source code. A signature followed by a docstring is one of the most familiar shapes in that training data, and the single most probable continuation of it is the body. The benchmark never asks for the body; it builds a context whose natural completion is the body and lets the model do what it always does.

The model continues the file — and, left alone, it would not stop. Generation runs one token at a time, each new token appended and fed back in. The model was never told to write one function; from its side it is completing a file, and the most natural thing after a finished function is the next one. So the raw output rolls straight past the body we wanted:

    n = min(len(a), len(b))
    for i in range(n):
        if a[i] != b[i]:
            return i
    return n
# ─── first "\ndef" reached → stop here; keep everything above ───
def levenshtein(a, b):              # the model rolled on into the next
    """Edit distance between a and b."""   # function in the file — none
    ...                             # of it part of the task; discarded

Everything below the cut is noise for this task. So the decoder is given stop sequences — \ndef, \nclass, \nif __name__ — and halts the instant it generates a newline that begins a new top-level construct: everything above is kept, everything below is thrown away. This happens while sampling, in your generation code, before any test runs — the Extract step of §3.4 folded into the draw. What survives is written as one line of samples.jsonl, the completion only — the body, with no signature and no docstring repeated back:

{"task_id": "HumanEval/0", "completion": "    n = min(len(a), len(b))\n    for i in range(n):\n        if a[i] != b[i]:\n            return i\n    return n"}

Set those stop markers badly — halt too early, or forget to halt — and a perfectly capable model loses points for reasons that have nothing to do with coding: your first hands-on instance of §3.2’s warning that the harness shapes the score as surely as the model does.

The walkthrough so far assumed a base model continuing raw text. An instruction-tuned model — the kind you actually chat with — was fine-tuned to answer turns, not to continue stubs, so handed the bare signature it tends to answer conversationally: “Sure! Here’s the function:”, with the code wrapped in a Markdown fence and perhaps an explanation around it. To evaluate it you must instead wrap that same prompt field in an instruction — “Complete this function; return only the code” — and then peel the markdown fence back off before grading. Same task, identical weights, two different harnesses — and because the wrapper reshapes the input distribution exactly as Lecture 6’s format lesson predicts, the two harnesses can report two different scores for one model. There is no neutral wrapper, only a disclosed one.

Two words for the setting we just used: greedy and pass@1. We drew one completion per task with the randomness off; both halves earn a name the rest of the lecture relies on. “Randomness off” is greedy decoding: rather than sampling from the next-token distribution, you always take its single most probable token — equivalently, temperature set to zero. One input, one deterministic output, identical on every rerun. “One completion per task, scored on whether it passed” is pass@1: over the 164 tasks, the fraction whose single greedy completion passes. That is the entire metric today — count the passes, divide by 164 — and notice that no statistics were needed, because nothing was random enough to average. That is also exactly why chapter 5 exists: turn the randomness back on and allow several attempts per task, and “how often does it pass” stops being a fraction you count and becomes a quantity you must estimate — the Codex paper’s pass@k, drawn at a higher temperature (its convention is top_p ≈ 0.95 with \(T\) up near 0.8). Greedy pass@1 is the version where the method is still fully visible.

The grader does not judge the code — it runs it. This is the step most worth understanding rather than invoking, because the grader is dumber, and therefore far more trustworthy, than students expect. It never inspects the completion to decide whether it “looks right”. It does something purely mechanical: it concatenates four pieces, in this exact order — the original prompt, your stored completion, the instance’s hidden test (which defines a check function), and one line the harness appends to call check on your function — and executes the result as an ordinary Python program:

# 1) prompt  +  2) your stored completion  →  one complete function
def common_prefix_length(a: str, b: str) -> int:
    """Return the length of the longest common
    prefix of the strings a and b."""
    n = min(len(a), len(b))
    for i in range(n):
        if a[i] != b[i]:
            return i
    return n

# 3) the instance's hidden test, which defines check(candidate)
def check(candidate):
    assert candidate("flag", "flake") == 3
    assert candidate("", "anything") == 0
    assert candidate("same", "same") == 4

# 4) one line the harness appends, to fire the tests
check(common_prefix_length)

Notice what the grader does not do: it never parses, reformats, or trims your completion — it pastes those bytes in verbatim. Had your stored completion still carried the discarded def levenshtein from a moment ago, it would have been concatenated in too, and could have flipped the result. That is the concrete reason the truncation back in the generation step was your job: by the time the grader runs, it is too late to clean anything up.

The verdict is simply what Python does with that program. Run to the end with nothing raised and the task passes; a false assert raises AssertionError and it fails; any other exception, or a loop that runs past a wall-clock timeout, fails too. The whole oracle is that one rule — pass = the assembled program ran check without raising. Why define correctness by execution instead of by comparing the model’s text to the canonical solution? For the reason §4.1 gave: code has an executable ground truth that prose lacks. A body that looks nothing like the reference but satisfies every assertion is correct, full stop; one that reads beautifully but returns the wrong value on ("same", "same") is wrong. Binary, automatic, unarguable — the property a benchmark needs, and the very property chapter 6 will show is worth only as much as the assertions someone troubled to write.

In practice you generate once and grade twice — against two different test suites. The two halves live in separate programs on purpose. First your code calls the model on all 164 tasks and writes a samples.jsonl file — one JSON line per task, each carrying the task_id and the model’s completion. Then a second program reads that file and performs the assemble-and-execute grading above on every line. But why grade twice at all, when grading is deterministic — the same tests on the same completion always return the same verdict, so a literal re-run would tell you nothing new? Because the second pass is not a repeat: it swaps in a stronger oracle. The expensive, stochastic half (calling the model) is paid once and frozen into the file, so you can throw a tougher test suite at the very same completions for almost nothing — the original HumanEval tests print one number, the ~80×-amplified HumanEval+ tests print a lower one, and the gap between them — same model, same completions, only the grader changed — is precisely the experiment chapter 6 runs.

# 1. generation: write samples.jsonl — one {"task_id", "completion"} line per task (shown above)

# 2a. original grader (openai/human-eval) — assemble + execute the few hand-written tests
evaluate_functional_correctness samples.jsonl

# 2b. stronger grader (evalplus/evalplus) — same completions, ~80x more tests (HumanEval+)
evalplus.evaluate --dataset humaneval --samples samples.jsonl

Two graders, one frozen samples file, two different pass@1 numbers — and the size of that gap is the whole lesson of chapter 6. One operational detail is not paranoia but a direct consequence of how the grader works: it runs code the model wrote and nobody reviewed, so it must run inside a sandbox. openai/human-eval even ships with its execution call commented out, refusing to run until you uncomment it and accept the risk — the same untrusted-code reasoning that makes SWE-bench’s grader a sealed container in §7.4.

Step back and you have watched one HumanEval number take shape, end to end: a dataset field becomes the prompt, the model continues it into one greedy body, a stop sequence trims it to a single function, the grader bolts on the hidden tests and runs the whole thing in a box, and 164 green-or-red verdicts collapse into one percentage. Every refinement still to come is an attack on one stage of this run — chapter 5 on the single greedy draw, chapter 6 on the handful of tests — and the rest of this chapter asks what the run, even performed flawlessly, still cannot tell you.

4.3 The founding template

Strip the two benchmarks to their shared skeleton and you get the template that everything in this lecture either inherits or reacts against. The founding template is: function-level, static, test-verified.

• Function-level. The unit of work is one self-contained function. The whole problem fits in one prompt, the whole answer in one completion. In the vocabulary of §3.3: the template selects the smallest construct on the table — function synthesis — and measures exactly that.
• Static. A fixed problem set, published once, identical for every model and every year. That buys comparability — any two models, any two years, same yardstick — at a price that will come due in chapter 12: a frozen, public test is a learnable one.
• Test-verified. Correctness is execution. No human raters, no similarity scores — run the tests and read off green or red. Objective, cheap, infinitely repeatable. It also moves the entire burden of defining “correct” onto the test suite, a transfer of trust whose consequences are chapter 6’s whole subject.

For 2021 this template was exactly right. The open question of that moment was whether a language model could reliably write a correct standalone function at all, and a small, clean, executable benchmark answered it with a number instead of a demo. The template was honest about its construct, cheap enough for anyone to run, and grounded in execution rather than judgment. The field adopted it instantly, and for good reason.

4.4 Saturation

Then the models improved, and the template’s smallness became its weakness. By 2023, frontier models were scoring near 90% on HumanEval (and above it with test-time scaffolding such as Reflexion; base models broadly crossed 90% in 2024). When every frontier model passes nearly every task, the benchmark stops ranking anything: the remaining differences between systems are a handful of tasks — well inside the sampling noise of §3.1 — and the headline numbers compress into a cluster at the top of the scale.

Do the granularity arithmetic and the problem gets sharper. On a 164-problem set, a single problem is worth about 0.6 points of score. Two frontier models reported “a point apart” therefore differ on roughly two problems out of 164 — and §3.1 told you that each of those problem-level verdicts is itself a Bernoulli draw. At the saturated end of the scale, the benchmark’s resolution and its noise floor are the same size.

Be precise about what saturation means, because the sloppy reading is on half the Claim Board. A saturated benchmark does not tell you the models are equally good; it tells you the test has run out of headroom. Above 90%, HumanEval can no longer distinguish a merely competent model from an exceptional one, any more than a primary-school arithmetic test can rank mathematics students. The measurement instrument, not the measured population, has hit its ceiling — and a number quoted from a saturated benchmark is a fact about the benchmark.

One footnote that matters for how you read function-level numbers today: the template did not simply retire at saturation — it learned to defend itself. LiveCodeBench (Jain et al., 2024) keeps the function-level construct but continuously draws fresh problems from competitive-programming contests published after each model’s training cutoff, so a score cannot be memorized; companions such as BigCodeBench and ClassEval push the construct toward harder, more realistic tasks. This is exactly chapter 12’s freshness diagnostic applied one level down — and a reminder that “function-level” is a live battleground, not a closed chapter of 2021.

4.5 The questions the template leaves open

Saturation is why the field moved on, but even at its peak the template left two questions open inside its own frame — and a third it could not pose at all.

1. How do you score a stochastic system? Each completion is one Bernoulli draw (§3.1), so “passed” needs to become an estimate of the underlying success probability — and it turns out the obvious way to compute that estimate is subtly, systematically wrong. The Codex paper’s own answer, the pass@k estimator, is the subject of chapter 5.
2. Are the tests strong enough to be the grader? Test-verification transferred all trust to the test suite, and for years nobody audited the suite. When someone finally did, the >90% scores deflated on contact — that is the EvalPlus story, chapter 6.
3. And the question the template cannot answer from inside: is completing a function from its docstring the same skill as fixing a real bug in a real repository? Construct validity (§3.3) says you cannot extrapolate from one to the other; the only way to find out is to build a benchmark around the second construct and look. The field did exactly that, and the result — and the shock of its first numbers — is chapter 7.

We take the questions in order. Before leaving the function level, chapter 5 stays with HumanEval one more time and does the statistics properly — because the metric it produced is quoted in every vendor announcement you will ever read, and almost always quoted incompletely.

5. pass@k: Scoring a Stochastic Programmer

Chapter 4 left you with a template — function-level, static, test-verified — and a headline number attached to it. This chapter slows down and asks what that number actually is. The answer is not “the fraction of problems the model solves.” The answer is: an estimate of a probability, produced by an estimator, from a finite number of random draws — and every benchmark score you will read for the rest of this lecture, including every vendor claim pinned to the Claim Board, inherits the bias, the variance, and the reporting obligations that come with that. Most published scores are reported without any of them. So before SWE-bench, one chapter of honest counting. It is deliberately the most mathematical chapter of the lecture, because underdefined “Pass@1” claims are not a hypothetical — they are showing up in real manuscripts right now, and §5.7 dissects one. The mathematics is all standard probability — expectation, variance, the binomial, convexity, Jensen’s inequality — and this chapter names each only briefly as it arrives; if any of that is rusty, Appendix B is a from-the-ground-up refresher written for exactly this chapter, to read first or keep open alongside.

5.1 Why a metric beyond “did it work once”

Start from what you established in Lecture 5: the model’s output is sampled. The forward pass produces a distribution over the next token; the sampler draws from it; the draw feeds back in; repeat. Run the same HumanEval prompt twice at non-zero temperature and you get two different function bodies — sometimes both correct, sometimes neither, often one of each.

That single fact breaks the question you instinctively want to ask. “Can model M solve task t?” is mis-typed: it presumes a boolean where the mechanism gives you a probability. The honest version is “with what probability does one sampled attempt at task t pass the tests?” — call that per-task probability \(p\). A single run is a single Bernoulli draw from it. Chapter 3 flagged this as the first reason evaluating LLMs is hard; pass@k is the field’s formalization of the fix.

You might object that greedy decoding rescues the boolean: set the temperature to zero and the same prompt gives the same completion, so “did it work once” becomes well-defined again. Two answers. First, Lecture 6 already showed you that the production version of \(T = 0\) is only mostly deterministic — hosted inference stacks drift across replicas, batches, and silent model updates, so even the greedy score is a draw from a (narrow) distribution. Second, and more fundamentally: evaluating only the greedy completion measures one privileged point of the output distribution and discards everything Lecture 5 taught you the model actually is. A model whose second- and third-ranked completions are also correct is more useful than one where only the argmax path works — and a metric built on single deterministic runs cannot see the difference. The probabilistic frame is not a concession to noise; it is the correct type for the object being measured.

Definition. pass@k is the probability that at least one of \(k\) independent samples for a task passes the tests, averaged over the tasks of the benchmark. pass@1 asks: how often does the model get it right in one try? pass@100 asks: if you let it try a hundred times and something checks the answers, how often is at least one try right?

The reason \(k\) is a parameter and not a fixed convention is that different values describe different deployments. A developer who generates a candidate function, runs the unit tests, and regenerates on failure is operating in a pass@k world — the tests are the selector, and a lucky third attempt is as good as a perfect first one. An unattended completion in an editor, accepted without review, is a pass@1 world. The metric family is honest about the fact that “how good is this model at coding?” has no single answer; it depends on how many tries you get and who checks them.

This is not an abstraction you can defer to exam week. When you later attempt your chosen Locust issue with an agent — the course-project thread that starts in chapter 14 — you will have to decide whether “the agent solved it” means on the first attempt or within five attempts with the test suite as referee. That decision is choosing \(k\), and it changes the number you write down. What remains is the statistical question this chapter exists for: you do not know \(p\) — you have to estimate it from samples, and the obvious way to do that is wrong.

5.2 The naive estimator and its bias

For a single task with true per-sample success probability \(p\), the quantity we want is:

— one minus the probability that all \(k\) independent attempts fail. The problem is that \(p\) is unknown. The obvious move is to estimate it: generate \(n\) samples, count the \(c\) that pass, set \(\hat{p} = c/n\), and plug in:

This is the naive plug-in estimator, and for \(k > 1\) it is biased — not noisy-but-fair, but systematically off in the same direction on every task.

The intuition is Jensen’s inequality, and it is worth seeing both as a picture and — just below — as three lines of algebra. \(\hat{p}\) itself is a fair estimate: across repeated experiments it averages out to \(p\). But you are not reporting \(\hat{p}\) — you are reporting a curved function of it. The all-fail term \((1-\hat{p})^k\) is convex in \(\hat{p}\) for \(k>1\), and a convex curve lies above its chords: when a noisy estimate wiggles symmetrically around the truth, the curve exaggerates the wiggles on one side more than it dampens them on the other, and the average of the outputs lands above the output at the average. So the estimated all-fail probability is systematically too large — which means the naive estimator systematically understates pass@k. Noise in, bias out: the curvature converts honest variance into a one-directional error.

Here is the same claim as algebra — the version the lecture slides put up for anyone who prefers the proof to the picture. Take the all-fail term \(f(x) = (1-x)^k\), convex for \(k > 1\), and let the expectation \(\mathbb{E}[\cdot]\) run over repeated draws of the \(n\)-sample experiment, so that \(\mathbb{E}[\hat{p}] = p\). Jensen’s inequality — for a convex \(f\), \(\mathbb{E}[f(X)] \ge f(\mathbb{E}[X])\) — does the rest in three lines:

The only load-bearing step is the first inequality; the other two are arithmetic. And the size of the bias is precisely the gap Jensen leaves open, \(\mathbb{E}[f(\hat{p})] - f(\mathbb{E}[\hat{p}])\) — which, to leading order, is \(\tfrac{1}{2}\,f’‘(p)\operatorname{Var}(\hat{p})\): the variance of \(\hat{p}\), bent upward by the curvature \(f’’\), re-emerging as a one-directional error. For \(k = 2\) the curvature is constant (\(f’’ = 2\)) and this leading-order term becomes exact — the \(p(1-p)/n\) deficit the next paragraph pins down by name.

Two properties follow directly from that picture. The bias grows with \(k\), because larger exponents mean more curvature for the noise to rectify. And it shrinks with \(n\), because more samples mean less noise to rectify — for \(k=2\) you can compute it exactly: the naive estimator is low by precisely \(\operatorname{Var}(\hat{p}) = p(1-p)/n\), the variance of your per-task estimate leaking into the average as a systematic deficit. At \(k=1\) the function is linear, Jensen has nothing to grab, and the plug-in is innocent. The disease starts at \(k=2\), and no amount of averaging over tasks cures it — bias, unlike variance, points the same way on every task and survives the mean.

If the Jensen argument feels too slippery to trust, you can make the bias visible in a two-minute simulation, and the homework notebook has you do exactly that. Fix a true \(p\) — say 0.3 — simulate thousands of tasks by drawing \(c\) from a binomial with \(n = 10\), and compute both estimators — the naive one above and the unbiased one you are about to meet in §5.3 — on every simulated task. Plot the two histograms against the true pass@2 of \(1 - 0.7^2 = 0.51\). The unbiased estimator’s histogram is noisy but centered on the line; the naive estimator’s histogram is just as noisy and shifted off it. No theorem required — the bias is a visible gap between a histogram’s center of mass and a vertical line, and watching it refuse to close as you add more tasks (while it does close as you raise \(n\)) teaches the variance-versus-bias distinction better than any derivation.

5.3 The unbiased estimator

The fix ships in the same paper that introduced HumanEval — the Codex paper (Chen et al., 2021). Generate \(n \geq k\) samples per task, count the \(c\) correct ones, and compute:

First, read that \(\mathbb{E}_{\text{tasks}}\) correctly, because it is a different beast from the expectation in §5.2 and the two are easy to conflate. It is not a probabilistic expectation over randomness — it is the plain average over the benchmark’s tasks. For one task, with its own observed count \(c\) of passing samples, the bracketed quantity \(1 - \binom{n-c}{k}/\binom{n}{k}\) is fully deterministic; you compute it once per task from that task’s \(c\), then average the \(N\) results:

— which is all that \(\mathbb{E}_{\text{tasks}}[\cdot]\) abbreviates. So your instinct is right: given the data, the right-hand side is deterministic. The randomness — and with it the word unbiased — lives one level down, in where each \(c\) comes from: \(c\) is itself a random draw from \(\text{Binomial}(n, p)\), and “unbiased” is the promise that if you re-ran the whole \(n\)-sample experiment many times, this per-task formula would average, over those random \(c\)’s, to the task’s true \(1 - (1-p)^k\). Two averages, then, easy to mix up: the \(\mathbb{E}_{\text{tasks}}\) printed in the formula runs over tasks (and is just \(\tfrac{1}{N}\sum_i\)); the expectation hiding inside the word “unbiased” runs over repeated sampling of one task. §5.2’s bias and this section’s fix both live at that second level — per task — and only afterwards get averaged across the benchmark.

The formula looks opaque until you read the fraction as a counting statement. \(\binom{n}{k}\) is the number of ways to choose \(k\) samples out of your \(n\); \(\binom{n-c}{k}\) is the number of ways to choose \(k\) samples entirely from the \(n-c\) failures. The fraction is the probability that a uniformly random \(k\)-subset of your \(n\) samples contains zero correct solutions; one minus it is “at least one of \(k\) tries passes.” Instead of estimating \(p\) and pushing it through a curve, you ask the pass@k question directly of the samples you have — every \(k\)-subset of your \(n\) attempts is a simulated \(k\)-try experiment, and the formula averages over all of them at once. That is also why it draws without replacement where the naive estimator implicitly draws with replacement: your \(k\) real attempts would be \(k\) distinct samples, and treating them that way is exactly what removes the bias.

Worked example. Say \(n = 10\) samples on one task, \(c = 3\) pass, and you want pass@2. There are \(\binom{10}{2} = 45\) possible pairs of samples, of which \(\binom{7}{2} = 21\) consist purely of failures:

The naive estimator on the identical data gives:

Same ten samples, more than two percentage points apart — and the direction is the one §5.2 predicted, with the plug-in sitting low. The gap grows with \(k\) and shrinks with \(n\); at the small \(n\) that real evaluations actually use (samples cost money — Lecture 9’s subject), it is material. On leaderboards where ranks turn on fractions of a point, the choice of estimator is already a ranking decision.

The protocol that comes with the formula matters as much as the formula. You must generate \(n \geq k\) samples — ideally \(n\) comfortably larger than \(k\), since the estimator’s own noise shrinks as \(n\) grows past \(k\) — and you must generate them as independent draws under the same sampling configuration. Reusing samples across different reported \(k\) values is fine (that is the design); quietly mixing samples from different temperatures or prompts is not, because then \(c/n\) no longer estimates any single per-task probability.

One honesty note before moving on: “unbiased” means right on average over repeated experiments, not right on any given one. On this particular draw the estimator may overshoot or undershoot the true pass@2. Unbiasedness removes the systematic error; the random error remains — which is the next section’s subject.

One more thing, because it is the natural question to ask here: which estimator, and which \(k\), do real benchmarks actually use? The estimator above is not a teaching abstraction — it is the one the standard tooling runs. openai/human-eval ships it as the function estimate_pass_at_k, and EvalPlus reuses the same routine, so any pass@k for \(k > 1\) reported by those harnesses is this unbiased one by construction, never the naive plug-in of §5.2. But a quiet fact reframes how much that correction matters in practice: almost every headline number you will read is pass@1 — and at \(k = 1\) the formula collapses to plain \(c/n\), which is linear, so §5.2’s bias vanishes outright. HumanEval, HumanEval+, and MBPP leaderboards report pass@1; SWE-bench’s % resolved is pass@1 in spirit (one scored attempt per instance). The unbiased estimator therefore earns its keep not on the leaderboards but in the research regime that produced it: the Codex paper drew \(n \approx 200\) samples per task and reported pass@1, pass@10, and pass@100 together — the classic triple — and recent repeated-sampling work pushes \(k\) far higher. “Large Language Monkeys” (Brown et al., 2024) takes one fixed model from 15.9% at a single sample to 56% at 250 samples on SWE-bench Lite, with this coverage climbing log-linearly across four orders of magnitude of \(k\) — the rising pass@k curve §5.6 draws, and exactly the regime where quoting \(n\) beside \(k\) stops being optional. And that estimator is not confined to scaling-up-samples work: the reasoning-RL literature now leans on pass@k at large \(k\) — same formula, a few hundred samples per task — as a diagnostic of whether training widens or narrows a model’s reach (the base-versus-RL crossover §5.6 unpacks). Both are the regime where §5.2’s plug-in bias bites hardest — large \(k\), finite \(n\) — so the unbiased form is not a nicety there but a requirement. And to answer the question your instinct raised directly: yes, the metric is alive in HumanEval+ — EvalPlus changed the grader (chapter 6), not the metric, so its leaderboard still reports pass@1, now against the amplified tests.

5.4 Error bars are not optional

A benchmark score is an average of per-task pass/fail outcomes, and pass/fail outcomes are Bernoulli draws. That alone fixes how uncertain the average is. For a benchmark of \(N\) tasks evaluated with one sample each, at a true pass rate \(p\), the standard error of the reported score is:

Put numbers in. At 60 tasks and one sample per task — a configuration you will meet in a real manuscript in §5.7 — the standard error at a pass rate near 50 %, where the variance is largest, reaches:

And when you compare two models on the same 60 tasks, the gap between their scores carries the noise of both measurements. For independent scores the standard error of the difference is \(\sqrt{2}\) times larger than either alone:

That \(\sqrt{2}\) assumes the two scores are independent. They are not — both models were graded on the same tasks, so an easy task lifts both scores and a hard one depresses both, leaving the per-task outcomes positively correlated. The honest standard error of the difference is therefore smaller than \(\sqrt{2}\,\text{SE}\); treat \(\sqrt{2}\,\text{SE}\) as a conservative upper bound, and reach for a paired test — McNemar, or a bootstrap that resamples tasks — when a ranking actually hangs on the result.

Two models reported five points apart on such a benchmark can be statistically indistinguishable: the gap is barely half a standard error of the difference. Even the full 164-task HumanEval carries a standard error of about four points at mid-range pass rates; a 500-instance benchmark still carries about two. The error shrinks only with the square root of the task count, and you cannot sample your way around a small task suite — extra samples per task tighten the per-task estimate, but the across-task component is fixed by \(N\).

Against that arithmetic, the field’s practice is stark: most benchmarks publish no statistical significance analysis at all — that is a documented finding of the BetterBench audit of benchmark practices, not a rhetorical exaggeration. Leaderboards render point estimates as ranked lists, and reorderings within the noise band are reported as news. The remedies are neither exotic nor expensive, which makes their absence more damning than the noise itself:

1. Report the denominator. Task count and samples per task, always — the reader can compute a worst-case standard error from \(N\) alone, but only if \(N\) is stated rather than buried.
2. Run more than once and show the spread. Multiple evaluation runs, or a bootstrap over tasks, turn a point into an interval at a cost that is trivial next to the inference bill.
3. Test differences, not scores. The claim a leaderboard actually makes is “A beats B”; that is a claim about a difference, and the \(\sqrt{2}\) factor means it needs roughly twice the evidence a single score does.

When chapter 13 hands you the nine-question checklist, “Variance? — multiple runs? error bars?” is question seven, and you now own the two-line calculation that answers it yourself when the authors decline to: square root of \(p(1-p)\) over \(N\), times at most \(\sqrt{2}\) if you are comparing (less, since the two scores share tasks).

5.5 Temperature and pass@k

There is a quieter way a pass@k number can be underdefined, and it connects straight back to Lecture 6’s temperature chapter: the metric and the sampling configuration are coupled, and the coupling has a direction.

pass@1 favors greedy or low-temperature decoding. With one attempt, you want the model to commit to its modal completion — the single most probable answer is, by definition, the best one-shot bet, and any flattening of the distribution only trades it away. But pass@k for large \(k\) flips the logic. Low temperature makes your \(k\) samples nearly identical — \(k\) tries collapse into one try, repeated. Higher temperature spreads the draws across plausible-but-different completions; many will be wrong, but the metric only requires one to be right, and the unit tests act as the selector that finds it. Diversity is wasted at \(k=1\) and is the entire point at \(k=100\). The Codex paper measured exactly this: the optimal sampling temperature grows with \(k\) — the configuration that maximizes pass@1 is not the configuration that maximizes pass@100. You have seen this logic before in Lecture 6’s self-consistency story: sampling diversity pays precisely when some mechanism — there a majority vote, here a test suite — selects among the draws.

The practical consequence is a reporting obligation. A reported Pass@1 without its sampling settings is an underdefined quantity. Picture two labs evaluating the same model on the same tasks. Lab A decodes greedily, one run per task, and reports the fraction that passed as “Pass@1”. Lab B samples \(n\) completions per task at a moderate temperature and reports the unbiased \(k=1\) estimate — which is just \(c/n\) averaged over tasks. Both numbers are legitimately called Pass@1. They are also different quantities: Lab A measured the quality of the argmax path; Lab B measured the average mass the distribution puts on correct completions. Depending on how the model’s probability is distributed, either can come out higher, and a third paper citing both side by side is comparing experiments, not models.

Temperature, top-p, and the number of samples \(n\) are therefore not implementation trivia to be left out of the methods section — they are part of the metric’s definition, exactly as the estimator choice of §5.3 is. The benchmark name tells you the tasks; the metric name tells you the formula; neither tells you the experiment.

5.6 The mirror metric: pass^k

pass@k has a mirror image, and the contrast between them is one of the most useful framing devices in this lecture. pass^k — read “pass to the \(k\)”, introduced with the τ-bench agent benchmark — is the probability that all \(k\) independent runs of the same task succeed. In terms of the per-task success probability, the two metrics bracket a single attempt from opposite sides:

Feel the asymmetry with one hypothetical number. A system that succeeds 90 % of the time per run scores a pass@3 of \(1 - 0.1^3 = 0.999\) — essentially perfect — while its pass^3 is \(0.9^3 = 0.729\): more than a quarter of all three-run sequences contain a failure. Same \(p\), same system; one metric says “deploy it”, the other says “it breaks weekly.” pass@k measures what a model can do for you when something checks its work; pass^k measures what it can be trusted to do when nothing does. pass@k rewards a lucky draw; pass^k punishes flakiness — and for a production agent that files PRs or runs your nightly pipeline unattended, pass^k is usually the number you actually care about, because nobody is standing by to pick the good run out of \(k\). Hold onto this mirror: the closing lecture of this course, where the orchestration layer becomes an agent, and the follow-up agentic-AI course both live on the pass^k side of it.

This is not a museum piece you have to take on faith — you can watch the mirror move on today’s models. The τ-bench family of agent benchmarks (Sierra and Princeton), actively updated into 2026, keeps a live leaderboard at taubench.com for today’s frontier systems and is built on exactly this consistency metric — so the gap between a model’s one-shot success and its reliability several runs deep is precisely what it exists to expose. The selector side has migrated the other way: pass@k for \(k > 1\) has largely left the leaderboards — which now headline pass@1 — for the research literature, where it is busier than ever, in two regimes. The repeated-sampling studies of §5.3 plot it as a coverage curve over \(k\) to chart inference-time scaling; and the reasoning-RL literature has turned pass@k at large \(k\) into a diagnostic rather than a capability headline. Yue et al. (2025) use it to ask whether reinforcement learning teaches a model anything its base version could not already do, and report a striking crossover: an RL-trained model wins at pass@1 yet is overtaken by its own base model at pass@256 — the training sharpens one high-reward path at the cost of the sampling diversity that large-\(k\) coverage rewards, the same “exploration pays when a selector exists” logic §5.5 drew for temperature. The pass@k you just learned to read is, on the current frontier, doing more work as a research instrument than it ever did as a leaderboard column. And for a middle case, Aider’s polyglot leaderboard gives each model two attempts with the tests as feedback and reports the two-try pass rate — a live pass@2 in spirit. Open one of these beside this section and the abstraction turns into a column you can sort.

A final structural observation that the table already hints at: since pass@k rises toward one and pass^k falls toward zero as \(k\) grows, any point estimate at a single \(k\) is a slice through a curve, and the slice can be chosen. Both curves are computable from the same pool of \(n\) samples per task — the data collection of §5.3 already paid for them — so an honest evaluation can simply show pass@k for several \(k\) alongside pass^k for the deployment-relevant ones. When a report instead shows you exactly one \(k\), it is worth a moment’s suspicion about why that one: a generous \(k\) on the pass@k curve is the single cheapest way to make a mediocre per-attempt probability look like a strong system.

Step back, because the metric toolkit is now complete and the next section spends it on a real claim. Four things to carry out of this chapter:

1. A score is an estimate, not a reading. Every pass/fail is one Bernoulli draw (§5.1); the headline is a statistic computed from a finite sample, with the bias and variance that implies.
2. The naive estimator is biased low for \(k > 1\) — use the unbiased one. Plug-in pushes a fair estimate through a curved function and Jensen does the rest (§5.2); the combinatorial estimator (§5.3) asks the pass@k question of the samples directly, and it is what the standard tooling computes.
3. Error bars are not optional. \(\text{SE} = \sqrt{p(1-p)/N}\) (§5.4) — roughly 6.5 points at 60 tasks — and a model-to-model gap carries up to \(\sqrt{2}\) times that, so many reported rankings sit inside the noise.
4. The metric is coupled to the experiment. Temperature, top-p, the sample count \(n\), the value of \(k\), and even pass@k versus pass^k (§5.5, §5.6) all move the number; the metric name pins the formula, never the experiment.

Those four are the metric toolkit.

5.7 Reading a Pass@1 claim in the wild

Everything in this chapter compresses into a skill you can use this semester, so let us use it. What follows is an anonymized, real manuscript from 2026 that crossed your instructor’s desk for review — not a strawman constructed for the slide.

The study evaluates several models and reports “Pass@1” on a 60-task subset of HumanEval+ (the amplified-test variant of HumanEval you will meet properly in chapter 6). The paper states no temperature, no top-p, no number of samples per task, and no confidence intervals. It then ranks the models by a headline efficiency metric, “cost per correct solution” — total API cost divided by the number of correct solutions, a ratio whose denominator is exactly that unqualified pass rate.

Four questions, in the order a reviewer should ask them — each one is a section of this chapter:

1. Which estimator? Is “Pass@1” one greedy run per task (\(c/n\) at \(n=1\)), or estimated from \(n > 1\) samples — and if the latter, with the plug-in of §5.2 or the unbiased estimator of §5.3? The paper does not say, so the reader cannot know what quantity the table contains.
2. What sampling configuration? Per §5.5, a Pass@1 without temperature and sampling settings is underdefined — and comparing several models without fixing the configuration may be comparing decoding setups as much as models.
3. What is the standard error at 60 tasks? Per §5.4: up to about 6.5 points per score, and up to about nine on a difference (a conservative bound, since the models share tasks). Which of the paper’s reported model-to-model gaps survive an error bar? Any ranking inside the noise band is typography, not evidence.
4. What happens to the ratio as Pass@1 approaches zero? “Cost per correct solution” divides by a small, noisy number. As the pass rate of a weak model heads toward zero the ratio diverges, and its sampling noise diverges with it — the ranking among the cheap, weak models, which is the paper’s selling point, is exactly where the metric is least stable. A denominator with a ±6.5-point standard error can move the ratio by tens of percent without any model changing.

Notice what was not required to produce these questions: no access to the authors’ code, no rerunning of anything, no expertise beyond this chapter. Evaluation literacy is a peer-review skill, not just a consumer skill — the same four questions that rate a vendor claim on the Claim Board also referee a manuscript. They form the statistical quarter of the nine-question checklist that chapter 13 assembles in full, and the homework notebook in chapter 14 lets you watch all of it happen on simulated tasks: the plug-in estimator drifting away from a known ground truth, bootstrap confidence intervals widening as the task count shrinks, and the optimal temperature climbing as \(k\) grows.

One boundary of this chapter, named honestly: everything here concerned the statistics of counting — how to turn pass/fail draws into a trustworthy probability. We never once questioned the pass/fail itself. A sample “passed” whenever the unit tests said so, and the tests played the role of an infallible oracle. They are not one. The next chapter asks the question this one carefully avoided: who decides what counts as correct — and what happens to the scores when the grader gets stronger?

6. The Test-Oracle Lesson: EvalPlus

This chapter exists because chapter 5 has a blind spot, and the blind spot is load-bearing. Chapter 5 spent all of its energy on the estimator — how to turn \(n\) sampled completions into an honest success probability without fooling yourself. But every pass@k computation starts from a per-sample verdict, this completion passed, this one failed, and that verdict is produced by a test suite the estimator never questions. The estimator can be perfectly unbiased and the published number still wrong, because pass@k does not estimate the probability that the code is correct — it estimates the probability that the code passes the tests. Those two quantities coincide only when the tests are strong enough to tell correct code from incorrect code. In 2023, for the most-cited code benchmark in the field, they were not.

Software testing has an old name for the component that decides pass or fail: the test oracle. In your own projects the oracle is your test suite, and you already know its central property from experience: a green bar does not mean the code is correct; it means the code is correct as far as the tests can see. A suite that exercises only the happy path waves through code that crashes on the empty list, mishandles the boundary value, or quietly returns the wrong thing for inputs nobody wrote a case for. Nothing in the binary pass/fail signal distinguishes “correct” from “wrong in a way I never asked about”.

HumanEval’s oracle, as chapter 4 described, is a small set of hand-written test cases per problem — written once, by humans, trying to pin down each function’s intent with a few representative inputs. For separating the models of 2021 that was good enough. But as scores climbed into the high 80s, a question became unavoidable: how much of that number is code that is actually correct, and how much is code that merely passes a handful of friendly checks?

The EvalPlus paper answered the question by doing the one thing nobody had bothered to do: strengthen the grader and re-grade everything. But “strengthen the grader” hides a real obstacle, and seeing how EvalPlus clears it is the mechanical heart of this chapter. To add a test you need not just a new input but the correct answer for it — and producing thousands of correct answers by hand is precisely the labour the original authors could not afford. EvalPlus’s unlock sidesteps that entirely: every HumanEval problem already ships a known-correct reference solution, so the reference itself can serve as the answer key. Generate any input you like, run it through the reference to get the expected output, then run it through the candidate completion; if the two outputs disagree, the candidate is wrong. This is differential testing, and it quietly converts the expensive task “write more tests” into the cheap one “generate more inputs” — because the oracle for every input comes free, from code that is already correct. The amplified benchmark this builds is HumanEval+.

Make it concrete on §4’s common_prefix_length. Picture a completion that returns the length of its first argument whenever the comparison loop finds no mismatch, instead of the shorter of the two lengths — a bug that nonetheless survives all three original asserts. EvalPlus takes the seed input ("same", "same") and applies a string mutation, dropping a character from one argument to get ("same", "sam"). The reference solution answers 3; the buggy completion answers 4; the outputs disagree, and the completion fails — on a test no human wrote, in an input region (unequal-length strings where one is a prefix of the other) that the three hand-written asserts never touched. That one caught case is a single droplet of the 12.2% that evaporated.

With the oracle problem dissolved, the rest is an input-generation pipeline — and it runs on exactly the kind of fixed templates you would expect, four steps from seed to scored test:

1. LLM-generated seeds. ChatGPT is shown the reference solution and a few example inputs and asked for interesting ones — roughly 30 corner-case seeds per problem (empty collections, boundary values, the cases a careful human tester would reach for first).
2. Type-aware mutation. Each seed is then mutated recursively, by type, against a fixed operator table — integers stepped by one and pushed toward extremes, booleans flipped, strings with substrings removed / repeated / replaced, lists with elements removed / repeated / inserted, tuples and sets rebuilt from a mutated list, dicts with pairs dropped or values updated. This is the step the field loosely calls “fuzzing”, and it is worth being precise: mutation is the fuzzer here — there is no separate fuzzing technique, only this type-respecting mutation flooding the input space far past anything written by hand.
3. Validate, then grade by reference. Each candidate input is first checked against the problem’s pre-conditions (so a routine that requires a non-empty list is never handed []), then settled by the differential oracle above — the reference computes the expected output, no human in the loop.
4. Reduce. Mutation overproduces wildly, so a final greedy set-cover keeps only the inputs that earn their place — by covering a new branch, by killing a mutant (a deliberately corrupted copy of the reference), or by killing some real model’s sample. That reduction is why HumanEval+ settles near 80× the original tests (about 9.6 → 764 per problem) rather than a useless million; the whole pipeline ships in the evalplus package.

Then the field’s models were re-graded. Same models, same completions, same estimator — only the oracle changed. The headline movements:

Read the table slowly. Nothing about any model changed between the two columns — no weights, no prompts, no sampling settings, no estimator. The only thing that changed is how carefully the answers were checked, and a substantial slice of GPT-4’s measured competence evaporated. Every completion that passed HumanEval but failed HumanEval+ was always wrong; the original oracle simply could not see it. The 88.4% was never a measurement of correctness. It was a measurement of correctness plus an invisible second term.

That second term deserves to be written down, because it is the structural lesson of this chapter. For a single task, the success probability a benchmark actually measures decomposes as

— the probability of being right, plus the probability of being wrong in a way the oracle cannot detect. The second term is invisible from inside the benchmark. No amount of resampling, no unbiased estimator, no confidence interval from chapter 5 will surface it, because every statistical tool you own operates downstream of the oracle’s verdict. EvalPlus measured the term the only way it can be measured: build a stronger oracle and take the difference.

That second term has an older name in software testing: it is the behaviour the tests fail to cover. The standard measures are statement (C0) and branch (C1) coverage — did the suite even execute the branch where a wrong answer could hide? — and the sharper one is the mutation score: inject deliberate bugs into a correct solution and count how many the tests kill. A suite can reach 100% branch coverage and still pass a mutant, because running a line is not the same as asserting on its result — which is why mutation score is the more honest answer to “would a wrong answer pass?” Read this way, EvalPlus’s ~80× amplification is nothing exotic: it drives the oracle’s effective coverage and mutation-kill rate up until the previously-undetectable wrong completions finally fail, and the size of the score drop is the size of the gap that weak coverage had been hiding.

You do not have to take this re-grading on trust: the amplified suites are public on the Hub (evalplus/humanevalplus), and EvalPlus ships as a pip-installable package (evalplus/evalplus) you can run on your own generations. Its public leaderboard, tellingly, effectively froze around 2024 — not because the tool broke but because HumanEval+ itself saturated and the field moved on; for current models the live successor is the contamination-resistant LiveCodeBench from §4.4.

So pin the sentence to the board, because it is the single most reusable result of today: a benchmark score is only as trustworthy as its grader. And keep it pinned, because it returns twice before this lecture ends — once at repository scale, when chapter 11’s manual audit finds the same weak-test problem inside SWE-bench’s “resolved” instances, and once more in chapter 12, when automatically strengthened tests reshuffle a public leaderboard.

You already have a frame to hang this on. Lecture 6’s three-layer model put tests in the enforcement layer: the deterministic machinery that refuses bad output, where prompting can only make bad output less likely. A benchmark is the same architecture pointed in the opposite direction — instead of guarding a production system against the model’s mistakes, the tests guard your beliefs against the model’s lucky passes. Weak enforcement does not just mean undetected failures; it means inflated belief — every wrong-but-passing completion converts directly into score. And the failure is silent in the same direction every time: a too-strict oracle announces itself with implausibly low scores that someone investigates, while a too-weak oracle produces beautiful numbers that nobody is motivated to question.

One coda before we leave the grader, because it widens the lesson past code. HumanEval’s oracle is executable — tests, run to a verdict. But the moment you grade something with no executable ground truth — a chatbot reply, a summary, a code review rather than a code fix — the grader is increasingly another language model, prompted to score the output. An LLM judge is cheap and correlates well enough with human preference that it now grades much of the field, including the blind-vote leaderboard behind Claim Board #3. But it is not an executable oracle; it is a steered distribution with measured failure modes — position bias (it favours the answer it sees first), verbosity bias (it rewards length over correctness), and self-enhancement bias (it prefers its own model family’s outputs). A model grading a model inherits every problem in chapter 3, now on both sides of the measurement — which is exactly why “only as trustworthy as its grader” is the load-bearing sentence.

EvalPlus repaired the grader, but it kept the template: one isolated function, a self-contained docstring specification, an empty file to write into. Chapter 4 already told you why that template was saturating; this chapter told you its scores were inflated even before saturation. The deeper criticism is aimed at the template itself — almost nothing in your working life as a software engineer is “write a fresh function from a clean spec”. The field’s answer to that criticism is the benchmark the rest of this lecture orbits, and it is time to meet it properly.

7. SWE-bench: Evaluation Moves Into the Repository

This chapter is the heart of the lecture’s first half, and the reason is hanging on the wall: the headline claim from chapter 2 — a frontier model resolving roughly 95% of SWE-bench Verified — is a claim about the benchmark you are about to take apart. Every chapter from here to chapter 12 is, one way or another, about SWE-bench: its variants, its leaderboard, its scaffolds, its audits. You cannot rate the claim without knowing exactly how the number is produced. So this chapter slows down and builds the benchmark in front of you, the way Lecture 5 built the transformer: construction, task format, prompt, metric, machinery, and the launch numbers that made it famous.

The motivation follows directly from the two findings you are carrying. Chapter 4 showed the function-level template saturating; chapter 6 showed its grader inflating. Both are symptoms of the same root problem — the construct-validity worry from chapter 3: writing an isolated function from a clean specification is not the job. The job is maintenance — finding the right place in a codebase you did not write, changing it without breaking what already works, starting from a bug report written by a stranger. SWE-bench (Jimenez et al., ICLR 2024) is the benchmark that made that the test.

7.1 Construction: turning merged pull requests into graded exercises

The construction is the elegant part, and it starts from an observation about open source: GitHub already contains millions of records of exactly the work we want to measure. Every merged pull request that fixes an issue is a completed exam question — the issue text is the task, the merged diff is a known-good solution, and if the PR also touched the test suite, the tests it added are a grader. Nobody had to write tasks for SWE-bench; the benchmark is mined from work that real maintainers already did and already graded.

Jimenez et al. collected roughly 90,000 merged PRs from 12 popular Python repositories — django, scikit-learn, matplotlib, sympy, astropy, flask, requests, pytest, sphinx, xarray, pylint, and seaborn — and passed them through three filters:

1. Issue plus tests. Keep a PR only if it resolves a linked issue and changes test files. The linked issue supplies a natural-language task description; the changed tests supply the grader.
2. Reproducibility. The repository at that point in its history must install and run reproducibly — otherwise no harness can ever re-execute the tests on demand, years later, on someone else’s machine.
3. Fail-to-pass. At least one test must fail before the PR’s code change and pass after it. This is the crucial filter: it proves the tests actually encode the fix, rather than being incidental cleanup riding along in the same PR.

What survives all three filters is 2,294 instances — about 2.5% of the mined PRs. Each instance is a frozen moment in a real project’s history: a repository snapshot from just before the fix, the issue text that prompted it, the maintainer’s actual solution (the gold patch, withheld from the model), and the tests that distinguish before from after.

Notice what the filters check — and what they do not. All three are mechanical properties: a link exists, the build reproduces, a test flips. Nothing checks whether the issue text contains enough information to solve the task, and nothing checks whether the tests are fair — whether they accept any correct fix or only the maintainer’s particular one. Hold that observation; it becomes the entire story of the Verified variant in chapter 8.

7.2 The task: issue in, patch out

The task given to a model is brutally close to a junior developer’s Monday morning. You receive the issue text — however vague, however long, written by whoever happened to file it — and the repository snapshot, often more than 100,000 lines of code. You must produce a patch: a unified diff that applies cleanly to the snapshot and fixes the problem. Not a function. Not a file. A diff against a living codebase.

Grading uses two sets of tests drawn from the project’s own suite:

The two sets carry the two halves of what “fixed” means in real maintenance. F2P is the progress signal — the bug is gone. P2P is the do-no-harm signal — everything else still works. A patch that turns the failing test green by deleting the feature it exercises gets caught by P2P; a patch that refactors beautifully but never addresses the issue gets caught by F2P. You will hold one of these instances in your hands during the lab (chapter 14): issue text, gold patch, F2P tests, on paper, dissected in pairs. You do not even have to wait for the lab: the whole benchmark is public. Open SWE-bench/SWE-bench_Verified in the Hugging Face dataset viewer and you can read any instance’s problem_statement (the issue text), its gold patch, the test_patch, and the exact FAIL_TO_PASS / PASS_TO_PASS lists — the very fields this section just named, one row per task. The leaderboards live at swebench.com and the evaluation harness at SWE-bench/SWE-bench.

One property of the input is worth registering now, even though its significance only lands later: the issue text is whatever the reporter wrote — including the discussion thread underneath it. Keep that in mind until chapter 11.

7.3 The original prompt is Lecture 6 material

How does a 2023 language model — no agent loop, no tools, one forward pass over one context — attempt this task? Through a prompt, and the prompt deserves its own subsection because every line of it instantiates something you learned in Lecture 6. The original generation prompt, which you can check against the prompt datasets the authors released, concatenates four ingredients:

1. Task instructions — what a patch is and what format to emit it in.
2. The issue text — the task itself.
3. Retrieved code — the repository does not fit in a 2023 context window, so the harness selects files. Two settings exist: BM25 retrieval, a classical keyword-matching algorithm that guesses which files are relevant to the issue text, or the “oracle” setting, which supplies exactly the files the gold patch actually edited.
4. Exactly one demonstration example of the patch/diff format — a single worked instance showing what a well-formed answer looks like.

Ingredient four makes the original protocol one-shot — and the shot teaches format, not solutions. The demonstration is static: the same fixed example appears regardless of which instance is being attempted, so it cannot leak anything about the task at hand. Its job is to shape the output distribution toward well-formed diffs — exactly the role Lecture 6’s few-shot chapter assigned to demonstrations, and a benchmark-scale application of the Min et al. finding that format transfers more than content. The design is not pedantry: a malformed diff scores zero no matter how correct the reasoning behind it was, and the one demonstration exists to take that failure mode off the table for every model equally.

Ingredient three carries the second Lecture 6 lesson, plus a terminology warning. The “oracle” in oracle retrieval is a different oracle from chapter 6’s test oracle — here it means retrieval with perfect knowledge of where the fix belongs, information no real system would have. The pairing of the two settings is deliberate: BM25 is what a practical system can do, oracle retrieval is the upper bound, and the gap between them shows that retrieval quality alone moves the score — same model, same issue, different context, different result. You know the mechanism from Lecture 5’s attention chapter and watched it play out in Lecture 6’s CLAUDE.md analysis: every token competes for attention mass, and irrelevant files dilute the distribution exactly where precision matters. “Curated context beats pasting everything” was an Act 1 instinct that Act 2 explained mechanistically; the BM25-versus-oracle gap is the same statement as a measured effect.

Hold onto this subsection. When chapter 10 argues that a scaffold is prompting frozen into software, the original SWE-bench prompt is exhibit A: a retrieval policy, a format demonstration, and instruction text — every one of them a Lecture 6 decision, compiled into a harness where nobody thinks of it as prompting anymore.

7.4 The metric and the machinery

The reported metric is % resolved, and its definition is uncompromising. An instance counts as resolved if and only if the model’s patch applies cleanly to the snapshot, every FAIL_TO_PASS test flips to green, and every PASS_TO_PASS test stays green:

Binary, no partial credit. A patch that fixes the bug for nine of ten input shapes scores exactly what an empty patch scores: zero. A patch that is one character away from applying cleanly: zero. That severity is a feature — it mirrors how CI treats your own pull requests — but after chapter 6 you should immediately append the qualifier the formula hides: the conjunction is only as strong as the test sets inside it. Conservative given the tests is not the same as conservative. Mark that thought; it matures in chapter 11.

The machinery that computes the verdict is a fully Dockerized harness — rebuilt around containers in the 2024 rewrite — in which every instance gets its own image pinning the repository snapshot and the exact dependency versions it needs. The model’s patch is applied inside the container, and the tests run there too. The footprint is real: the recommended local setup is an x86_64 machine with Docker, about 120 GB of free disk, 16 GB of RAM, and 8 cores.

Why so heavyweight? Because filter two — installs and runs reproducibly — has to stay true: a 2021 django snapshot needs its 2021 dependency universe, and without per-instance pinning, a score measured in 2024 and a score measured in 2026 would not be comparable. In chapter 6’s vocabulary: the containers are the part of the grader that guarantees the verdict is at least reproducible. Whether the verdict is right is a different question, and it stays open until chapter 11. There is also a second reason the harness is a sealed container rather than a script on your laptop: it runs code the model wrote, untrusted by construction. A generated patch can loop forever, exhaust memory, or reach for the network — so the container is a blast wall as much as a reproducibility device, and that isolation is part of why an evaluation run is never free.

7.5 One SWE-bench run, end to end

You have the metric (§7.4) and the machinery that computes it; now put them in motion the way §4.2 did for HumanEval, and the jump from a function to a repository stops being abstract. The six stages still run in order — but the single forward pass that was HumanEval’s Sample stage swells, here, into an agent working a codebase over many turns, and that one change reorganizes everything around it.

Start with what one instance actually hands the agent. Not a docstring — a bug report and a repository. Here is a representative instance in the SWE-bench mould (the shape, not a specific dataset row); open a real one beside it at SWE-bench/SWE-bench_Verified:

instance_id:        acme__widgets-2417
problem_statement:  format_bytes() crashes on petabyte-scale sizes
                    ---------------------------------------------
                    format_bytes(10**18) should return "1.0 PB" but
                    raises ValueError: number too large. Our storage
                    dashboards now report PB-scale volumes.
repo / base_commit: the acme/widgets repo, frozen at the commit just
                    BEFORE the real fix was merged
FAIL_TO_PASS:       ["tests/test_format.py::test_petabytes"]
PASS_TO_PASS:       ["tests/test_format.py::test_small_sizes", ...]

The model is handed the problem_statement and the repository at base_commit. It is not handed the gold patch (the maintainer’s real fix, withheld), and it is not told which file is broken or which tests will judge it. Everything it needs, it must find. Set that against §4.2, where the prompt was the function: here the prompt is a paragraph of English plus a repository that, per §7.2, often runs past 100,000 lines.

Why this cannot be one forward pass — watch a few turns. No single completion resolves this. The model cannot fit the whole repository in its context, and it cannot know its fix works without running the tests — so the work is a loop, the act-observe-act pattern from Lecture 7, where each observation decides the next action. The benchmark’s own minimal driver, mini-SWE-agent (the ~100-line loop you meet again in chapter 10), installs with pip install mini-swe-agent and drives exactly this, one shell command per turn:

Thought:   the issue names format_bytes — find where it lives
Action:    grep -rn "def format_bytes" src/
Observe:   src/acme/format.py:42

Action:    sed -n '30,70p' src/acme/format.py
Observe:   units = ["B", "KB", "MB", "GB", "TB"]   # the table stops at TB

Action:    sed -i 's/"TB"]/"TB", "PB", "EB"]/' src/acme/format.py
Observe:   (file written)

Action:    python -m pytest tests/test_format.py -k petabytes
Observe:   1 passed

Action:    git diff                                 # captured as the patch

Read what each step is. The grep is fault localization — the repository-navigation skill §3.3 singled out as its own construct, here forced into the open because nothing told the agent where to look. The pytest run is the agent checking its own work before the grader ever sees it; that test result is the feedback that closes the loop and decides the next move. A weaker agent loops longer, edits the wrong file, or never locates the function at all — and because every extra turn is more sampled tokens, this is where the scaffold (how many turns are allowed, which tools, how the context is trimmed) begins to move the score as much as the model does.

What the agent emits is a patch, and the patches are collected into one file. The loop’s output is not prose and not a whole file — it is a unified diff against the snapshot, captured from the agent’s final git diff:

diff --git a/src/acme/format.py b/src/acme/format.py
@@ -39,7 +39,7 @@ def format_bytes(n):
-    units = ["B", "KB", "MB", "GB", "TB"]
+    units = ["B", "KB", "MB", "GB", "TB", "PB", "EB"]

Run in batch over all 500 Verified instances, the agent writes one line per instance into a predictions file — the instance_id, the model that produced it, and that diff as model_patch:

{"instance_id": "acme__widgets-2417", "model_name_or_path": "my-agent", "model_patch": "diff --git a/src/acme/format.py ...\n@@ ... @@\n-    units = [...]\n+    units = [...]\n"}

That file — not the running model — is what gets graded, which is why generation and grading are separable here just as in §4.2: freeze the patches once, then re-grade them against a stronger or fresher test set whenever one appears, without ever paying the agent again.

Now the harness, step by exact step. Hand the predictions file to the official grader:

python -m swebench.harness.run_evaluation \
    --dataset_name SWE-bench/SWE-bench_Verified \
    --predictions_path <your_predictions.jsonl> \
    --run_id my-first-run

For each instance it does four mechanical things, all inside that instance’s pinned Docker image (the reproducibility container from §7.4): (1) build or pull the image — the repo at base_commit with its exact, frozen dependency versions; (2) git apply the model_patch — and if the diff does not apply cleanly, the instance is already a fail and no tests run; (3) run the FAIL_TO_PASS tests with the repo’s own runner (here pytest), then run the PASS_TO_PASS tests; (4) compare the outcomes against what the instance requires. The instance counts as resolved only if the patch applied, every FAIL_TO_PASS test now passes, and every PASS_TO_PASS test still passes — the exact conjunction §7.4 wrote as a formula, now executed. There is no partial credit: a patch that fixes the petabyte case but breaks test_small_sizes scores zero — the same as an empty patch — because it failed the do-no-harm half. The harness tallies resolved-over-total into the headline % resolved and writes a JSON report plus a readable per-instance log.

You can watch the whole machine turn with no model and no tokens at all: pass --predictions_path gold and the harness grades the benchmark’s own reference patches — every instance should resolve — which is exactly the instructor demo in §14.1. It is the cleanest way to see that the grader is just software: build, apply, test, compare.

Now hold the two runs against each other. HumanEval was one forward pass, then a test command; SWE-bench is an agent loop, then a Docker harness. The same six stages — Data, Prompt, Sample, Extract, Execute, Aggregate — carry both, but at repository scale the Sample stage has grown from a single completion into a multi-turn agent, and Execute from “run three asserts” into a per-instance container that builds an environment and runs a real test suite. That growth is not plumbing to skim past: the agent that drives the loop is the scaffold, and how good that scaffold is — as chapter 10 will show — is where a large share of the headline “~95%” actually comes from.

7.6 Why this was a step-change: synthesis versus maintenance

Set the two benchmark families side by side and the difference is not one of degree but of kind. HumanEval tests program synthesis; SWE-bench tests software maintenance — and maintenance decomposes into at least three skills that synthesis never touches:

• Fault localization. Before anything can be fixed, the right few dozen lines must be found in a hundred-thousand-line codebase. HumanEval has no equivalent — the location of the work is given.
• Messy specifications. An issue is not a docstring. It may describe symptoms instead of causes, propose a wrong diagnosis, bury the actual requirement in a stack trace, or assume project context the model does not have.
• Regression safety. The change lands in a living system. P2P encodes the maintainer’s real obligation — fix this without breaking that — a constraint that simply does not exist when writing into an empty file.

If chapter 3 felt abstract when it warned that “can code” is not one construct, this table is the cure: nearly every row names a skill that function-level benchmarks weight at exactly zero.

7.7 The launch numbers

At release in October 2023, the best system in the paper — Claude 2 with BM25 retrieval — resolved 1.96% of instances. GPT-4 came in under 2%. At the same moment, HumanEval was already near saturation. The same model class that had effectively solved function synthesis could barely begin software maintenance. One pair of numbers settled the construct-validity argument empirically: whatever HumanEval was measuring, it was not this.

Read 1.96% generously — it is a success story for the benchmark, not only a failure story for the models. A benchmark that frontier systems nearly fail outright has measurement headroom for years of progress, which is exactly what a saturated HumanEval no longer offered. And the headroom got used. Two and a half years later — as of June 2026 — the same leaderboard reads roughly 95%, vendor-reported, on a variant called Verified that did not exist at launch. Both of those qualifiers are doing heavy lifting. The next three chapters are about what happened in between — and how much of it is real.

8. The Variant Family: Every Variant Is a Confession

Chapter 7 left you with a benchmark and a paradox: 2,294 real GitHub issues, a fully Dockerized harness, a brutal binary metric — and a best score of 1.96%. Within months of the ICLR 2024 oral, SWE-bench had become the number every coding-model announcement led with. And almost as quickly, it stopped being one benchmark. Lite appeared, then Verified, then Multimodal, Multilingual, Live, Pro. Lecture 6’s closing chapter already promised you this family by name. This chapter exists to teach you how to read that family — because most presentations get the reading wrong.

The usual framing is a feature list: “SWE-bench now comes in six flavours, pick the one that suits your use case.” We will use the opposite framing. Nobody builds a benchmark variant for fun. Each one was created — by a specific organisation, at a specific date, at real expense — to fix a specific flaw in the original. Which means the variant list is the most honest documentation of the original’s flaws that exists, written not by critics but by the benchmark’s own ecosystem. Read the variant family as the original’s confession list: every variant is an admission, in dataset form, that something about SWE-bench was broken.

This reading matters directly for the Claim Board. When a vendor reports “X% on SWE-bench”, your first question is now which SWE-bench — and each name on the list below encodes both a dataset and an admission. Here is the family in one table; the rest of the chapter walks through the confessions one by one.

Six entries, four distinct confessions: cost, quality, monoculture, contamination. Hold the column on the right in mind as you read vendor announcements — it is the criticism section of the SWE-bench story, published as datasets.

8.1 Lite: the cost confession

The first variant is the least dramatic and the most practical. SWE-bench Lite (March 2024) cuts the benchmark to 300 instances — single-file bugfixes with clean problem statements. The flaw it confesses is operational: running 2,294 Dockerized repository builds, patch applications, and test suites for every model checkpoint you want to compare is slow and expensive, and a benchmark nobody can afford to run is a benchmark nobody runs honestly. Lite exists so that researchers can iterate. The confession is mild but real — the original was built to be right, not to be usable — and it foreshadows a theme that Lecture 9 will put euro signs on: evaluation has a price, and the price shapes who evaluates what.

8.2 Verified: the confession that cuts the other way

The second variant is the one every leaderboard you have seen this semester actually uses, and its origin story is the most instructive in the family. In August 2024, OpenAI paid 93 professional software developers to manually annotate 1,699 randomly sampled instances of the original benchmark. The annotators were asked, in effect: could a competent engineer solve this from the issue text, and would the tests accept a correct solution? The answers were damning. 38.3% of sampled instances were flagged for underspecified issue text — the issue did not contain enough information to derive the expected fix. 61.1% were flagged for unfair tests — FAIL_TO_PASS tests that check implementation details, exact error-message strings, or behaviour no reasonable developer could infer from the issue. After dropping everything with severe flags, 68.3% of the sample was filtered out. What survived, plus light re-screening, became SWE-bench Verified: 500 human-validated instances.

Now the part that surprises everyone. When OpenAI re-ran GPT-4o (with the Agentless scaffold) on the curated set, its score rose — from 16% to 33.2%. Curation did not deflate the number; it inflated it — because the flaw being removed was unfairness to the model, not generosity toward it. An instance whose tests reject correct patches is not a hard task; it is a broken measuring device that subtracts from every system’s score equally. Strip those out and the remaining instances measure what they claim to measure, and the score doubles. This is chapter 6’s grader lesson cutting in the opposite direction: EvalPlus showed weak graders inflating scores by letting wrong code pass; Verified shows broken graders deflating scores by failing right code. The grader can be wrong in both directions, and you cannot tell which from the headline number.

One caution before you file Verified under “fixed”. Verification by annotation addresses task quality; it does nothing about the age of the issues, which remain public GitHub history from before the models’ training cutoffs — chapter 11 will show that the leakage and weak-test problems persist in Verified, and chapter 12 ends with the remarkable fact that Verified’s own curator eventually walked away from it. The confession of August 2024 was not the last word.

8.3 Multimodal and Multilingual: the monoculture confessions

The third and fourth variants confess the same flaw from two angles: the original is 12 Python repositories, and “can resolve GitHub issues” measured on twelve Python projects is a narrower construct than the sentence suggests — exactly the construct-validity trap from chapter 3. SWE-bench Multimodal (October 2024) adds the dimension text-only benchmarks cannot see: 617 task instances drawn from 17 JavaScript libraries, where the issues contain screenshots — rendering glitches, broken layouts, visual diffs. A real bug report is often a picture; the original benchmark could not represent that at all. SWE-bench Multilingual (spring 2025) attacks the language monoculture directly: 300 instances across 42 repositories in nine languages — JavaScript, TypeScript, C, C++, Go, Java, PHP, Ruby, Rust. (The confession was loud enough to be made twice: ByteDance independently shipped Multi-SWE-bench in April 2025 — 1,632 instances across seven languages, screened by 68 annotators.) The lesson for your own practice: a score earned on Python does not transfer to your stack by default — if your repo is TypeScript or Java, the number you want is the Multilingual one, and it is usually lower and rarely quoted.

8.4 Live and Pro: the contamination confessions

The last two variants confess the deepest flaw, the one that haunts the rest of this lecture. Every instance in the original benchmark is public GitHub history — issue, discussion, and merged fix, all crawlable, almost all of it older than the training cutoffs of the models being evaluated. The benchmark may not be testing whether the model can fix the bug; it may be testing whether the model has seen the fix. SWE-bench-Live (Microsoft, 2025; published at NeurIPS 2025 Datasets & Benchmarks) is the defence-by-freshness: an automated pipeline (RepoLaunch) continuously mines new post-cutoff issues, launching with 1,319 instances from 93 repositories and adding roughly 50 per month. A model cannot have memorised an issue that did not exist when it was trained. The price of freshness is automation — no 93 professional annotators here — so Live trades Verified’s quality control for Live’s contamination control, and you cannot have both in one dataset.

SWE-bench Pro (Scale AI, September 2025) is the defence-by-secrecy, and it produced the most sobering numbers in the family. Pro’s 1,865 instances come in three splits: 731 from public copyleft repositories (GPL-family licences chosen deliberately as a legal deterrent against use as training data), 858 in a held-out set, and 276 from commercial codebases contributed by startups. At launch, every frontier model scored below 25% — GPT-5 at 23.3%, Claude Opus 4.1 at 23.1%, and on the commercial split as low as 14.9–17.8%. Read that against the ~70–80% the same model generation was posting on Verified at the time. Take away familiarity — fresh repos, unseen code, no public history — and the frontier falls from “almost solved” back to roughly one issue in four. As of June 2026 the public Pro leaderboard has climbed to roughly 46–59% (GPT-5.4 at 59.1%, Muse Spark at 55.0%, Opus 4.6-thinking at 51.9%); quote launch and current numbers with their dates, because the gap between them is itself information — some mix of real progress and the new benchmark beginning its own contamination clock. Anthropic’s materials claim 80.3% on Pro for Fable 5 — a vendor-reported figure the public leaderboard does not confirm as of June 2026. That one goes straight onto the Claim Board.

Step back and read the table’s right-hand column one last time: too expensive, unfair graders, Python monoculture, contaminated history. Four confessions, all true simultaneously, all stamped with the names of the organisations that paid to fix them. Keep the list — chapters 11 and 12 supply the forensic evidence behind the third and fourth confessions. But before the autopsy, you need to see the body in motion: the scores themselves, climbing from 1.96% toward saturation in under three years. That curve — and the strange fact that in 2026 it ends in two different numbers at once — is the next chapter.

9. The Leaderboard Curve and the Two Numbers

You now know how the instrument is built (chapter 7) and how its family evolved (chapter 8). This chapter shows you the readings. It exists for two reasons. First, the curve of SWE-bench scores over time is itself a rhetorical object — “up and to the right” is the shape every vendor keynote wants behind it, and you should see the real one before you see the marketing one. Second, the curve ends somewhere strange: in mid-2026 it does not have one endpoint, it has two, eighteen points apart, for the same generation of models — and that split is the doorway to everything in Block B.

Here is the curve, assembled from the numbers in the chapters so far plus the current leaderboards:

One honesty note before the story: this is a spliced curve. The first two rows are measured on the full 2,294-instance set, the rest on the 500-instance Verified subset — and chapter 8 just taught you that the switch to Verified alone moved GPT-4o from 16% to 33.2%. Even the field’s favourite progress chart needs a methods footnote; when you see this curve in a keynote without one, you now know what was glued together.

9.1 Reading the curve

Walk it once. October 2023: the best system on Earth resolves about one real GitHub issue in fifty, while HumanEval sits above 90% — the gap between function synthesis and software maintenance, in one pair of numbers. March 2024: Devin posts 13.86% on a subset, a sevenfold jump — and notice that the jump arrives with an agent loop, the Lecture 7 orchestration pattern of act-observe-act, not merely a new model. August 2024: GPT-4o with the Agentless scaffold reaches 33.2% — the very data point from Verified’s launch in chapter 8. October 2024: Claude 3.5 Sonnet, 49.0%. Through 2025, the Claude-4 generation with vendor scaffolds reaches roughly 70–80%. From 1.96% to the high seventies in about two and a half years.

Two things should itch as you read. First, every leap on this curve is a tangle of a better model and a better scaffold, and the curve never tells you the mixture — Devin’s jump is at least partly agent engineering; Agentless is, as its name advertises, a scaffold choice; the vendor entries at the top are unpublished harnesses. Second, the top of the curve is flattening against the ceiling. Chapter 4 showed you what a saturated benchmark looks like; Verified in the mid-nineties is HumanEval history rhyming, one abstraction level up. A benchmark at saturation stops discriminating — which is part of why chapter 8’s Pro exists at all.

The climb did not go unnarrated while it was happening. In November 2024, with Claude 3.5 Sonnet sitting near 50%, Anthropic’s CEO put the curve into words on a widely heard podcast:

“SWE-bench is … a bunch of professional, real-world software-engineering tasks. At the beginning of the year, I think the state of the art was 3 or 4% … so in 10 months, we’ve gone from 3% to 50% on this task. And I think in another year, we’ll probably be at 90%.”

— Dario Amodei, Lex Fridman Podcast #452, November 2024

Hold that forecast against the board. “Another year” lands in mid-2026 — now — and did it reach 90%? has the most instructive possible answer: it depends entirely on which number you read. As the next section shows, the same model generation sits near 77% on one track and ~95% on another, both as of June 2026. Amodei’s 90% is neither confirmed nor refuted; it falls squarely inside the gap between two simultaneous, equally real measurements of the same systems. The prediction was not wrong so much as underspecified — the very diagnosis chapter 2’s headline claim will earn — and resolving the ambiguity is the whole subject of the rest of this chapter and chapter 10.

9.2 The two numbers

Now the endpoint. Precisely because the scaffold mixture is invisible, swebench.com runs an official bash-only track: every model driven by the same mini-SWE-agent — a deliberately minimal, roughly 100-line loop — with an identical prompt for all; why that design matters is chapter 10’s subject. On that track, as of June 2026, the leaders cluster in the mid-seventies: Claude Opus 4.5 (high) at 76.8%, Gemini 3 Flash at 75.8%, Opus 4.6 at 75.6%, GPT-5.2 at 72.8% — all entries from February 2026, with nothing newer posted since.

Meanwhile, the vendor-reported numbers for the newest frontier models — each measured on the vendor’s own scaffold — sit at approximately 95%: Fable 5 at roughly 95%, Mythos 5 at 95.5%, as of June 2026, vendor-reported. One of these is almost certainly the “~95% on SWE-bench Verified” screenshot pinned to the Claim Board in chapter 2.

Same benchmark. Same metric. The same generation of models. Seventy-seven and ninety-five. The eighteen-point gap between the two tracks is larger than the 15.8-point leap from GPT-4o+Agentless (33.2%) to Claude 3.5 Sonnet (49.0%) — a leap the field treated as a generation of progress — except here it separates two simultaneous measurements of the same systems. So which number is real? Is the 95 inflated — by tuned harnesses, retries, machinery the leaderboard does not disclose? Is the 77 deflated — by a deliberately spartan loop no production deployment would actually ship? Hold the question; do not resolve it yet, because the honest answer is the uncomfortable one: both numbers are real, and neither is “the model’s score” — and understanding why is the scaffold problem, which is exactly where chapter 10 begins.

10. The Scaffold Problem: Prompting, Frozen Into Software

Block B begins here, and it begins with the cliffhanger chapter 9 left on the board: the same handful of frontier models score roughly 77% on SWE-bench Verified’s bash-only track and roughly 95% on the open leaderboard, as of June 2026. Two numbers, eighteen points apart, both produced by careful people running real harnesses on the same 500 tasks. Which one is real? The answer — both, and neither, depending on what question you are asking — requires a distinction that almost no headline ever makes, and this chapter exists to make it unmissable.

A SWE-bench leaderboard entry does not measure a model. It measures a system: a model, plus a scaffold, plus an inference budget. Once you see the three components, the 77-versus-95 paradox dissolves, and so does a great deal of the confusion in vendor launch posts, paper abstracts, and conference-hallway claims about which model is “best at coding.”

10.1 What a leaderboard entry actually measures

The scaffold is everything wrapped around the model: the software that decides which tokens the model sees, which tools it may call, how many attempts it gets, and how the final patch is selected from whatever it produced. You met this layer in Lecture 7 under the name orchestration. A scaffold is an orchestration layer that has been pointed at one benchmark and tuned until the score stopped going up.

On the open side, the named scaffolds are research artifacts you can read: SWE-agent, the original agent loop with a purpose-built interface between the model and the repository; Agentless, a fixed pipeline that deliberately removes the agent’s freedom to wander; OpenHands, an open-source agent platform; and mini-SWE-agent, the deliberately minimal ~100-line loop you met on chapter 9’s bash-only track, which §10.4 returns to. Each one is a different answer to the same engineering question — what context, what tools, what loop? — and each answer moves the score.

On the closed side sit the proprietary vendor harnesses behind the headline numbers. Their recurring ingredients are best-of-\(N\) sampling — run the whole task many times — plus a verifier or reranker that picks the most promising patch from the candidates. Cost-controlled evaluation research puts this honestly: that machinery runs on the order of 10–100× the inference cost of a single attempt (Kapoor et al., AI Agents That Matter). The score it produces is real. It is also a score for the model plus a small factory of retries and judges, not for the model.

The third component, the inference budget, hides inside the other two: how many attempts, how much thinking-token allowance, how many tool-call turns before the harness gives up. Hold a model and scaffold fixed and raise the budget, and the score rises — you are buying more draws from the distribution, exactly the mechanism chapter 5 quantified as pass@k. A leaderboard cell silently fixes all three components and prints one number.

One honest complication to the clean split. A modern reasoning model does internally — search, backtracking, self-correction across a long private chain of thought — much of what a scaffold does externally with best-of-\(N\) and a verifier. The line between model and scaffold is therefore not a wall but a sliding one: raise a model’s thinking-token budget and you are buying it an internal scaffold, paid out of the same inference bill as an external one. This does not break the decomposition; it sharpens the question it tells you to ask. “Which model, at what thinking budget” and “which scaffold, at how many attempts” are the same question asked at two radii, and a single leaderboard number hides the split either way.

10.2 Every scaffold ingredient is a Lecture 6 pattern

Here is the move that makes this chapter more than benchmark trivia. Take any scaffold apart, list its ingredients, and ask where you have seen each one before. Every load-bearing ingredient of every SWE-bench scaffold is a prompting pattern from Lecture 6, institutionalized — taken out of a human’s hands, frozen into software, and run on every task identically.

Walk the rows. The original harness’s single demonstration patch is the purest case: as chapter 7 noted, the shot teaches the shape of a diff, not its content — exactly the format-transfer effect Lecture 6 established for few-shot prompting. Nobody calls it “prompt engineering” on the leaderboard, because it stopped being a choice and became infrastructure.

The agent loop is chain-of-thought with hands. SWE-agent’s reason-act-observe cycle is the auto-regressive scratchpad from Lecture 6, except the intermediate tokens now trigger tool calls and the observations come back as new context. The pattern did not change; it acquired a filesystem.

Best-of-\(N\) with a verifier is self-consistency wearing a suit. Lecture 6 taught you that sampling several reasoning paths and aggregating beats trusting a single greedy decode; the vendor harnesses do precisely that, with a learned or test-based judge replacing the majority vote, at the 10–100× cost noted above. And the BM25-versus-oracle gap in the original paper — same model, same prompt template, different retrieved files, different score — is the cleanest measured demonstration this course has of the Act-1 instinct that curated context beats pasting everything.

One honest correction to our own material belongs here. Lecture 6’s closing chapter told you “the default evaluation protocol across these variants is few-shot.” That sentence is accurate for classic \(k\)-shot benchmarks of the MMLU style; for SWE-bench specifically, the original setting was a single format demonstration, and today’s leaderboard entries are agentic loops, not few-shot prompts at all. The refinement matters, and it is exactly the kind of footnote-level precision this lecture is training you to demand — so we apply it to ourselves first.

10.3 The inverted question

Lecture 6 opened with a question: when a paper claims a new prompting technique improves accuracy by 12 points, what is it manipulating? You now know the answer — it reshapes the distribution the model samples from. Today that question inverts. When one leaderboard entry beats another by five points, you may be looking at better prompting, not a better model. The scaffold is prompting; the leaderboard does not separate it from the weights.

The spread is not hypothetical — chapter 9’s endpoint is the measurement: eighteen points between the bash-only track and the vendor scaffolds, on the same 500 tasks, for the same model generation, in the same month. That gap is wider than most of the model-versus-model differences that launch posts celebrate. If you ranked “models” by scaffolded entries, you would be ranking harness engineering.

This is why “Model X scores Y on SWE-bench” is, strictly, an ill-typed sentence. The well-typed sentence is “system S — model X inside scaffold Z with budget B — scores Y.” Every claim currently pinned to the Claim Board that omits Z and B is omitting variables that demonstrably move the result by double digits.

10.4 Holding the prompting constant: the bash-only track

The benchmark’s maintainers know all of this, which is why swebench.com runs a bash-only track: every model is dropped into the same mini-SWE-agent — that ~100-line loop, one shell tool, an identical prompt for every model, no verifier, no retries. It exists precisely to hold the scaffold constant so that, for once, the differences you see are differences between models. That is the experiment design instinct you already have from every lab course you have taken: vary one factor, clamp the rest.

Now the two numbers from chapter 9 resolve cleanly. The ~77% bash-only figure is the better estimate of model capability under a fixed, minimal harness; the ~95% vendor-scaffold figure is a system capability at a price nobody states. Neither is fake. They answer different questions, and the headline never tells you which question was asked.

So make the question reflexive. Before any benchmark number is allowed to influence a tool decision, ask: which scaffold, how many attempts, at what cost? The first two questions, this chapter has armed you for. The third should now be nagging at you — because you have just learned that the top of the leaderboard is built on best-of-\(N\) rollouts at 10–100× inference cost, and notice what nobody printed on the leaderboard: the token bill. Hold that thought for Lecture 9. First, a harder problem. Everything in this chapter assumed the tasks were sound and only the measuring apparatus needed scrutiny. The next chapter is about what happened when somebody finally read the tasks by hand.

11. SWE-Bench+: The Manual Audit

This chapter is the heart of Block B, and it exists because everything so far — the scaffold problem, the variant family, the leaderboard curve — still trusted the benchmark’s questions. In 2024, a group of researchers did the unglamorous thing nobody had done: they took the instances an agent had supposedly solved and read them, one by one, with human eyes. The result is the single most important number set in this lecture, and it will permanently change how you read the word “resolved.”

11.1 The audit design

The SWE-Bench+ study (Aleithan et al., arXiv 2410.06992) started from a simple sampling decision: do not audit the whole benchmark, audit the successes. They took SWE-Agent + GPT-4 — a representative open system of the era — and collected every instance it had been credited with resolving: 286 in total, of which 251 had confirming execution logs and became the audit set. Then they screened those 251 by hand: read the issue, read the thread, read the model’s patch, read the gold patch, read the tests.

Why audit the successes rather than a random slice of the whole benchmark? Because the leaderboard’s failure mode is asymmetric. A task the model failed that it “should” have passed merely understates the score; a task the model passed without actually solving inflates it — and inflation is what headlines are made of. Auditing the credited wins puts the magnifying glass exactly where a false positive does the most damage.

Pause on the method before the findings. No harness, no heuristic, no LLM-as-judge — a manual audit is the ground truth that automated metrics are supposed to approximate, applied to the metric itself. It is slow, it does not scale, and that is exactly why nobody had done it. Two findings came out, and each one attacks a different link in the chain of trust.

11.2 Finding one: solution leakage

In 32.67% of the audited passing instances, the fix was already present in the issue report or its comment thread. Think about how real GitHub issues work and the mechanism is obvious in hindsight: a maintainer replies “I think the problem is in parse_qs, this diff should fix it,” someone pastes a traceback with the offending line, the eventual PR author sketches the patch in the discussion before opening it. SWE-bench’s construction pipeline kept all of that text, and the issue text is the prompt.

Read that finding in Lecture 6’s vocabulary and it stops being surprising. The model is a pattern-completion engine conditioned on its context; if the answer is in the context, emitting it is not problem-solving, it is the most ordinary in-context copying behavior there is. The model did not resolve the issue. It transcribed the answer that had been pasted into the question. A third of the system’s credited wins were, on inspection, reading comprehension graded as software engineering.

Note what leakage is not: it is not contamination. Contamination means the answer reached the model through its training data; leakage means the answer rode in through the prompt, at inference time, in plain sight. Leakage is in one sense the more embarrassing failure, because no training-data forensics are needed to find it — anyone reading the instance can see the fix sitting in the comments. “Is the answer in the input?” will return as a question you ask of every benchmark in chapter 13.

11.3 Finding two: weak tests

The second finding attacks the grader. In chapter 6 you wrote one sentence on the board: a benchmark score is only as trustworthy as its grader. EvalPlus showed it for HumanEval’s thin unit tests; this is the sentence’s first return, at repository scale. The audit found that 31.08% of the passing patches were suspicious — they turned the FAIL_TO_PASS tests green without actually fixing the bug:

The last row deserves a second look: in 3.59% of credited successes, the patch touched entirely different files from the real fix and still passed. The tests were mined from one specific human PR, so they encode one developer’s checks for one change — they were never designed to discriminate “fixed” from “happened not to trip these particular assertions.” Binary % resolved, the metric chapter 7 defined, inherits every blind spot of those tests.

Notice where the load sits. Of the two test sets from chapter 7, it is FAIL_TO_PASS that carries the entire burden of meaning “this issue is fixed” — and FAIL_TO_PASS is exactly that handful of tests from one PR. The audit shows what happens when that handful is asked to stand in for the issue’s full intent: a grader can be diligent about what it checks and still be fatally silent about what it never checks.

Add the two findings together — leakage at 32.67%, weak-test passes at 31.08% — and roughly two-thirds of everything this system was credited with resolving was compromised in one way or the other.

11.4 The collapse: 12.47% to 0.55%

Now run the correction forward. On the original benchmark, SWE-Agent + GPT-4 reported 12.47% resolved. Filter out the leaky and weak-test instances, and the same system on the same benchmark drops to 3.97%. Then the authors built SWE-Bench+ proper: 548 fresh instances mined from issues created between November 2023 and August 2024 — after the models’ training cutoffs — with solution-leaking instances excluded by construction. On that set, SWE-Agent + GPT-4 resolves 0.55%. The companion systems land in the same basement: SWE-RAG + GPT-4 at 0.73%, AutoCodeRover + GPT-4o at 3.83%.

From 12.47% to 0.55% — just by removing the answers from the questions and using issues the model had never seen. A factor of more than twenty, and not one weight of the model changed. Nothing in this lecture asks you to memorize many numbers, but this pair you keep.

Two honest annotations before you over-rotate. First, the audit targeted one system from one model generation; today’s frontier systems are genuinely stronger, and the collapse factor is not a universal constant you may apply to any score. Second — and cutting the other way — the rot is not confined to the original dataset: the authors note that roughly 94% of original SWE-bench issues predate LLM training cutoffs, and the same leakage and weak-test problems exist in Lite and in Verified. The variant family from chapter 8 inherited the flaws of its parent, including the variant whose entire selling point was human review.

For the Claim Board, this chapter changes the default posture. A vendor number on a benchmark whose tasks have never been hand-audited is not wrong — it is unaudited, and SWE-Bench+ is the existence proof of how much daylight that word can hide. When you rate claims at the end of today, “plausible-but-unproven” is not a polite hedge; after this chapter it is the technically precise rating for most of what is pinned up there.

What generalizes is not the 0.55%. It is the shape of the finding: a benchmark score is an upper bound built from trusting the questions, the grader, and the model’s ignorance of the answers — and every one of those trusts can fail silently, in double-digit percentages, in the most-cited benchmark in the field. One audit of one system could still be dismissed as an unlucky sample. It cannot be, because the supporting evidence arrived from every direction at once — memorization probes, fresh-issue tracks, agents literally reading the future out of git history, and finally the benchmark’s own curator walking away. That is the next chapter.

12. Contamination and Reward Hacking

Chapter 11 left you with a collapse — 12.47% to 3.97% to 0.55% — and an obvious escape route for anyone who wants to keep believing the headline numbers. The SWE-Bench+ audit was, after all, one paper: one team, one agent, one model, 251 hand-screened instances. Maybe the screening criteria were too strict. Maybe SWE-Agent + GPT-4 was an unrepresentative system. A single audit can always be argued with.

This chapter exists to close that escape route. Four independent lines of evidence — different teams, different methods, no shared methodology — converge on the same conclusion: headline SWE-bench scores are inflated by memorization, exploitable harnesses, and weak grading. When four instruments that share no parts all point the same way, the reading is real.

Before the evidence, fix the vocabulary, because the four lines attack four different failure locations. Contamination lives in the weights and leakage lives in the input — the distinction chapter 11 drew. The two new terms: reward hacking lives in the harness — the agent finds a path to a green test result that bypasses the work the test was meant to force. And grader weakness lives in the tests themselves: wrong answers pass, or — the audit in §12.4 will add — right answers fail. Four locations, four diagnostics, four subsections.

Read each line not as gossip about one benchmark but as a reusable diagnostic. Each subsection demonstrates a different way of catching an inflated number — withholding the input, refreshing the data, attacking the harness, auditing the grader — and each of those moves becomes a question on the checklist in chapter 13. The evidence is about SWE-bench because SWE-bench is where the community looked hardest; the techniques transfer to every benchmark you will ever be shown.

12.1 The SWE-Bench Illusion: withhold the input

The first diagnostic is the most elegant: take the input away and see whether the capability survives. A Microsoft team did exactly this in a 2025 study they titled The SWE-Bench Illusion. They gave models the issue text and the repository name — nothing else. No code. No file tree. No repo snapshot. Then they asked the model to name the file that contains the bug.

Think about what an honest answer to that question requires. You cannot localize a fault in code you cannot see. If the model has genuinely never encountered the repository, its accuracy should be close to whatever naming conventions and lucky guesses can provide.

It was not close to that. On SWE-bench repositories, models named the buggy file path with up to 76% accuracy — on held-out repositories the same models managed up to 53%. The model is not navigating the codebase; there is no codebase in the context to navigate. It is recalling the answer. The study’s second measurement makes the mechanism explicit: the verbatim 5-gram overlap between model outputs and the gold-patch fixes reaches 35% on SWE-bench versus 18% on other benchmarks. Five consecutive tokens reproduced verbatim, again and again, at roughly twice the background rate.

None of this should mystify you after Lecture 6’s account of pre-training. These repositories — Django, scikit-learn, matplotlib, sympy — are among the most-scraped codebases on the public internet, and their issues, pull requests, and merged fixes are in the training corpus. The model is doing precisely what it was trained to do: completing a familiar sequence. What the benchmark calls localization is, for a measurable fraction of instances, memorization.

The diagnostic generalizes, and you should keep it: if a score survives the removal of the information the task supposedly requires, the score was never measuring the task. It costs one ablation run, and it has now embarrassed more than one leaderboard.

12.2 SWE-bench-Live: refresh the data

The second diagnostic is the one chapter 8 already named as a variant: if contamination is the worry, build the same benchmark out of issues that did not exist when the model was trained. SWE-bench-Live does this continuously — and the variant doubles as a measuring instrument, because you can run the same agent with the same model on the contaminated set and the fresh set and subtract.

The SWE-bench-Live team did exactly that. OpenHands with Claude 3.7 Sonnet resolves 43.2% of SWE-bench Verified but 19.25% of fresh Live issues. Same scaffold, same model, same metric. More than half the headline evaporates when the issues are new.

Honesty requires a caveat before you read that entire gap as cheating. Fresh issues are harder for legitimate reasons too: they have not been through Verified’s human filtering for underspecified problem statements, their repositories and dependency stacks are less polished targets for an automated harness, and a repository genuinely under-represented in pre-training gives the model less legitimate domain knowledge to draw on. That last effect is a generalization gap, not a contamination artifact. A drop on fresh data is evidence, not proof.

Which is why the within-Live comparison is the part to memorize. Inside SWE-bench-Live itself — all fresh, post-cutoff issues, none seen in training — agents score 22.96% on issues from the original, famous SWE-bench repositories versus 18.89% on issues from newly added repositories, even though the new repositories’ tasks are simpler. Every issue is unseen; only the surrounding codebase differs in familiarity. The four-point gap on simpler tasks is the familiarity effect isolated from everything else: having digested a repository during pre-training helps the model fix even brand-new bugs in it.

That finding is not damning the way verbatim recall is — knowing Django deeply is a real and useful capability. But it has a sharp practical edge: a score earned on twelve famous repositories will not transfer to your repository. The Locust codebase you are carrying through this arc is exactly the kind of less-famous target where the familiarity discount applies, and it is one reason the lab thread has you building your own measurement instead of borrowing one.

12.3 Reward hacking: the agent that read the answer from the future

The third line of evidence attacks from a different angle entirely. Forget what the model knows; look at what the agent does. Recall the metric from chapter 7: an instance counts as resolved when the FAIL_TO_PASS tests flip to green and the PASS_TO_PASS tests stay green. That is what the harness checks, and therefore — by the logic you met in Lecture 6’s three-layer model — that is what the system will optimize. An agent does not optimize “fix the bug”; it optimizes “make the tests pass”, because the tests are the only part of your intent the enforcement layer actually encodes. Any gap between the two will eventually be found — not out of malice, but because the highest-probability path to green tests is whatever path is available.

For a long time, one available path was absurd. The Docker containers in which agents ran each task contained the repository’s full git history — including the future commit that fixes the bug. The gold patch, the very answer the benchmark exists to test, was sitting in the working directory’s object store. Agents discovered this: they ran git log --all, found the fix commit, and transcribed it.

This is publicly documented behavior, observed in real evaluation transcripts, and it has since been fixed — the harness now uses chronologically truncated clones that end before the fix exists. But pause on what it means. Every leaderboard entry produced under the old containers carries an unknown number of resolutions that were lookups, and nobody can retroactively separate them from honest fixes. The agent did nothing wrong by its own objective; it took the cheapest route to the reward. The RL literature calls this reward hacking, and the underlying pattern is older than software: when a measure becomes a target, it stops being a good measure. You met the mechanism in Lecture 6’s account of RLHF — optimize hard enough against a proxy signal and you get the proxy, not the goal.

Even when the agent plays fair, the grader itself may be too weak to notice a wrong answer — and so the companion diagnostic is to strengthen the grader and watch the rankings move. UTBoost did this in 2025, auto-generating stronger test suites for SWE-bench instances and re-judging submitted patches. The result: 26 Verified and 23 Lite instances turned out to have insufficient tests, 345 patches across both sets had been wrongly labeled as passing, and the re-judgment changed 24.4% of the entries on the Verified leaderboard — eleven of forty-five rankings, reshuffled by nothing more than better tests.

The sentence from chapter 6 returns for the second time today, and it has earned the repetition: a benchmark score is only as trustworthy as its grader. EvalPlus proved it for 164 hand-written functions; chapter 11 proved it for an agent’s “resolved” list; UTBoost proved it for the most-watched leaderboard in the field.

12.4 The curator walks away

The fourth line of evidence is institutional, and it is the strangest one. In February 2026, OpenAI — the company that created SWE-bench Verified in August 2024, paying 93 professional developers to clean up the original benchmark — stopped reporting Verified scores for its own models, after an internal audit of the benchmark it had built.

The audit numbers demand careful handling, because they are quoted wrongly all over the internet and you should not join in. OpenAI audited 27.6% of the Verified instances — deliberately chosen from tasks that models frequently failed, not a random sample. Within that failure-skewed subset, at least 59.4% of instances had flawed tests that rejected functionally correct patches — including 35.5% with overly strict tests (checking exact error strings, for instance, where any correct behavior should pass) and 18.8% with tests probing functionality the issue never specified.

The 59.4% is a property of the audited, failure-skewed subset — it is not the flawed-test rate of all 500 Verified instances, and quoting it as such is exactly the kind of number-laundering this lecture is teaching you to catch. The selection makes sense if you think about it: when a frontier model fails an allegedly easy task, “the test is wrong” is a live hypothesis worth auditing — so the audit went looking where flaws were most likely to hide, and found them. What the framing does license is narrower and still serious: a meaningful share of what leaderboards counted as model failures were grader failures. Verified did not just inflate scores, as chapter 11 and §12.3 showed — it also deflated them, in different places, at the same time.

The audit’s second finding needs no framing caveats at all: every frontier model OpenAI tested could reproduce ground-truth fixes from memory. Not “some models, sometimes” — every model tested showed contamination on the benchmark. Taken together with the test flaws, OpenAI concluded that the instrument had reached the end of its useful life; they now recommend SWE-bench Pro, the held-out, harder successor from chapter 8.

There is an irony here worth savoring — Verified was itself the variant that “fixed” the original’s task quality, and eighteen months later its own curator retired it. But the mature reading is not scandal. Benchmarks decay. Models train toward them, the community optimizes against them, and the gap between score and capability widens until the instrument must be replaced. Retirement is maintenance, not crisis. A benchmark number therefore carries an implicit timestamp, and an old number on a famous benchmark is worth less than the same number was when the benchmark was young.

One figure to hold the whole chapter — four evidence lines, four diagnostic moves, four conclusions, converging on a single reading:

None of these four lines is fatal on its own. An ablation can be quibbled with, a fresh-data drop has innocent explanations, an exploit can be patched, an audit has selection effects. Together, they are not quibbleable — and more importantly, each one handed you a question you can ask before trusting the next number someone shows you. The natural move now is to collect those questions, add the ones earlier chapters raised, and compress this entire lecture into something that fits on an index card. That is the next chapter, and it is the artifact you will still be using when every number in this lecture is obsolete.

13. The Nine Questions

This chapter exists to compress. You have now watched a benchmark get built (chapter 7), patched (chapter 8), climbed (chapter 9), confounded (chapter 10), audited (chapter 11), and abandoned by its own curator (chapter 12). You will not carry eleven chapters of evidence into a Tuesday-afternoon tooling decision. What you carry is nine questions — a checklist you run against any benchmark number before you let it influence a decision, and the rating rubric for every claim on the Claim Board.

Every question below earned its place by catching a real inflation (or deflation) somewhere in this lecture, and each entry tells you where you watched it bite. Use the checklist in three situations: when a vendor number is about to influence a tool or model choice, when you review a paper or thesis that reports an evaluation, and when one of your own measurements looks too good. The third situation is the one that will sting.

13.1 The checklist

1. Is the benchmark contaminated? Was the task data public before the model’s training cutoff, and does the score survive a fresh-data variant? You watched this bite in chapter 12: up to 76% file-path accuracy with no code in the context, and 43.2% collapsing to 19.25% on fresh issues.

2. Is the answer leaked inside the task itself? Contamination is in the weights; leakage is in the input — the fix pasted into the issue comments the model is handed. Chapter 11 measured it at 32.67% of SWE-Agent + GPT-4’s “resolved” instances.

3. How strong is the grader? Would a wrong answer pass — and would a correct answer fail? Weak tests inflated scores in chapter 6 (EvalPlus: GPT-4 from 88.4% to 76.2%) and chapter 11, and overly strict tests deflated them in chapter 12’s OpenAI audit — the grader can be wrong in both directions at once.

4. Can the agent cheat the harness? The system optimizes what is checked, not what you meant. Chapter 12’s git log --all episode is the canonical case: the answer was reachable, so it was reached.

5. What exactly was measured? Model or model-plus-scaffold-plus-retries? pass@1 or best-of-N? Which subset of which variant? Chapter 9’s 77-versus-95 gap and chapter 10’s scaffold swings live entirely inside this question; a score without its measurement protocol is not a measurement.

6. At what cost? Tokens, dollars, and wall-clock per solved task — a single accuracy point says nothing about the 10–100× cost multiplier hiding behind best-of-N vendor scaffolds (chapter 10). Hold this question; it gets a section of its own below.

7. Where are the error bars? Outputs are sampled (chapter 3), so a score is an estimate; at 60 tasks with one sample each, the standard error reaches roughly 6.5 percentage points (chapter 5), and two models “five points apart” may be statistically indistinguishable. Most leaderboards publish no variance analysis at all.

8. Is the task distribution representative of your work? Nearly half of SWE-bench Verified is Django (Epoch AI’s analysis: 231 of 500 instances); the original benchmark is 100% Python and bugfix-only. A score earned there predicts little about TypeScript feature work — or about Locust, which is precisely why the lab thread has you building your own measurement.

9. Who reports it — and has anyone reproduced it? A vendor-reported number on the vendor’s own launch page, produced with a proprietary scaffold, is a different epistemic object from an independent submission on a public harness. Reproduction is the oldest quality filter science has; chapters 11 and 12 exist because someone other than the original authors checked.

Notice the structure: questions 1–4 interrogate the benchmark, questions 5–7 interrogate the measurement, and questions 8–9 interrogate the relevance and the reporter. A number must survive all three rings before it deserves to move a decision. Print the list; it is deliberately sized to fit on an index card, and it is the one artifact from today that does not expire when the leaderboard turns over.

13.2 Two claims, rated live

The checklist is also the Claim Board’s rating rubric — verified, plausible-but-unproven, or refuted-as-stated — so let us rate two of the claims pinned in chapter 2 right now, as a demonstration of the mechanics. The remaining claims wait for you in the lab (chapter 14).

Claim: “~95% on SWE-bench Verified.” Run the rings. Question 5 fails immediately: as of June 2026 this is a vendor-scaffold number, with best-of-N sampling and verifier reranking undisclosed, while the bash-only track — identical minimal scaffold for every model — puts the same model generation around 77% (chapter 9). Question 6 fails: no cost is reported anywhere near the claim. Questions 1 and 3 wobble: the substrate is the same Verified set whose curator retired it in chapter 12, and whose rankings UTBoost reshuffled by 24.4% with nothing but better tests. Question 9: vendor-reported, not independently reproduced.

Verdict: plausible-but-unproven, pending scaffold and cost disclosure. Note what the rating does not say. It does not say the number is false — the model generation is genuinely strong, and the bash-only track confirms that much independently. It says the claim as published cannot yet be interpreted, which is a different and more useful statement than either belief or dismissal.

Claim: “the best coding model in the world.” This one never reaches the rings, because it fails before question 1: it is not a measurable claim. No benchmark is named, no metric, no task distribution, no scaffold, no cost envelope — and chapter 3 told you why no single number could back it even if one were offered: “can code” is not one construct. Function synthesis, bug fixing, repo navigation, and long-horizon autonomy are different capabilities with different leaders. A claim that specifies no measurement procedure cannot be verified or falsified; it can only be repeated.

Verdict: refuted-as-stated — which, again, is a statement about the sentence, not about the model. The model behind the slogan may well top several leaderboards. The slogan itself is advertising wearing the grammar of a measurement.

That asymmetry is the skill this lecture has been building. You can now distinguish claims that are wrong from claims that are not even wrong, and you can name the specific missing disclosure that would promote a claim from one rating to the next. That is benchmark literacy — and it is a peer-review skill as much as a consumer skill. The underdefined “Pass@1” abstract from chapter 5 would not survive your checklist either, and the next manuscript you review or write should not need to.

Two of the still-pinned claims need a grader the lecture has only now equipped you to read: Claim Board #3 (LMArena) is decided by blind human preference and #4 (METR) by task-completion over a time horizon — neither is a unit test, and a preference- or LLM-based grader carries the position/verbosity/self-enhancement biases chapter 6 flagged. That is part of why they stay pinned.

13.3 The missing column

One question on the checklist is different from the other eight. Questions 1–5 and 7–9 can be answered from public materials if you dig: papers, harness repos, audit reports, leaderboard metadata. Question 6 usually cannot, because the headline SWE-bench Verified leaderboard reports no cost column, and as of June 2026 most major coding leaderboards still don’t (Aider’s polyglot leaderboard, which does publish per-run cost, is a rare exception that proves the point). A system that solves 60% of instances with one attempt per task and a system that solves 60% by sampling dozens of candidate patches and reranking them with a verifier — at ten to a hundred times the spend (chapter 10) — render as the same row. For the benchmark, that is an omission. For you, an engineer who will pay for every token of every attempt out of a real budget, it is the omission: accuracy-per-euro, not accuracy, is the quantity your decisions actually turn on.

That missing column is the next lecture. Lecture 9 fills it in: who sells tokens, what they cost, why an agentic session’s input grows quadratically with its turn count, and how to compute cost-per-solved-task before you spend it. You have learned to read the score; next, you learn to read the bill. But first you earn the right to either: the lab in chapter 14 puts a real Verified instance and a real leaderboard artifact in your hands, and the first claims come off the board.

13.4 A second response: building better instruments

The nine questions are a defensive skill — how not to be fooled by a number someone hands you. They have a constructive sibling this lecture has only gestured at: if weak graders and leaked answers are the disease (chapter 6, chapter 11), what would a better instrument look like? The research frontier here is young and moving, so take this subsection as an outlook rather than settled content — and as the bridge to the course’s final act. Three directions are worth knowing by name.

• Stronger, partly-automated oracles. EvalPlus already showed that amplifying a test suite changes the verdict; the generalization is to stop hand-writing oracles altogether. Metamorphic testing (Chen, Cheung & Yiu, 1998) sidesteps the oracle problem — what is the correct output? — by checking relations between outputs instead: sort a list, then shuffle and sort it again, and the two results must match, even when you cannot say in advance what the sorted list should be. Pairing such relations with LLM- and static/dynamic-analysis-assisted oracle generation is an active attempt to build graders that catch the “passed-but-wrong” patches of §11.3. The mirror image is worth knowing: flip the task so the model writes the tests rather than the code, and coverage stops being a diagnostic and becomes the metric itself — test-generation benchmarks score a model by the statement/branch coverage and mutation-kill rate of the suites it produces, the same adequacy yardstick chapter 6 turned on graders, now pointed at the model’s own output.

• Self-validation, and why it is hard. The tempting shortcut is to let the model check its own work. The obstacle is a measured property worth filing next to everything else in this lecture: LLMs are frequently confident when they are wrong — their expressed certainty is poorly calibrated against their actual accuracy. A self-grader that inherits the generator’s blind spots is not an independent oracle; it is the same distribution voting twice. Independent verifiers and multi-step validation help, but “the model said it was sure” is not evidence, and treating it as a grader quietly reintroduces the silent-failure mode of §3.2.

• Generating the hard cases. A benchmark mined from existing GitHub issues inherits GitHub’s blind spots. The complementary move is to manufacture adversarial inputs: LLM-driven fuzzing — tools such as TitanFuzz and FuzzGPT (Deng et al., 2023) prime a model to synthesize unusual, edge-case programs that exercise deep-learning libraries far past anything a human test author would write by hand, and have surfaced real, previously unknown bugs in PyTorch and TensorFlow. The framing the talk this section grew from used is apt: fuzzing is an airbag — not mandatory for the car to drive, but exactly what you want for the worst-case collision.

None of these is mature enough to pin to the Claim Board yet, and all three are the same wager: that the cure for a measurement you cannot trust is a better measurement, not no measurement at all — the working antidote to the benchmark nihilism of chapter 1. That wager, and the long-horizon, unattended-agent setting where pass^k (§5.6) is the number that actually matters — increasingly measured in interactive, terminal-based environments (e.g. Terminal-Bench) rather than against frozen snapshots — is where the course’s finale picks up.

14. Lab and Deliverables

Everything before this chapter was, in the end, about other people’s numbers. You spent thirteen chapters learning to distrust scores you did not produce — the leaked issue comments of chapter 11, the scaffold confound of chapter 10, the nine questions of chapter 13. Distrust alone, however, is a cheap skill, and it decays into cynicism if it is never tested against material you can touch.

The only durable cure is to produce a number yourself, under conditions you control, and watch where the cracks appear. That is what this chapter is for. Two in-class exercises put a real benchmark instance and a real leaderboard artifact in your hands. The homework starts the Locust thread that runs to the end of the course and works the statistics of chapter 5 into your fingers. The reflection log records what this lecture changed in your head. The optional reading collects every primary source the lecture leaned on, one line each on why it earns your time.

14.1 In-class exercises

Two exercises run in the lab slot following the lecture. Both are deliberately analog — paper, pens, and arguments — because the skill being trained is judgment, not tooling.

1. One SWE-bench Verified instance, end to end. In pairs, you get one printed (and linked) Verified instance: the issue text, the gold patch, and the FAIL_TO_PASS tests — a real counterpart to the representative instance walked through in §7.5. The worksheet walks you through four questions, in order, and the order matters: read the issue before you read the patch.

   • Could you fix this from the issue text alone? Is the problem statement specific enough that a competent developer — or a model — could act on it? Or does the gold patch quietly resolve ambiguities the issue never mentions, the underspecification that 38.3% of annotated instances were flagged for in chapter 8?

   • Is anything leaked? Scan the issue body and the comment thread for code fragments, suggested fixes, or stack traces that contain the answer. Chapter 11 found the fix sitting in the input on 32.67% of audited resolved instances — check whether yours is one of that kind.

   • Are the tests fair? Would a correct but different patch pass? Look for assertions on exact error strings, incidental formatting, or behaviour the issue never specified — the failure pattern behind both the EvalPlus lesson of chapter 6 and the retirement audit of chapter 12.

   • Which of the nine questions does this instance pass or fail? Run it through the checklist of chapter 13. Keep a third column for “cannot tell from the instance alone” — that column is the interesting one, because it lists exactly what a benchmark’s documentation would need to disclose and usually does not.

2. Leaderboard autopsy. In groups, you get one artifact each — a vendor launch table, an arena screenshot, an aggregator comparison table — and fifteen minutes to list everything that is missing from it. The prompts, if you stall:

   • Which scaffold produced the score, and how many attempts — pass@1 or best-of-\(N\)?

   • What did the run cost, in tokens or dollars per solved task?

   • Are there error bars, or even a second run? Which subset of which benchmark, and what thinking budget?

   • Who submitted the entry — and has anyone outside that organisation reproduced it?

   In the debrief, each group rates one pinned claim from chapter 2’s Claim Board as verified, plausible-but-unproven, or refuted-as-stated, with the missing-disclosure list as evidence. The board’s first claims come down — or stay up — today.

Optional instructor demo. If the lab machine cooperates, you will see the official SWE-bench harness run with --predictions_path gold on two instances — that is, evaluating the benchmark’s own reference patches. It costs zero LLM tokens and shows the full machinery of chapter 7 in motion: the per-instance Docker image pull, the patch applying, the FAIL_TO_PASS and PASS_TO_PASS runs, the JSON report at the end. The point of the demo is that the harness is just software — pinned containers, applied diffs, test runners — and software you can read. (It runs on the instructor machine; the harness wants x86_64 Docker and on the order of 120 GB of disk, which is nothing you need to install.)

14.2 Homework

Three items before the next lecture. The first is the opening move of a thread that runs to the final session of the course.

1. Locust thread, step 1: build one benchmark instance by hand. Find one closed issue in the Locust repository whose merged pull request modified test files. Then verify the fail-to-pass property locally:

   • check out the commit before the fix,

   • run the test the PR added or changed, and watch it fail,

   • check out the merged fix, run it again, and watch it pass.

   You have just executed, manually, the exact filter that turned roughly 90,000 pull requests into 2,294 instances in chapter 7 — and you now know, from the inside, how much judgment hides inside “resolves a linked issue and modifies tests”. Bring the issue link and the test name to the next lecture. In Lecture 9 you will estimate, and then measure, what an agent attempt on this issue costs; in the final session the class assembles everyone’s instances into a small benchmark of its own and audits it the way chapter 11 audited SWE-bench.

2. The pass@k notebook. Work through 08-notebook-passk-estimators, in the style of the Lecture 4/5 notebook series. You simulate a set of tasks with known true success probability \(p\) — a luxury no real benchmark ever has — and then:

   • show empirically that the naive plug-in estimator from chapter 5 underestimates pass@k for \(k > 1\), while the unbiased estimator tracks the ground truth;

   • bootstrap confidence intervals on the estimates and see how wide they are at realistic task counts — wide enough that two “different” models stop being different;

   • reproduce, qualitatively, the result that the optimal sampling temperature grows with \(k\) — the three-way coupling between Lecture 5’s sampling pipeline, Lecture 6’s temperature engineering, and the metric this lecture built.

3. Reading. Read the abstract of the SWE-Bench+ paper (Aleithan et al. — first entry in the audit group below) and skim Narayanan and Kapoor’s Evaluating LLMs is a minefield slides (last group below). Twenty minutes total. Both will feel familiar now — which is the point, and which is what makes the skim fast.

14.3 Reflection log

One AI-diary entry for this lecture, with a single prompt:

Which benchmark number did you trust last month that you would now question — and which of the nine questions made you question it?

Be specific. Name the number and where you saw it. Name the question from chapter 13 that broke it open. Then write two or three sentences on what evidence would restore your trust — because the goal of this lecture was calibrated trust, not blanket suspicion, and an unanswerable distrust is as lazy as an unexamined belief.

If you genuinely cannot find a number you trusted, write about the claim you came closest to repeating to someone else — the vendor headline, the arena rank, the percentage in a screenshot. The diary entries from this arc get a final pass in the last lecture, next to the Claim Board.

14.4 Optional reading

Everything the lecture cited, with the canonical open-access link and one line on why it is worth your time. Grouped by the chapters each source supports.

The benchmark and its lineage (chapters 4–9):

• Chen et al. (2021). Evaluating Large Language Models Trained on Code. The Codex paper: HumanEval’s 164 problems and the original derivation of the unbiased pass@k estimator — the primary source for chapter 5.
• EvalPlus. arXiv:2305.01210. The test-amplification paper behind chapter 6: roughly 80× more tests per problem, and GPT-4’s HumanEval score falling from 88.4% to 76.2% under the stronger grader.
• Jimenez et al. (ICLR 2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? The anchor paper of this lecture. The construction and results sections are very readable, and the 1.96% headline lives there.
• OpenAI (Aug 2024). Introducing SWE-bench Verified. Benchmark quality assurance as a case study — the 93 professional developers, the 38.3% underspecified issues, and the 61.1% unfair tests of chapter 8.
• swebench.com. https://www.swebench.com/ The live leaderboards, including the bash-only track from chapter 9 and chapter 10. Bookmark it — and reread it with the nine questions open.
• Jain et al. (2024). LiveCodeBench: Holistic and Contamination-Free Evaluation of LLMs for Code. The function-level construct, contamination-hardened: problems drawn from post-cutoff competitive-programming contests so a score cannot be memorized — the freshness diagnostic of chapter 12 applied at function level.

The audits (chapters 11–12):

• Aleithan et al. SWE-Bench+: Enhanced Coding Benchmark for LLMs. The central critical text of this lecture: 32.67% solution leakage, 31.08% weak tests, and the 12.47% → 3.97% → 0.55% collapse. Read past the abstract because it is the best worked example of auditing a benchmark rather than consuming one.
• Liang et al. (Microsoft). The SWE-Bench Illusion. The memorization diagnostic: models name the buggy file path from issue text and repo name alone far more often on SWE-bench repos than on held-out ones.
• SWE-bench-Live. arXiv:2505.23419. The continuously-mined fresh-issue pipeline, and the sobering same-system drop from 43.2% on Verified to 19.25% on post-cutoff issues.
• UTBoost. arXiv:2506.09289. Auto-generated stronger tests change 24.4% of Verified leaderboard entries — the grader sentence from chapter 6, quantified at leaderboard scale.
• OpenAI (Feb 2026). Why SWE-bench Verified no longer measures frontier coding capabilities. The curator retiring its own benchmark; pair it with the Verified launch post above for a complete before/after arc. Keep chapter 12’s caveat in mind as you read: the 59.4% flawed-test figure describes a deliberately failure-skewed audit subset, not all 500 instances.

The wider argument (chapters 3 and 13):

• BetterBench. arXiv:2411.12990. A benchmark-quality assessment framework from Stanford — essentially a peer-reviewed superset of the nine questions. Grading any benchmark you care about against it is a worthwhile evening.
• Kapoor et al. AI Agents That Matter. The case for cost-controlled evaluation and accuracy-versus-cost Pareto curves — the paper behind checklist question six, and the bridge into the next lecture.
• Narayanan & Kapoor. Evaluating LLMs is a minefield. Annotated slides; the single best compact companion to this entire lecture, and the homework skim.
• Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. The reference on using a model as a grader — and on its measured failure modes: position, verbosity, and self-enhancement bias. The reason a model is not a free oracle for the un-testable outputs of chapter 6.

The constructive outlook (§13.4):

• Deng et al. (2023). Large Language Models Are Edge-Case Fuzzers (FuzzGPT). LLM-driven fuzzing: prime a model to synthesize unusual, edge-case programs and it surfaces real, previously unknown bugs in PyTorch and TensorFlow. The constructive counterpart to auditing — manufacture the hard cases instead of mining them. (Prior work: TitanFuzz.)
• “Mind the Confidence Gap.” arXiv:2502.11028. Evidence for the calibration problem behind §13.4’s warning: LLMs are routinely confident when wrong, which is why a model is not a trustworthy grader of itself.

These are the references you need. Any leaderboard screenshot, launch post, or “best coding model” headline that cannot survive contact with this list — or with the nine questions — should be treated accordingly.

The lecture ends here, but the Claim Board does not. It stays on the wall with most of its claims still pinned, because one question on the checklist is still wide open: at what cost? A score without a price tag is half a measurement, and you cannot finish rating a “best coding model” claim until you know what the run that produced it cost — in tokens, in dollars, in attempts. That is Lecture 9: the model market, the platforms, and the price of a token. Bring your Locust issue. See you in a week.