Welcome to Act 3. Acts 1–2 are two completed movements: you experienced the shift, then understood it mechanistically. Act 3 asks a different question — not "why does the system behave this way?" but "how do we engineer with it?" And engineering begins with measurement. The whole lecture is built around one scene: June 2026, a launch post reads "~95% on SWE-bench Verified." We are going to take that number apart and reassemble it with every flaw visible. Measurement, not mechanism.

Poll the room; collect WRITTEN predictions. This is predict–observe–explain: the surprise, when it comes, is the teaching moment, and it only works if the prediction is on paper first. The bet is NOT a setup for a cynical reveal — students spend the rest of the course earning the right to answer it properly, starting with the chapter-14 Locust homework. We come back to this bet in chapter 13.

If the instinct says "no, it won't transfer" — good instinct. But press on WHY, and notice how fast the vocabulary runs out. None of these are rhetorical: each has a precise answer for SWE-bench specifically. Don't dismiss the number ("obviously marketing") and don't worship it either — both are ways of not thinking. This slide seeds the four-component decomposition that comes next.

The master frame of the deck — every chapter resolves one of these four diagnostic questions. Model → chapters 3, 5. Scaffold → chapters 9, 10. Grader → chapters 6, 11, 12. Data → chapters 4, 7, 11, 12. A number is meaningless without its measurement procedure, and the procedure has parts. By the end of today, students can decompose any score into these four and know which question to ask before believing it.

The geometry IS the argument. Walk students from L7's v1.3 to v1.4: the evaluation harness is the outermost ring, wrapping the orchestration layer (L7) and the model (L5). A benchmark is not a probe that touches the model directly — it is a system built to measure another system from the outside, and everything between the two (scaffold, grader, data) leaves fingerprints on the number. This diagram returns as a callback in the closing chapter, where every fingerprint will have an explanation.

The thesis sentence, verbatim, the one to carry to the exam. Nothing in it is cynicism — it is the ordinary engineering move: a number is meaningless without its measurement procedure, and the procedure has parts. The two failure modes to avoid: benchmark WORSHIP (take the headline at face value) and benchmark NIHILISM ("all benchmarks are gamed, go by vibes"). Both are ways of not thinking. A benchmark is an instrument; this lecture is the calibration manual.

Roadmap: in the room this is two 90-minute blocks. Block A runs to the leaderboard curve's cliffhanger at the end of chapter 9; Block B opens with its resolution. The Block A recap is a safety net — anyone lost in pass@k can re-anchor before Block B.

Act 2 had the Mystery Board ("why does this happen?"). Act 3 gets its successor with the opposite polarity: the Claim Board asks "should I believe this?" Real claims go up today, quoted from actual launch posts, uncommented. A claim only comes down RATED — no claim fades away unexamined, and no claim is dismissed by eye-roll. Every verdict carries an "as of" stamp, because a claim about a model is a claim about a moving market.

Pin each claim. All five are real, current, quoted as found as of June 2026 — and some will already be stale by the exam, which is itself part of the lesson. Invented strawman claims would teach you to spot strawmen; these teach you to interrogate the real thing. Lecture 9 adds a second wave (pricing, "cheapest model"); the course finale makes the last pass. Bold row 1 — it is the headline we take apart for the next eleven chapters.

Name the verdicts but do NOT apply them yet — that happens in chapter 13, after students have the tools. Expect the MIDDLE verdict to do most of the work: the uncomfortable finding of this lecture is not that vendors fabricate numbers, it is that the conditions that give a number its meaning are routinely omitted — and an unomitted version would often look considerably less impressive. The commitment: by the end of today, students can already rate claims 1 and 2.

Three problems, three resolution chapters — the map of the rest of Block A. None of the three is sloppiness; each is built into the system being measured. And each maps onto machinery you already own from Lectures 4–7: nothing here is new mechanism, it is the old mechanism viewed from the perspective of someone trying to attach a number to it.

Sampling → ch5 pass@k. Harness → ch6 EvalPlus, plus ch10 scaffold and ch11 audit. Construct validity → ch7 SWE-bench. Keep the triple in view and the next eleven chapters organize themselves. The cleanest demonstration of the construct problem sits right at the seam between this lecture's two founding benchmarks: by 2023, frontier models scored near 90% on function synthesis while the best system resolved under 2% of real repository issues — a 45-fold difference in apparent ability, same models, same year.

The master pipeline. Two shapes of the same machine: at the function level (HumanEval) the whole thing is one forward pass; at the repository level (SWE-bench) the Sample and Execute stages swell into an agent loop with tools. What changes between them is never the six stages — only how much machinery hides inside Sample and Execute. The two worked runs that follow (§4.2 HumanEval, §7.5 SWE-bench) are these two shapes, walked end to end.

Two deliberate design choices: the problems were HAND-WRITTEN (composed fresh) so they could not already sit in the training corpus — an early acknowledgement of the contamination problem ch12 escalates; and correctness is decided by RUNNING TESTS, not by comparing against a reference. A generated function that looks nothing like the reference but passes every test is correct, full stop. For 2021 this template was exactly right: it answered "can a model write a correct standalone function at all?" with a number, not a demo. The schematic is written for this lecture, not taken from the dataset.

Greedy + pass@1: one completion per task with randomness off. "Greedy" = always take the most probable next token (temperature 0), one deterministic output per rerun. "pass@1" = the fraction of the 164 whose single greedy completion passes — count the passes, divide by 164, no statistics needed because nothing was random. That is exactly why ch5 exists: turn the randomness back on, allow several attempts, and "how often does it pass" stops being a fraction you count and becomes a quantity you must estimate. Set the stop markers badly — halt too early, or forget to halt — and a capable model loses points for harness reasons, not coding reasons (§3.2).

The grader is dumber, and therefore far more trustworthy, than students expect. Pure mechanical concatenation in a fixed order, executed in a sandbox because you are about to run code a machine wrote and nobody read. "Generate once, grade twice" lives here: freeze the stored completions, then re-grade the SAME bytes against a stronger oracle (ch6 EvalPlus) without re-running the model. The grader's verdict is only as good as those hidden asserts — the blind spot ch6 opens.

Be precise about what saturation means, because the sloppy reading is on half the Claim Board. Above 90%, HumanEval can no longer distinguish a merely competent model from an exceptional one — any more than a primary-school arithmetic test can rank mathematics students. This is why the field moved on, and why the third open question — "is completing a function from a docstring the same skill as fixing a real bug in a real repository?" — could only be answered by building a benchmark around the second construct. That is SWE-bench.

~27 min elapsed. Quick checkpoint, brief pause for questions. We now know WHAT to measure and that the first tools ran out of headroom. Next: pass@k, deliberately the most mathematical chapter of Act 3 — intuition first, formulas after.

Tell students: this is deliberately the most mathematical chapter of Act 3. Intuition first, formulas after. The mathematics is an estimator-bias argument built from binomial coefficients, not new transformer machinery — and it exists in this lecture because underdefined "Pass@1" claims are appearing in real manuscripts right now.

The model's output is sampled. "Can model M solve task t?" is mis-typed: it presumes a boolean where the mechanism gives a probability. You might object that greedy decoding rescues the boolean — but production T=0 is only mostly deterministic (hosted stacks drift), and evaluating only the argmax discards everything Lecture 5 taught you the model actually is. The probabilistic frame is the correct TYPE for the object being measured.

Intuition, not proof. p-hat itself is a fair estimate; but you report a CURVED function of it. The disease starts at k=2 (at k=1 the function is linear, Jensen has nothing to grab). The bias grows with k (more curvature) and shrinks with n (less noise to rectify). Bias, unlike variance, points the same way on every task and survives the mean.

This is the slide §5.2 promises "for anyone who prefers the proof to the picture." The convexity of (1-x)^k for k>1 is the whole engine; the size of the bias is the second-order Taylor term ½f''Var, exact at k=2 because f''=2 is constant there. At k=1 the function is linear, Jensen has nothing to grab, and the plug-in is innocent.

Every k-subset of your n attempts is a simulated k-try experiment; the formula averages over all of them at once. It draws WITHOUT replacement where the naive estimator implicitly draws WITH replacement — your k real attempts would be k distinct samples, and treating them that way is exactly what removes the bias. NB: math here is rendered by Marp's KaTeX; binom and mathbb must work.

Walk through the arithmetic live. The 2-point gap is small here but grows with k; at the small n real evaluations use (samples cost money — Lecture 9's subject), it is material.

The two-minute simulation in the homework notebook: fix a true p = 0.3, draw c from a binomial with n = 10, compute both estimators on thousands of simulated tasks, plot against the true pass@2 of 0.51. The unbiased histogram is noisy but centred on the line; the naive one is just as noisy AND shifted off it. Watch the bias refuse to close as you add more tasks (while it DOES close as you raise n) — that teaches the variance-versus-bias distinction better than any derivation.

The remedies are neither exotic nor expensive: (1) report the denominator — task count and samples per task; (2) run more than once and show the spread; (3) test differences, not scores — "A beats B" needs roughly twice the evidence a single score does. When chapter 13 hands you the nine questions, "Variance? error bars?" is question 7, and you now own the two-line calculation that answers it yourself: root of p(1-p)/N, times root-2 if comparing.

You've seen this logic in Lecture 6's self-consistency: sampling diversity pays precisely when some mechanism — there a majority vote, here a test suite — selects among the draws. Two labs can both legitimately report "Pass@1" and mean different quantities: Lab A's greedy run measures the argmax path; Lab B's n-sample estimate measures the average mass on correct completions. A third paper citing both is comparing experiments, not models.

pass@k measures what a model can do for you when something checks its work; pass^k measures what it can be trusted to do when nothing does. A 90%-per-run system scores pass@3 = 0.999 ("deploy it") but pass^3 = 0.729 ("it breaks weekly"). For a production AGENT that files PRs or runs your nightly pipeline unattended, pass^k is usually the number you actually care about — nobody is standing by to pick the good run out of k. Hold this mirror: the course finale and the agentic-AI follow-up both live on the pass^k side.

The honest summary of "which k": pass@1 dominates every headline, and the unbiased estimator (§5.3) earns its keep in the research regime that produced it — Codex's n≈200 → pass@1/10/100, repeated-sampling coverage curves, and the reasoning-RL frontier where pass@k at LARGE k diagnoses whether RL widens or narrows a model's reach. Ties straight back to §5.5: exploration pays exactly when a selector (tests, or the eventual reward) picks among the draws. This is also the live link between this lecture and the course finale / agentic follow-up.

~45 min elapsed. Pause for questions. Transition from metric to grader. Everything in this chapter concerned the statistics of COUNTING — how to turn pass/fail draws into a trustworthy probability. We never once questioned the pass/fail itself. The next chapter asks the question this one carefully avoided: who decides what counts as correct?

Two problems resolved in one chapter: the grader (EvalPlus) and the construct (SWE-bench). Chapter 5 spent all its energy on the estimator — but every pass@k computation starts from a per-sample verdict produced by a test suite the estimator never questions. pass@k does not estimate the probability the code is CORRECT; it estimates the probability the code PASSES THE TESTS. Those coincide only when the tests are strong.

Read the bars slowly: no weights, no prompts, no sampling settings, no estimator changed — only how carefully the answers were checked, and a substantial slice of GPT-4's measured competence evaporated. The 88.4% was never a measurement of correctness; it was correctness PLUS an invisible second term.

The structural lesson: what a benchmark measures decomposes as p_pass = p_correct + p_(pass AND wrong) — the probability of being right, plus the probability of being wrong in a way the oracle cannot detect. The second term is invisible from inside the benchmark: no resampling, no unbiased estimator, no confidence interval surfaces it, because every statistical tool operates downstream of the oracle's verdict. EvalPlus measured it the only way it can be measured: build a stronger oracle and take the difference.

Pin this sentence to the board. Weak enforcement does not just mean undetected failures; it means inflated belief — every wrong-but-passing completion converts directly into score. And the failure is silent in the same direction every time: a too-strict oracle announces itself with implausibly low scores someone investigates, while a too-weak oracle produces beautiful numbers nobody is motivated to question.

GitHub already contains millions of records of exactly the work we want to measure. Every merged PR that fixes an issue is a completed exam question — the issue text is the task, the merged diff is a known-good solution, and the tests it added are a grader. Nobody had to write tasks for SWE-bench; it is mined from work real maintainers already did and already graded. ~90,000 PRs from 12 Python repos → 3 filters → 2,294 instances (~2.5%).

The task is brutally close to a junior developer's Monday morning: the issue text (however vague) plus a repo snapshot often >100,000 lines, and you must produce a unified diff that applies cleanly. Not a function. Not a file. A diff against a living codebase. You will hold one of these instances in your hands during the lab — issue text, gold patch, F2P tests, on paper, dissected in pairs.

The grep is fault localization (the §3.3 construct, forced into the open because nothing told the agent where to look); the pytest run is the agent checking its own work before the grader sees it — the feedback that closes the loop. A weaker agent loops longer, edits the wrong file, or never finds the function — and because every extra turn is more sampled tokens, this is where the SCAFFOLD (turn budget, tools, context trimming) starts moving the score as much as the model. Run in batch over all 500 Verified instances → one line per instance in a predictions file: instance_id, model_name_or_path, model_patch (the diff). That file, not the running model, is what gets graded.

"Generate once, grade twice" again: the predictions file is frozen, so you can re-grade the same patches against stronger or fresher tests (ch11 UTBoost) without paying the agent again. The do-no-harm half (P2P) is why a patch that fixes the petabyte case but breaks test_small_sizes scores zero — same as an empty patch. Watch the whole machine with no model and no tokens: --predictions_path gold grades the benchmark's own reference patches (every instance should resolve) — the §14.1 instructor demo.

The conjunction is only as strong as the test sets inside it: "conservative given the tests" is not the same as conservative — mark that thought, it matures in chapter 11. The verdict machinery is a fully Dockerized harness (per-instance images pinning the snapshot and exact dependency versions) — the part of the grader that guarantees the verdict is at least REPRODUCIBLE. Whether it is RIGHT stays open until chapter 11. Read 1.96% generously: a benchmark frontier systems nearly fail has measurement headroom for years — exactly what saturated HumanEval no longer offered.

EXTENDED — fold into the launch slide's notes if compressing. If chapter 3 felt abstract when it warned "can code is not one construct", this is the cure: nearly every row names a skill that function-level benchmarks weight at exactly zero. The original SWE-bench prompt is itself Lecture 6 material — a retrieval policy (BM25 vs oracle), a single format demonstration, and instruction text, every one a Lecture 6 decision compiled into a harness where nobody calls it prompting anymore.

~65 min elapsed. Brief pause. Transition: the rest of Block A is the SWE-bench deep dive. The headline claim on the wall — ~95% on Verified — is a claim about the benchmark we just built. You cannot rate it without knowing exactly how the number is produced.

Within months of the ICLR 2024 oral, SWE-bench became THE number every coding-model announcement led with — and almost as quickly it stopped being one benchmark. Lite, Verified, Multimodal, Multilingual, Live, Pro. The usual framing is a feature list. We use the opposite: nobody builds a variant for fun. Each fixes a specific flaw in the original — so the variant list is the most honest documentation of the original's flaws that exists. Every variant is a confession.

Six entries, four distinct confessions: cost, quality, monoculture, contamination. When a vendor reports "X% on SWE-bench", your first question is now WHICH SWE-bench — each name encodes both a dataset and an admission. Read the right-hand column: it is the criticism section of the SWE-bench story, published as datasets.

Counter-intuitive: most people expect curation to make a benchmark harder. An instance whose tests reject correct patches is not a hard task; it is a broken measuring device that subtracts from every system equally. This is chapter 6's grader lesson cutting the OTHER way: EvalPlus showed weak graders INFLATING scores by passing wrong code; Verified shows broken graders DEFLATING scores by failing right code. The grader can be wrong in both directions, and you cannot tell which from the headline.

"Can resolve GitHub issues" measured on twelve Python projects is a narrower construct than the sentence suggests — exactly the construct-validity trap from chapter 3. A real bug report is often a picture; the original benchmark could not represent that at all. The confession was loud enough to be made twice: ByteDance shipped Multi-SWE-bench in April 2025, 1,632 instances across seven languages.

The hero chart of the deck. Walk it once: Oct 2023, the best system on Earth resolves about one GitHub issue in fifty while HumanEval sits above 90%. Mar 2024, Devin posts 13.86% — and the jump arrives with an AGENT LOOP, not just a new model. Aug 2024, GPT-4o + Agentless 33.2%. Through 2025, the Claude-4 generation reaches ~70-80%. ONE honesty note: this is a SPLICED curve — first two rows on the full 2,294 set, the rest on the 500-instance Verified subset. Even the field's favourite progress chart needs a methods footnote. The fork at the right end (77 vs 95) IS the cliffhanger.

The hype voice. Don't dismiss it — the trajectory IS real. Amodei's 90% is neither confirmed nor refuted; it falls squarely inside the gap between two simultaneous, equally real measurements of the same systems. Resolving that ambiguity is the whole subject of the rest of this chapter and chapter 10.

One of these is almost certainly the "~95% on SWE-bench Verified" screenshot pinned to the Claim Board. So which number is real? Is the 95 inflated by tuned harnesses and retries? Is the 77 deflated by a deliberately spartan loop no production deployment would ship? Hold the question — do not resolve it yet. The honest answer is the uncomfortable one: both are real, and neither is "the model's score."

Dramatic pause. This is the hype → realism hinge. The break falls here. Both numbers are real, and neither is the model's score — and understanding why is the scaffold problem, which is exactly where chapter 10 begins.

Block A recap is a safety net — students who were lost in pass@k can re-anchor before Block B. The break falls on the cliffhanger; Block B opens with its resolution.

Block B opens. The first move is to resolve the cliffhanger the break left on the board: the same handful of frontier models score ~77% on the bash-only track and ~95% on the open leaderboard, as of June 2026. Two numbers, eighteen points apart, both produced by careful people running real harnesses on the same 500 tasks. The answer — both, and neither, depending on what question you are asking — requires a distinction almost no headline ever makes.

60 seconds in pairs. Re-establish the cliffhanger after the break — students just saw this 10 min ago. Let them argue it before the resolution lands. The "single missing word" they are reaching for is SCAFFOLD.

The resolution. The scaffold is everything wrapped around the model — the software that decides which tokens it sees, which tools it may call, how many attempts it gets, how the final patch is selected. You met this layer in Lecture 7 as ORCHESTRATION. A scaffold is an orchestration layer pointed at one benchmark and tuned until the score stopped going up. Ranking scaffolded entries ranks harness engineering.

The move that makes this chapter more than benchmark trivia: every load-bearing ingredient of every SWE-bench scaffold is a prompting pattern from Lecture 6, institutionalized — taken out of a human's hands, frozen into software, run on every task identically. Nobody calls it "prompt engineering" on the leaderboard because it stopped being a choice and became infrastructure. Best-of-N with a verifier is self-consistency wearing a suit; the agent loop is chain-of-thought with hands. The bold row is the one that buys the vendor's extra ~18 points — at 10–100× the inference cost (Kapoor et al.).

Now the two numbers resolve cleanly. Neither is fake; they answer different questions, and the headline never tells you which. So make the question reflexive: before any benchmark number influences a tool decision, ask — WHICH SCAFFOLD, HOW MANY ATTEMPTS, AT WHAT COST? The first two, this chapter armed you for. The third should now be nagging: the top of the leaderboard is built on best-of-N at 10-100× inference cost, and nobody printed the token bill. Hold that for Lecture 9.

EXTENDED — fold into the bash-only notes if compressing. The spread is not hypothetical: chapter 9's endpoint IS the measurement — eighteen points between bash-only and vendor scaffolds, same 500 tasks, same generation, same month. Wider than most model-versus-model differences launch posts celebrate. If you ranked "models" by scaffolded entries, you would be ranking harness engineering.

Two lecture chapters merged: the manual audit (ch11) and the convergence of evidence (ch12) are one argument in two movements. Everything so far — the scaffold problem, the variant family, the leaderboard curve — still trusted the benchmark's QUESTIONS. In 2024 a group did the unglamorous thing nobody had done: they took the instances an agent had supposedly solved and read them, one by one, with human eyes. The result will permanently change how you read the word "resolved."

Two findings came out, each attacking a different link in the chain of trust. Auditing the credited WINS puts the magnifying glass exactly where a false positive does the most damage.

Read in Lecture 6's vocabulary and it stops being surprising: if the answer is IN the context, emitting it is not problem-solving — it is the most ordinary in-context copying there is. A third of the system's credited wins were reading comprehension graded as software engineering. Leakage is in one sense more embarrassing than contamination: no training-data forensics needed — anyone reading the instance can see the fix sitting in the comments. "Is the answer in the input?" returns as a checklist question in chapter 13.

Add the two findings: leakage 32.67% + weak-test passes 31.08% — roughly TWO-THIRDS of everything this system was credited with resolving was compromised one way or the other. F2P carries the entire burden of meaning "this issue is fixed", and F2P is exactly that handful of tests from one PR. A grader can be diligent about what it checks and still be fatally silent about what it never checks.

Predict-observe-explain. Let them commit a number out loud. The reveal on the next slide — 0.55% — is the emotional centrepiece of Block B, and it only lands if they guessed first.

THE slide. The one pair to keep. 12.47% reported → filter leaky + weak-test instances → 3.97% → SWE-Bench+ proper (548 fresh instances, created Nov 2023–Aug 2024, after the cutoffs, leakage excluded by construction) → 0.55%. A factor of more than twenty, and NOT ONE WEIGHT of the model changed. The model didn't change; the measurement did. Two honest annotations: the audit was one system from one generation (today's are genuinely stronger), and — cutting the other way — ~94% of original SWE-bench issues predate the cutoffs, and the same flaws exist in Lite AND Verified.

The collapse sentence, verbatim. What generalizes is not the 0.55% — it is the SHAPE of the finding: a benchmark score is an upper bound built from trusting the questions, the grader, and the model's ignorance of the answers, and every one of those trusts can fail silently, in double-digit percentages, in the most-cited benchmark in the field. One audit of one system could be dismissed as an unlucky sample. It cannot be — because the supporting evidence arrived from every direction at once.

The most elegant diagnostic. These repos — Django, scikit-learn, matplotlib, sympy — are among the most-scraped codebases on the internet; their issues and merged fixes are in the training corpus. The model is doing exactly what it was trained to do: completing a familiar sequence. What the benchmark calls localization is, for a measurable fraction of instances, memorization. The diagnostic generalizes: if a score survives the removal of the information the task supposedly requires, it was never measuring the task.

Within-Live, the cleaner comparison: agents score 22.96% on issues from the famous original repos vs 18.89% on newly added ones, even though the new tasks are SIMPLER — the familiarity effect isolated. Knowing Django deeply is a real capability, but a score earned on twelve famous repos will not transfer to YOUR repo — which is why the lab thread has you building your own measurement. The git log --all behaviour is publicly documented and since fixed (chronologically truncated clones); but every entry under the old containers carries an unknown number of lookups nobody can separate from honest fixes.

Four independent lines — different teams, different methods, no shared methodology — converge on the same conclusion: headline SWE-bench scores are inflated by memorization, exploitable harnesses, and weak grading. Each attacks a DIFFERENT failure location: contamination (weights), leakage (input), reward hacking (harness), grader weakness (tests). None is fatal alone — an ablation can be quibbled with, a fresh-data drop has innocent explanations, an exploit can be patched, an audit has selection effects. Together they are not quibbleable. When four instruments that share no parts all point the same way, the reading is real.

Handle the audit numbers carefully — they are quoted wrongly all over the internet. OpenAI audited 27.6% of Verified instances, DELIBERATELY chosen from tasks models frequently failed (not random). Within that failure-skewed subset, ≥59.4% had flawed tests. The 59.4% is a property of the audited subset — NOT the flawed-test rate of all 500 instances, and quoting it as such is exactly the number-laundering this lecture teaches you to catch. The irony: Verified was the variant that "fixed" task quality, and eighteen months later its own curator retired it. An old number on a famous benchmark is worth less than the same number was when the benchmark was young.

Everything we've built compresses into one index card. You have watched a benchmark get built, patched, climbed, confounded, audited, and abandoned by its own curator. You will not carry eleven chapters of evidence into a Tuesday-afternoon tooling decision. What you carry is nine questions — and the rating rubric for every claim on the board.

Every question earned its place by catching a real inflation (or deflation) somewhere in this lecture. The structure: Q1–4 interrogate the BENCHMARK (contamination, leakage, grader, harness-cheating), Q5–7 interrogate the MEASUREMENT (scaffold, cost, error bars), Q8–9 interrogate the RELEVANCE and the reporter (your distribution, who reports + reproduction). A number must survive all three rings before it deserves to move a decision. Print the card — it is the one artifact from today that does not expire when the leaderboard turns over.

"Plausible-but-unproven" does NOT say the number is false — the generation is genuinely strong and the bash-only track confirms that much. It says the claim AS PUBLISHED cannot yet be interpreted, which is more useful than either belief or dismissal. "Refuted-as-stated" is a statement about the sentence, not the model: the model behind the slogan may top several leaderboards; the slogan is advertising wearing the grammar of a measurement.

Hard guard: name cost as the next lecture's subject, do NOT treat it here. Aider's polyglot leaderboard, which publishes per-run cost, is the rare exception that proves the point. You have learned to read the score; next, you learn to read the bill.

ONE outlook frame — names only, no deep treatment. The nine questions are a DEFENSIVE skill (how not to be fooled); these are the constructive sibling. The research frontier is young and moving — take this as an outlook and the bridge to the finale, where pass^k (the unattended-agent number) picks up.

Visual callback to the opening Claim Board (S008). The payoff of the device from chapter 2: the first claims come down — rated, not dismissed. Note the "as of Jun 2026" stamp — part of what the board teaches is that the stamp matters.

ONE compact foreshadow frame — do NOT author full worksheets; the lab instructions live in the exercise files. Distrust alone is a cheap skill that decays into cynicism if never tested against material you can touch. Bring your Locust issue link and test name to Lecture 9.

The closing callback. At the start, this was a diagram; now it is a diagnostic tool — students can name every layer and the chapter that explained it. Pair with the thesis sentence one last time: a benchmark score is a measurement of a system under specific, gameable conditions.

The "go touch the real thing" slide. Every benchmark in this lecture is a public dataset + an open-source harness — the surest cure for benchmark mysticism is to load one and read a row. Appendix B is the statistical backstop for anyone who wants the metric chapter from first principles.

Closing scaffold (placeholder). The real deck fills chapters 1–14 between the Claim Board and this slide. Next week: Lecture 9, Token Economics — the missing column (cost). Bring your Locust issue.