08 Lecture 8 Quiz: Measuring the Machine

The whole lecture decomposes a score into model, scaffold, grader, and data, and each component supplies “the question you learn to ask,” so the figure is a measurement of a system under conditions, not a property of the weights (A). The most tempting wrong answer is B — the “spec-sheet” reading the launch post invites, treating the number like a clock speed — which the lecture names a category error. C is benchmark nihilism, explicitly rejected as “a different way of not looking.” D inverts the framing: the four components are not independent validators that cross-check one another; they are four separate places where the number can be inflated or deflated.

The lecture states the polarity flip directly (C): the Mystery Board collected “why does this happen?” questions whose mechanistic answers stay answered — tokenization will still hide characters next year — while the Claim Board asks “should I believe this?” about claims tied to a moving market, where the number may be superseded, the model retired, the benchmark withdrawn. The most tempting wrong answer is B, but a mechanistic explanation is not a mere “opinion versus proof” distinction; the point is that mechanism is permanent while market facts are not. A confuses the dual format: both boards lived as a physical board plus a digital page. D invents a weekly re-voting rule; claims expire because the world moves, not because the vote repeats.

The lecture gives two answers (D): production temperature 0 is only mostly deterministic — inference stacks drift across replicas, batches, and silent updates — and even perfect greedy decoding evaluates one privileged point, the argmax path, discarding the rest of the distribution Lecture 5 taught you the model actually is. The most tempting wrong answer is C: it sounds rigorous, but greedy decoding measures the quality of the argmax path, not the per-task probability p. A is wrong because greedy decoding is not random; it deterministically takes the argmax. B overstates a practical nuisance into an impossibility.

This is the construct-validity lesson (B): “can code” decomposes into distinct skills — function synthesis, bug-fixing across files, repository navigation, long-horizon autonomy — each benchmark measures one operationalization, and rankings do not transfer; a model that tops the synthesis tables can sit mid-pack on repository tasks. The most tempting wrong answer is D, but precisely because the constructs differ, a synthesis score is not a transferable bound on repository ability — that is the extrapolation construct validity forbids. A misreads a real 45-fold gap as mere miscalibration. C invents a data explanation the lecture never gives and that the 1.96% of late 2023 contradicts.

The grader is purely mechanical (A): paste the prompt, the stored completion, the hidden check function, and one call line, run the whole thing as an ordinary Python program, and pass means it ran check without raising. It never parses, reformats, or judges the code. The most tempting wrong answer is C, but it is explicitly rejected — a body that looks nothing like the reference yet passes every assertion is correct, full stop. B anthropomorphizes the grader into a reviewer with taste, the opposite of what makes it trustworthy. D describes differential testing against a reference — the EvalPlus technique of Chapter 6, not HumanEval’s original grader.

“A saturated benchmark does not tell you the models are equally good; it tells you the test has run out of headroom” (C). Above 90% the instrument, not the population, has hit its ceiling, and a number quoted from a saturated benchmark is a fact about the benchmark. The most tempting wrong answer is A — exactly the sloppy reading the lecture says is “on half the Claim Board.” B is wrong because on a 164-problem set the remaining differences are a handful of tasks well inside sampling noise. D overstates: the template was right for 2021, when the open question was whether a model could write a correct standalone function at all.

The lecture decomposes measured success as p(correct) plus p(pass but wrong): the second term is the chance of passing while being wrong, invisible from inside the benchmark because every statistical tool operates downstream of the oracle (D). EvalPlus surfaced it the only way possible — build a stronger oracle and take the difference — yielding “a benchmark score is only as trustworthy as its grader.” The most tempting wrong answer is C, but it conflates this with contamination; the completions were frozen, so the experiment isolates the grader, not the training data. A inverts the finding — those completions were genuinely wrong, not unfairly tricked. B is wrong because re-grading frozen completions is deterministic.

The fail-to-pass filter is crucial because it proves the tests genuinely encode the fix — a test that flips red-to-green across the change is tied to that change, not riding along as cleanup (C). The two sets then split the meaning of “fixed”: FAIL_TO_PASS (red→green) is progress, the bug is gone; PASS_TO_PASS (green→green) is the regression guard that nothing working broke. The most tempting wrong answer is A, but it misattributes the filter to issue clarity — the lecture stresses the mechanical filters check none of that. B names a different filter, reproducibility, and invents speed/memory roles. D is wrong: the sets judge the model’s patch, and the gold patch is withheld from the model.

The surprise is the lesson (A): “curation did not deflate the number; it inflated it — because the flaw being removed was unfairness to the model, not generosity toward it.” An instance whose tests reject any correct patch is not a hard task but a broken device that deflates every score equally; remove those and the rest measure capability, so the score doubles — the Chapter 6 grader lesson run in reverse. The most tempting wrong answer is D, but that describes leakage, which is real elsewhere and is not why curation raised this score. B is wrong because a smaller denominator does not by itself raise a percentage. C invents annotator cheating the lecture never alleges.

The resolution is deliberately uncomfortable (B): “both numbers are real, and neither is the model’s score.” The ~77% bash-only figure estimates model capability under a fixed, minimal harness; the ~95% vendor figure is a system capability — model plus scaffold plus best-of-N retries — at a price nobody states. They answer different questions, and the headline never says which. The most tempting wrong answer is A, but it just picks one number as “the real one,” exactly the mistake the chapter dismantles — and C makes the same mistake from the other side. D is wrong: an eighteen-point gap on the same 500 tasks is a scaffold effect, not sampling noise, and would reappear on re-runs.

The chapter’s central move (D): “every load-bearing ingredient of every SWE-bench scaffold is a prompting pattern from Lecture 6, institutionalized” — the one-shot patch demo is few-shot format teaching, the agent loop is chain-of-thought with tool calls, and best-of-N plus a verifier is self-consistency wearing a suit, at 10–100× inference cost. The most tempting wrong answer is C, but a scaffold wraps the model without touching its weights — no fine-tuning happens. A is wrong because the named open scaffolds (SWE-agent, Agentless, OpenHands, mini-SWE-agent) are readable research artifacts. B is contradicted by the eighteen-point bash-only-versus-vendor gap on the same frontier generation.

The lecture is precise (C): “contamination lives in the weights and leakage lives in the input.” When a maintainer pastes a suggested diff into the issue thread and SWE-bench keeps that text as the prompt, emitting the fix is ordinary in-context copying — “the model did not resolve the issue, it transcribed the answer pasted into the question.” The most tempting wrong answer is B, but it confuses leakage with the separate weak-test finding. A is wrong: the two terms are carefully separated. D invents a model-type split the lecture never draws.