Home

08 Lecture 8 Quiz: Measuring the Machine

June 2026 (3411 Words, 19 Minutes)

quiz evaluation benchmarks swe-bench pass-at-k

New: Interactive Quiz Available!

Want to track your progress and get instant feedback? Try our new interactive version with progress tracking, immediate explanations, and score calculation.

Start Interactive Quiz →

Instructions

The quiz follows the lecture’s arc: Questions 1–6 build the measurement toolkit — the four-component decomposition, the Claim Board, why a stochastic system is hard to measure, why “can code” is not one construct, HumanEval, and saturation. Questions 7–12 turn to how the benchmark is built and what scrutiny reveals — the EvalPlus grader, SWE-bench construction, the Verified variant, the two numbers, the scaffold problem, and leakage versus contamination.

Question 1: The Four Components

The lecture says a benchmark score is “a measurement of a system — model + scaffold + grader + data — under specific, gameable conditions.” What does this framing tell you to do when you read “~95% on SWE-bench Verified”?

A) Ask which scaffold was used, how many attempts it took, how strict the grader was, and whether the test data leaked — because all four can move the number
B) Treat the number as a fixed property of the model’s weights, like a chip’s advertised clock speed
C) Ignore the number entirely — since four components produce it, it is meaningless and going by vibes is more honest
D) Trust it more than usual, because four independent components each had to agree on it before it could be published

Show Answer

Correct Answer: A

The whole lecture decomposes a score into model, scaffold, grader, and data, and each component supplies “the question you learn to ask,” so the figure is a measurement of a system under conditions, not a property of the weights (A). The most tempting wrong answer is B — the “spec-sheet” reading the launch post invites, treating the number like a clock speed — which the lecture names a category error. C is benchmark nihilism, explicitly rejected as “a different way of not looking.” D inverts the framing: the four components are not independent validators that cross-check one another; they are four separate places where the number can be inflated or deflated.

Question 2: The Claim Board

The Claim Board asks “should I believe this?” about model claims, and its verdicts carry an “as of” date. The earlier Mystery Board’s answers did not. Why the difference?

A) The Claim Board is digital and the Mystery Board was only physical, so only the first one can be timestamped
B) The Mystery Board’s answers were proofs and the Claim Board’s are just opinions, and only proofs can last
C) The Mystery Board explained why something happens, and a mechanism does not expire; the Claim Board judges claims about a fast-moving market, which can change
D) The Claim Board expires because students re-vote on every claim each week

Show Answer

Correct Answer: C

The lecture states the polarity flip directly (C): the Mystery Board collected “why does this happen?” questions whose mechanistic answers stay answered — tokenization will still hide characters next year — while the Claim Board asks “should I believe this?” about claims tied to a moving market, where the number may be superseded, the model retired, the benchmark withdrawn. The most tempting wrong answer is B, but a mechanistic explanation is not a mere “opinion versus proof” distinction; the point is that mechanism is permanent while market facts are not. A confuses the dual format: both boards lived as a physical board plus a digital page. D invents a weekly re-voting rule; claims expire because the world moves, not because the vote repeats.

Question 3: Why Evaluation Is Hard

“A benchmark run does not observe capability; it samples it,” the lecture says. A student replies: “Then I will set temperature to 0 so the run is deterministic — now the score IS the capability.” Why does the lecture disagree?

A) Greedy decoding still picks tokens at random, just with a narrower spread, so a single run is still one random draw
B) Temperature 0 is impossible to reach on real hardware, so genuine determinism is never available to measure with
C) The capability just is the argmax path, so a temperature-0 score reads it off exactly — which is why such scores are the gold standard
D) Even at temperature 0, hosted inference still drifts across servers, and greedy decoding only measures the single most-likely path — one point of the distribution, not the whole model

Show Answer

Correct Answer: D

The lecture gives two answers (D): production temperature 0 is only mostly deterministic — inference stacks drift across replicas, batches, and silent updates — and even perfect greedy decoding evaluates one privileged point, the argmax path, discarding the rest of the distribution Lecture 5 taught you the model actually is. The most tempting wrong answer is C: it sounds rigorous, but greedy decoding measures the quality of the argmax path, not the per-task probability p. A is wrong because greedy decoding is not random; it deterministically takes the argmax. B overstates a practical nuisance into an impossibility.

Question 4: Construct Validity

By 2023, top models scored near 90% on function-writing benchmarks, but the best system fixed under 2% of real repository issues — in the same year. What does the lecture conclude?

A) The repository benchmark must have been miscalibrated, since the same models cannot really differ that much in ability
B) Writing a function and fixing a repository bug are different skills; no benchmark measures ‘coding ability’ as a whole, and a ranking on one does not carry over to the other
C) The models simply had not been trained on enough repository data yet — a gap that closed within about a year
D) Repository work includes function writing, so a high synthesis score is a safe upper bound on repository ability, and the two should track each other

Show Answer

Correct Answer: B

This is the construct-validity lesson (B): “can code” decomposes into distinct skills — function synthesis, bug-fixing across files, repository navigation, long-horizon autonomy — each benchmark measures one operationalization, and rankings do not transfer; a model that tops the synthesis tables can sit mid-pack on repository tasks. The most tempting wrong answer is D, but precisely because the constructs differ, a synthesis score is not a transferable bound on repository ability — that is the extrapolation construct validity forbids. A misreads a real 45-fold gap as mere miscalibration. C invents a data explanation the lecture never gives and that the 1.96% of late 2023 contradicts.

Question 5: HumanEval

The lecture calls HumanEval’s grader “dumber, and therefore far more trustworthy, than students expect.” How does it actually decide pass or fail?

A) It glues the prompt, the completion, the hidden test, and one call line into a single program, runs it, and passes if it runs without error
B) It reviews the completion’s logic and style against a rubric of good code, then gives a pass-or-fail verdict
C) It compares the generated function to the reference solution and passes it whenever the two look similar enough
D) It runs the model’s function and the reference solution on random inputs and passes only if their outputs always match

Show Answer

Correct Answer: A

The grader is purely mechanical (A): paste the prompt, the stored completion, the hidden check function, and one call line, run the whole thing as an ordinary Python program, and pass means it ran check without raising. It never parses, reformats, or judges the code. The most tempting wrong answer is C, but it is explicitly rejected — a body that looks nothing like the reference yet passes every assertion is correct, full stop. B anthropomorphizes the grader into a reviewer with taste, the opposite of what makes it trustworthy. D describes differential testing against a reference — the EvalPlus technique of Chapter 6, not HumanEval’s original grader.

Question 6: Saturation

By 2023, top models scored near 90% on HumanEval. The lecture warns against a “sloppy reading” of this. What is the correct reading?

A) The models have become genuinely equally good at coding, so a cluster of high scores at the top is a real tie between them
B) The few remaining unsolved problems are the hardest ones, so differences on exactly those problems are now the most meaningful signal
C) The test has run out of headroom: above 90% it can no longer tell a competent model from an exceptional one — the instrument hit its ceiling, not the models
D) Saturation proves the benchmark was always too easy and never measured anything useful in the first place

Show Answer

Correct Answer: C

“A saturated benchmark does not tell you the models are equally good; it tells you the test has run out of headroom” (C). Above 90% the instrument, not the population, has hit its ceiling, and a number quoted from a saturated benchmark is a fact about the benchmark. The most tempting wrong answer is A — exactly the sloppy reading the lecture says is “on half the Claim Board.” B is wrong because on a 164-problem set the remaining differences are a handful of tasks well inside sampling noise. D overstates: the template was right for 2021, when the open question was whether a model could write a correct standalone function at all.

Question 7: The Grader (EvalPlus)

EvalPlus re-graded the exact same (frozen) GPT-4 answers with a much stronger test suite, and HumanEval’s pass@1 fell from 88.4% to 76.2% — nothing about the model changed. What had the original 88.4% actually been counting?

A) The model’s true correctness, which EvalPlus then unfairly lowered by bolting on trick tests
B) Random run-to-run noise, since re-running any benchmark gives a slightly different number each time
C) The model’s ability to pass tests it had memorized during training, which the stronger suite then stripped away
D) Correctness plus a hidden extra: answers that passed the weak tests while actually being wrong — every completion that passed HumanEval but failed HumanEval+ was wrong all along

Show Answer

Correct Answer: D

The lecture decomposes measured success as p(correct) plus p(pass but wrong): the second term is the chance of passing while being wrong, invisible from inside the benchmark because every statistical tool operates downstream of the oracle (D). EvalPlus surfaced it the only way possible — build a stronger oracle and take the difference — yielding “a benchmark score is only as trustworthy as its grader.” The most tempting wrong answer is C, but it conflates this with contamination; the completions were frozen, so the experiment isolates the grader, not the training data. A inverts the finding — those completions were genuinely wrong, not unfairly tricked. B is wrong because re-grading frozen completions is deterministic.

Question 8: SWE-bench Construction

SWE-bench keeps only pull requests that have a “fail-to-pass” test: a test that fails before the change and passes after it. What does that filter guarantee, and what do the FAIL_TO_PASS and PASS_TO_PASS sets then mean?

A) It guarantees the issue text is detailed enough to solve; FAIL_TO_PASS checks the problem is clearly stated and PASS_TO_PASS checks the code compiles
B) It guarantees the repository builds years later; FAIL_TO_PASS measures the patch’s speed and PASS_TO_PASS measures its memory use
C) It proves the tests really capture the fix (not unrelated cleanup); FAIL_TO_PASS shows the bug is gone, and PASS_TO_PASS shows nothing that worked before is now broken
D) It guarantees the gold patch is minimal; both sets simply re-run the maintainer’s original patch to confirm it still applies

Show Answer

Correct Answer: C

The fail-to-pass filter is crucial because it proves the tests genuinely encode the fix — a test that flips red-to-green across the change is tied to that change, not riding along as cleanup (C). The two sets then split the meaning of “fixed”: FAIL_TO_PASS (red→green) is progress, the bug is gone; PASS_TO_PASS (green→green) is the regression guard that nothing working broke. The most tempting wrong answer is A, but it misattributes the filter to issue clarity — the lecture stresses the mechanical filters check none of that. B names a different filter, reproducibility, and invents speed/memory roles. D is wrong: the sets judge the model’s patch, and the gold patch is withheld from the model.

Question 9: The Variant Family

OpenAI built SWE-bench Verified by removing underspecified issues and unfair tests. GPT-4o’s score (with the Agentless scaffold) then rose from 16% to 33.2%. Why did removing tasks make the score go up?

A) The removed tasks were unfair: their tests reject even correct patches, so they were broken measuring devices dragging every score down. Remove them and the rest can measure real capability
B) The curated set was smaller, so each solved task was worth more percentage points — the score rose as a pure arithmetic artifact
C) The reviewers quietly swapped the hardest tasks for easier ones during review
D) Paying humans to check each task leaked the fix into the issue text, so the model could just read the answer out of the problem

Show Answer

Correct Answer: A

The surprise is the lesson (A): “curation did not deflate the number; it inflated it — because the flaw being removed was unfairness to the model, not generosity toward it.” An instance whose tests reject any correct patch is not a hard task but a broken device that deflates every score equally; remove those and the rest measure capability, so the score doubles — the Chapter 6 grader lesson run in reverse. The most tempting wrong answer is D, but that describes leakage, which is real elsewhere and is not why curation raised this score. B is wrong because a smaller denominator does not by itself raise a percentage. C invents annotator cheating the lecture never alleges.

Question 10: The Two Numbers

As of June 2026, the same models score about 77% on SWE-bench Verified’s minimal “bash-only” track and about 95% on the open leaderboard with vendor scaffolds. How does the lecture answer “which number is real?”

A) The 95% is the model’s real capability, and the 77% is an artificially crippled measurement from a minimal loop nobody would actually deploy
B) Both numbers are real, and neither is ‘the model’s score’: one measures the model under a fixed minimal harness, the other a whole system — model plus scaffold plus retries — at an undisclosed cost
C) The 77% is the only honest number, and the 95% is just inflated marketing to be discarded
D) The gap is just sampling noise; re-run either track and the two numbers would quickly converge

Show Answer

Correct Answer: B

The resolution is deliberately uncomfortable (B): “both numbers are real, and neither is the model’s score.” The ~77% bash-only figure estimates model capability under a fixed, minimal harness; the ~95% vendor figure is a system capability — model plus scaffold plus best-of-N retries — at a price nobody states. They answer different questions, and the headline never says which. The most tempting wrong answer is A, but it just picks one number as “the real one,” exactly the mistake the chapter dismantles — and C makes the same mistake from the other side. D is wrong: an eighteen-point gap on the same 500 tasks is a scaffold effect, not sampling noise, and would reappear on re-runs.

Question 11: The Scaffold Problem

The lecture says a SWE-bench leaderboard entry “measures a system, not a model.” It then makes a sharper point about where a scaffold’s power comes from. What is it?

A) Scaffolds are proprietary black boxes whose behaviour cannot be related to anything you already know from earlier lectures
B) Scaffolds matter only for weak models; once a model is strong enough, the harness makes no measurable difference
C) Scaffolds work by quietly fine-tuning the model’s weights on the benchmark’s tasks during the evaluation run itself
D) Each key scaffold part is a Lecture 6 prompting pattern turned into software — a one-shot demo (few-shot), a reason-act-observe loop (chain-of-thought), and best-of-N with a verifier (self-consistency)

Show Answer

Correct Answer: D

The chapter’s central move (D): “every load-bearing ingredient of every SWE-bench scaffold is a prompting pattern from Lecture 6, institutionalized” — the one-shot patch demo is few-shot format teaching, the agent loop is chain-of-thought with tool calls, and best-of-N plus a verifier is self-consistency wearing a suit, at 10–100× inference cost. The most tempting wrong answer is C, but a scaffold wraps the model without touching its weights — no fine-tuning happens. A is wrong because the named open scaffolds (SWE-agent, Agentless, OpenHands, mini-SWE-agent) are readable research artifacts. B is contradicted by the eighteen-point bash-only-versus-vendor gap on the same frontier generation.

Question 12: The Manual Audit

The SWE-Bench+ audit found that in 32.67% of passing instances, the fix was already written in the issue report or its comment thread. The lecture calls this leakage, not contamination. What is the distinction?

A) They are two names for the same thing, and the lecture uses the two words interchangeably
B) Leakage means the test suite is too weak to catch a wrong patch, while contamination means the issue text is too vague to act on
C) Contamination lives in the training weights; leakage lives in the input — here the fix was pasted right into the prompt, so the model just copied it
D) Contamination only affects open-source models, while leakage only affects closed vendor models

Show Answer

Correct Answer: C

The lecture is precise (C): “contamination lives in the weights and leakage lives in the input.” When a maintainer pastes a suggested diff into the issue thread and SWE-bench keeps that text as the prompt, emitting the fix is ordinary in-context copying — “the model did not resolve the issue, it transcribed the answer pasted into the question.” The most tempting wrong answer is B, but it confuses leakage with the separate weak-test finding. A is wrong: the two terms are carefully separated. D invents a model-type split the lecture never draws.

Scoring Guide

Key Takeaways to Remember

  1. A benchmark score is a measurement of a system, not a property of a model — model + scaffold + grader + data, under specific and gameable conditions; decompose every headline into those four parts before believing it
  2. The Claim Board asks “should I believe this?” with expiry dates — unlike the Mystery Board’s permanent mechanistic answers, a claim about a model is a claim about a moving market, so every verdict carries an “as of” stamp
  3. A run samples capability, it does not observe it — every pass/fail is one draw from a per-task probability p; even greedy decoding only measures the single most-likely path, not p
  4. “Can code” is not one construct — function synthesis, repository bug-fixing, navigation, and long-horizon autonomy are different skills, and rankings do not transfer between them (90% synthesis vs <2% repository in the same year)
  5. The grader runs the code, it does not judge it — assemble prompt + completion + hidden test + call line, execute, and pass = ran without raising; the harness shapes the score as surely as the model does
  6. A saturated benchmark is a fact about the benchmark — above ~90% the instrument, not the population, has hit its ceiling; it can no longer rank models
  7. A benchmark score is only as trustworthy as its grader — EvalPlus dropped GPT-4 from 88.4% to 76.2% by strengthening the tests alone, exposing the invisible “passed-but-wrong” term
  8. SWE-bench is mined, not authored — merged PRs become graded instances via the fail-to-pass filter; FAIL_TO_PASS is the progress signal, PASS_TO_PASS the do-no-harm signal
  9. Every variant is a confession — and Verified’s curation raised GPT-4o from 16% to 33.2%, because the flaw removed was unfairness to the model, not generosity toward it
  10. 77 and 95 are both real, and neither is “the model’s score” — the bash-only track holds the scaffold constant to estimate the model; vendor scaffolds add best-of-N retries at 10–100× cost; an eighteen-point gap hides inside the same headline
  11. A scaffold is Lecture 6 prompting frozen into software — few-shot format demos, reason-act-observe loops, and best-of-N with a verifier are demonstrations, chain-of-thought, and self-consistency institutionalized
  12. Leakage is in the input, contamination is in the weights — 32.67% of audited “resolved” instances had the fix pasted in the issue thread; the model transcribed the answer rather than solving the problem
© 2026 Dominik Müller   •  Powered by Soopr   •  Theme  Moonwalk