Home

08 Build and Break a Grader: Discovering Evaluation for Code Generation

June 2026 (2899 Words, 17 Minutes)

exercise evaluation code-generation test-oracle grader pass-at-k humaneval python copilot scaffold group-work lecture-8

An entry experience, not a recap. This is the first hands-on activity for Lecture 8, and it is deliberately a discovery exercise: you try to build an evaluation for a code-generation model before the lecture hands you the field’s answers, so that when chapters 3–6 name a problem, you already have a bruise where it hit you. The whole thing runs in class, in groups of 3–4, in roughly one 90-minute block — brainstorm → build → put a model under test → break a grader → present. Everything is in Python. It ends with each group presenting, and the class noting down everyone’s points together into a shared “evaluation playbook” — there is nothing to hand in and nothing to grade. There is no single right answer here — the point is the thought process, the shift into an evaluation mindset. Do it cold, before the lecture content: you will leave with more open questions than answers, and that is the design. The lecture that follows is where those questions get their names and their resolutions — this activity just makes sure you have already felt the problem they solve.

Goal

Your group will design, by hand, a tiny benchmark that answers one question with a number instead of a vibe: “Can this model write correct Python?” You will pick tasks, write graders, put a real model (via Copilot) under test, and then — the heart of the exercise — discover how easily a grader is fooled and what it takes to make one you trust. By the end you will have re-derived, from your own experience, why evaluating code generation is harder than it looks and what separates a grader that measures correctness from one that merely measures confidence.

You already own the key tool for this. You have written unit tests; you know about statement and branch coverage; you know from experience that a green test bar does not prove code is correct — only that it is correct as far as your tests can see. This exercise turns that single piece of knowledge into a way of thinking about machine-graded evaluation. Keep it in front of your mind: grading an AI’s code is just testing — but now the thing under test is a tireless author of plausible-looking wrong answers.

Learning objectives

After this exercise, you should be able to:

Lay out the design space of a code-generation evaluation — task source, specification, sampling, answer extraction, the grader, and how results become one number — and name the decision each step forces.
Derive the function-completion + executable-test design not because you were told to, but because a realistic constraint (automatic, repeatable, trustworthy across models) rules out the alternatives.
Produce Python that is wrong but passes a real test suite, and explain in coverage terms exactly why it slipped through.
Notice that “the model via Copilot” is not the bare model, and reason about what the tooling in between does to your measurement.
Strengthen a grader and justify “strong enough” in the vocabulary you already have — coverage, edge cases, and a planted bug (a mutation) your tests should catch.
Articulate what makes a grader good and what can still go wrong even when every test is green.

Format at a glance

Everything below is in class. Times are a realistic guide for a single ~90-minute block; treat them as soft — the presentation in Phase 6 is the part to protect.

Phase	Time	You produce
1. The €50 referee — brainstorm	~10 min	A raw, messy list of how you'd measure it
2. One constraint, build two tasks	~30 min	Two Python tasks (each: spec + grader + reference)
3. A model under test (Copilot)	~12 min	5 graded completions + the Copilot question answered
4. Break your neighbour's grader	~18 min	A wrong-but-passing solution
5. Strengthen + "good enough"	~8 min	A stronger grader + your reasoning for it
6. Present + build the playbook together	~15 min	A 3-min presentation; the class's shared evaluation playbook

Reading the phases. Each phase below ends with a bold Produce: line — the one concrete thing your group should have in hand before moving on. They chain: each phase’s output is the next phase’s input.

Ground rule — Python only. Every task is a plain Python function; every grader is Python (assert/pytest); every reference solution is Python. One language keeps the graders comparable when you swap them, and removes “which language” as a confound — it is also exactly the choice the benchmark you will meet next made.

Phase 1 — The €50 referee (~10 min)

Scenario. Two colleagues have bet €50. One says the new coding model “can basically code”; the other says “it just autocompletes Stack Overflow.” You are the referee. You have one afternoon and must settle it with a number and a method you could defend, not an anecdote.

Brainstorm how you would measure it. Do not converge yet — get the disagreements on the table. Use these only if you stall:

Tasks. What would you ask the model to do? Where do those tasks come from — and could the model have already seen them?
Correctness. When the model answers, who or what decides it is right? A human? A reference answer? Something else?
Attempts. One shot or many? If it succeeds on the 3rd of 5 tries, did it “succeed”?
“Good”. Good at what — correct? fast? cheap? Pick what your €50 actually rides on.

Capture your group’s rawest version on paper. You will be slightly wrong in interesting ways; that is the point.

Produce: one sheet with your group’s measurement ideas and the points you disagreed on. Nothing needs resolving yet — Phase 2 forces the choices.

Phase 2 — One constraint, build two tasks (~30 min)

Your manager adds the constraint that real benchmarks live under:

It has to be automatic and repeatable. No human reading each answer. It must run unattended on 1,000 tasks overnight, and you must be able to re-run the exact same evaluation on a different model next month and trust the comparison.

Spend two minutes on what that constraint kills from your Phase 1 ideas — and what it forces. Then build two Python tasks:

Task A — a provided type, your design. Your group is assigned (or draws) one task type from the bank below. The type is all you are given — you still decide the exact function, write the specification (signature + docstring, precise enough that two strangers agree on every output), write the grader (the tests that define pass/fail), and write one correct reference solution. This is the task you will swap and have attacked, so make the grader as strong as you think it needs to be.

Task B — your own. Invent a second function entirely of your own choosing and design it the same way (spec + a few tests + reference). Keep it small — this one is for putting the model under test in Phase 3.

Task-type bank (Python)

#	Type (this is all you get)	Flavour, not a spec
A1	String processing	transform or analyse a string
A2	Number / list crunching	compute something over a list of numbers
A3	A validator	decide whether an input is well-formed (returns a bool, or raises)
A4	A small parser / converter	turn one representation into another
A5	A real-world calculation	arithmetic with a wrinkle — rounding, leap years, units
A6	An aggregation	group, tally, or summarise a collection

Keep the reference solution from Task A safe — you will need it, and you will notice why you need it. Then swap Task A (spec + grader, but NOT your reference solution) with another group.

Produce: two designed Python tasks — Task A (spec + grader + reference, the careful one) and Task B (spec + a few tests + reference, lighter). Your Task A’s spec + tests are now in a neighbour’s hands (reference kept back), and theirs are in yours.

Phase 3 — Put a model under test, via Copilot (~12 min)

You all have GitHub Copilot. Pick one of the models Copilot offers and write down which one. Prompt it to solve your own Task B and collect 5 completions (vary the prompt slightly or regenerate). Run each through your grader and record pass/fail. Then look harder than your grader did:

Did any completion pass but is actually wrong? (Check by hand against your spec.)
Did any correct completion fail for a silly reason — wrong output format, extra prose, a print instead of a return, trailing whitespace? (That is your grader being brittle — the opposite failure.)
The model rarely returns only the function. How did you get the code out of its answer — and could that step change the verdict?

Now the question to actually argue about — write your answer down:

You are measuring Copilot, not the bare model. What sits between you and the model? What context might Copilot be feeding it that you did not choose — including, possibly, your open files? If your test file or reference solution is open in another tab while Copilot answers, what has that done to your measurement? Would two of you, on two machines, even get the same number?

Produce: your 5 Copilot completions for Task B with pass/fail recorded (and which model you picked), plus a written answer to the boxed question above.

Phase 4 — Break your neighbour’s grader (~18 min)

You hold another group’s Task A (spec + tests, no reference solution). Your job: produce a Python implementation that is wrong but passes their tests.

By hand. Write the laziest, most obviously-incomplete implementation that still goes green. Hint: what do their tests never check — an empty input? a boundary? a second example? a duplicate?
With the model. Ask Copilot directly: “Here is a Python function spec and its tests. Write an implementation that passes all the tests but is deliberately wrong or incomplete.” See what it finds.

Send the breaking solution back to the group that wrote the grader. (This is exactly what a weak model does by accident — and what a model rewarded only for “passing” learns to do on purpose.)

Produce: one wrong-but-passing implementation of your neighbour’s Task A, handed back to them — plus a one-liner on what their tests never checked.

Phase 5 — Strengthen, and decide when you are done (~8 min)

Take what your neighbour did to your Task A grader and fix it. Add tests until the cheat fails. Then answer the hard question honestly:

How do you know it is strong enough now? “It catches that cheat” is not “it catches the cheats I haven’t seen.”
Reason in the terms you already have: which lines and branches of your reference did the cheat exploit that your tests never executed? If you planted a small deliberate bug — a mutation like i+1 for i, or < for <= — in a correct solution, would your tests catch it?
Notice the quiet engine of all of this: every new test needs the right answer for its input. Where did that answer come from each time? (Look at the reference you kept in Phase 2.)
And the attempts thread: in Phase 3 you gave the model 5 tries. Would you score it “passed if ≥1 of 5 works” or “passed only if all 5 work”? Pick the one that matches your €50 bet, and say why the two numbers can differ enormously.

Produce: your strengthened Task A grader, one sentence on why it is now strong enough (in coverage / mutation terms), and your pick of “≥1 of 5” vs “all 5”.

Phase 6 — Present, and build the class’s evaluation playbook together (~15 min)

This is the point of the whole activity, so give it the most room. Each group gives a quick 3-minute report-out — not a polished result, but what you found and what you could not settle:

Your Task A and the cheat that beat a grader (yours or your neighbour’s) — why did it pass?
The fix, and how you decided it was enough — or why you are not sure it is.
Your group’s take on the two big ones: what makes a grader good? and what can still go wrong even when every test is green?
What were you actually measuring through Copilot?
One question you could not answer.

As groups speak, the class builds one shared board together — the instructor writing up every point as it is offered: each new principle for a good grader, each new way a grader can be fooled, each cheat, each open question. The goal is the collective picture no single group reached alone; by the end the board is the class’s first evaluation playbook, written by the room. There is nothing to hand in — this board, and the shift in how you think, is the whole output. Leave the open questions on it unanswered: they are the syllabus for what comes next, and the point is to look back, after the lecture, and count how many it takes down.

Produce (together, not handed in): the class’s shared evaluation playbook — every group’s good-grader principles, failure modes, and open questions on one board.

Instructor guide

Recommended placement — run it cold

This is a priming / productive-failure activity, so run it before any Lecture 8 content — ideally as a standalone block the session before, or as the very opening of the session. It needs no prior lecture material: the €50 scenario is self-contained. The whole value is that students wrestle with the problem and form their own (often wrong) models first, so the lecture later lands on prepared ground. Do not resolve the activity with the answers. Each phase is engineered so groups walk into a wall that a later chapter names — your job here is to let them feel the wall and leave it standing. The resolutions belong to the lecture.

Facilitation notes

Phase 1 should feel slightly chaotic. Don’t steer toward the “right” answer; harvest the disagreement. First ideas — “build an app and see if it works”, “compare to a reference answer”, “let a human grade it”, “let another AI grade it” — are each a thread.
Phase 2’s constraint is the pedagogical engine. “Automatic + repeatable + trustworthy across models” is what forces executable tests and a fixed task set with a reference answer. If a group lands on “let another AI grade it”, that is not wrong — note it for a later callback as a real technique (LLM-as-judge) with measurable biases, and let “trust the comparison next month” expose its fragility on the spot.
The provided type (Task A) is the comparability anchor — because every group’s swapped task is “a validator”, “a parser”, etc., the breaks rhyme and the lecture’s callbacks land cleanly. Assign types so each is held by at least two groups.
Phase 3’s Copilot question is the sleeper. Most groups will not realise they are testing a system, not a model, until you ask. Watch for the leak: a group with its test_*.py or reference open in the editor while Copilot writes the solution has just fed the answer key into the model’s context, live — a named failure mode they will meet later. That is a gift; bank it for the callback.
Keep functions small; a rough task is fine. The break in Phase 4 does not need a polished benchmark.
Both failure directions teach. A group whose grader is too brittle (it rejects correct solutions) has learned as much as one whose grader is too weak (it passes wrong code) — make sure the brittle direction gets airtime in Phase 6 too; it is the half students forget.
There is nothing to grade. This is a mindset-builder, not a deliverable. The output is the shared board the class writes together in Phase 6 — credit participation, not polish.

Your callback cues — hints here, the precise map in your own notes

This is not a debrief you deliver, and it is not a forecast you hand out — it is your cue sheet for calling back to the activity once the lecture is under way (“remember when your grader passed the obviously-wrong code? that is this.”). It is deliberately declawed: the right column hints rather than names, so a student who reads ahead in this file gets anticipation, not the punchlines. Keep the precise bruise → chapter mapping in your own lecture notes — you know where each one lands — and summon the row when the lecture reaches it. The artifacts the groups made (the cheats, the broken-then-fixed graders, the Questions Board) become the lecture’s running examples.

What the group hit	What it turns into later (your cue — name it only in class)
"Our tests passed obviously-wrong code"	The lecture's single most important lesson — a whole chapter exists for it.
Strengthening until a planted bug dies	Your coverage / planted-bug instinct is exactly the right tool; the field has formal names for it.
"Every new test needed the right answer → we ran our reference"	You reinvented a standard technique without being told its name.
"We were testing Copilot, not the model"	A theme that returns again and again: the wrapper is part of the measurement.
Copilot reading your open test file / reference	A notorious way benchmark numbers get quietly inflated.
Correct code failing on format / prose / `print` vs `return`	The unglamorous plumbing step that silently moves scores.
"≥1 of 5" vs "all 5"	Two metrics that look almost identical and pull in opposite directions.
"Could the model have seen this task already?"	The crisis that closes the first half of the lecture.
The function + spec + tests design itself	You re-derived the design the whole field started from.
Deliberately writing wrong-but-passing code	Doing this on purpose is a named, real problem.

The single sentence the lecture eventually lands — not you, and not during this activity: a benchmark score is only as trustworthy as its grader. By the time it is said aloud, the students will have already lived it, because they broke a grader and rebuilt it.