Home

09 The Model Market: Suppliers, Platforms, and the Price of a Token

June 2026 (26399 Words, 147 Minutes)

ai llm model-market pricing token-economics monitoring act-3

1. Introduction

Welcome to Week 9. Lecture 8 did not end so much as pause mid-measurement. The Claim Board is still on the wall, most of its claims still pinned, because one of the nine questions stayed wide open: question 6 — at what cost? It was the odd one out on the checklist, the only question that usually cannot be answered from public materials at all, because the headline coding leaderboards print no cost column. Lecture 8 §13.3 named it the missing column and closed on it: a score without a price tag is half a measurement. Today supplies the other half. You have learned to read the score; today you learn to read the bill.

You did not come empty-handed, either. The homework asked you to bring one closed Locust issue whose fail-to-pass property you verified with your own hands — check out the commit before the fix, watch the test fail; check out the merge, watch it pass. Keep it on the desk all day. It is not decoration: in the lab (chapter 14) you will first estimate, and then measure, what an agent attempt on that issue costs, and everything between here and there exists to make that estimate possible.

Re-anchor the scene we left. The “~95% on SWE-bench Verified” headline that opened last week was — as far as anyone outside the reporting lab can tell — produced by best-of-\(N\) machinery running on the order of 10–100× the inference cost of a single attempt (Lecture 8, chapter 10), and nobody printed the bill. The two numbers that ended Block A there — seventy-seven and ninety-five — differ not only in scaffold but in an undisclosed multiple of spend.

That omission is not a bookkeeping detail. Cost is not an accounting afterthought that someone in a finance department reconciles later; it is the suppressed axis of every leaderboard you have ever read, and a ranking re-plotted with that axis restored can reorder. Nor is the axis abstract for you: within a year of leaving this room you will be paying for models one of two ways — a per-token API bill, or a subscription whose usage windows meter the same token flows in a different currency (chapter 13 decides between them). Either way, every agentic session you run has a number attached, and the difference between knowing that number beforehand and discovering it afterwards is the difference between engineering and gambling.

That gives today its thesis, the sentence to carry to the exam: a model choice is a price–performance decision under constraints, and both the price and the performance are quantities you can compute before you spend — by tonight you can estimate the cost of an agentic task before running it, explain why that cost grows quadratically with the length of the conversation, and choose between models on cost-per-solved-task rather than cost-per-token.

Three abilities, and none requires new machinery — the bill decomposes into parts you already own. Lecture 4’s tokenizer decides how many tokens your text becomes, and charges German prose a premium for being German. Lecture 5’s KV cache is why providers price output tokens above input tokens — and why they can afford to sell cached input at a discount. Lecture 6’s test-time compute is why reasoning models bill you for thinking you never read. And Lecture 7’s orchestration loop — the entire conversation re-sent on every call — is the seed of the quadratic that chapter 9 grows. Today’s work is connecting each mechanism to its line item.

Here is the contract for the day — the three abilities, mapped to where each is built and where you will apply it:

By tonight you can… The machinery Where it is built Where you apply it
Estimate what an agentic task will cost — before running it A counting recipe (what sits in the prefix, what each turn adds) plus one closed-form formula Chapters 9 and 11 Your Locust issue, in the lab (chapter 14)
Explain why session cost grows quadratically with conversation length The API is stateless: every call re-sends the whole conversation — Lecture 7's loop, priced per token Chapter 9, with caching re-shaping the bill in chapter 10 Every agent session you will ever run — and pay for
Decide between models on cost-per-solved-task, not cost-per-token A six-step decision framework fed by the supplier map, the platform tour, and the pricing table Chapters 3–8, assembled in chapter 7 The pricing claims of chapter 2 — and your next tool decision at work

Two movements deliver on that contract, matching the two 90-minute blocks in the room — and the seam between them is engineered, not incidental.

Block A — chapters 2 through 7 — is the market. Chapter 3 maps who sells tokens: a snapshot with a printed expiry date. Chapter 4 splits the map along its deepest seam — open weights versus proprietary API — which for engineers working in Germany is not a philosophical preference but a question of data residency, works councils, and version pinning. Chapter 5 tours the platforms that compare the sellers, and is honest about what each one cannot tell you. Chapter 6 teaches you to read a model card the way Lecture 8 taught you to read a leaderboard: footnotes first. And chapter 7 assembles all of it into a six-step decision framework — which stalls, deliberately, on its decisive step, because computing a cost requires arithmetic Block A never provides. That is where the break falls.

Block B — chapters 8 through 13 — is token economics, and it opens by paying Block A’s debt: the pricing table and what is structural about it (chapter 8); the quadratic growth of an agent session’s input (chapter 9) — the most important arithmetic of the week; caching, the one lever that changes the bill’s shape rather than its rate (chapter 10); predicting a task’s cost before you run it (chapter 11); monitoring what you actually spent (chapter 12); and the subscription-versus-API decision (chapter 13). Chapter 14 is the lab, where your Locust issue finally gets its price.

The course’s running diagram is on screen today as well — and for the first time since it started growing, it does not grow. LLM-in-One-Picture stays at v1.4, the evaluation-harness ring Lecture 8 drew around the orchestration layer, structurally untouched. What changes is only the labelling: every arrow that carries tokens now carries a price tag. Deliberately no v1.5 — nothing about the system changed; what changed is what you read off it.

And the geometry, as usual in this course, is the argument. Cost lives on the arrows — the token flows — not in the boxes. The model in the middle of the picture costs you nothing while it idles; what costs money is every token pushed down an arrow into it and every token that comes back out. That is why Lecture 7’s layer — the software that decides what flows along the arrows — is also where the bill is decided, and why an orchestration choice as innocent as “include the whole file” is, on the annotated picture, visibly a purchasing decision.

One cultural note before the market opens, in the same spirit as Lecture 8’s pairing of benchmark worship and benchmark nihilism. Spending on models has two well-worn failure modes of its own:

We will do neither, for the same reason we did neither worship nor nihilism last week: both are ways of not computing. The point of today is that the computation is short, and you can do it.

So start computing on yourself, now, before a single price appears on a slide: write down what you think your last Claude Code session cost — in euros; order of magnitude is enough. Nobody grades the guess. By chapter 12 you will be able to look the real number up, and the gap between guess and log is where this week’s teaching happens — the same predict–observe–explain move the course has run since Act 1.

A warning about shelf life, and then we start. This lecture contains more perishable numbers than any other in the course — prices, model names, leaderboard positions — and every one of them is stamped as of June 2026. The stamp is load-bearing: some of these figures will be wrong by exam time, and the lecture is built so that this does not matter. The durable skill is the refresh procedure, not the snapshot — knowing where each number lives and how to re-fetch it — and chapter 3 promotes exactly that from apologetic footnote to explicit learning objective.

But before any economics — before a single price — the board gets its cargo. Lecture 8 promised a second wave of claims: pricing claims, “cheapest model” assertions, a per-seat ROI pitch. They are real, they are current, and they are every bit as carefully worded as the first five. Pinning them, and scheduling their verdicts, is chapter 2.

2. The Claim Board: The Pricing Wave

This chapter exists because the board takes on new cargo today, and the loading rules deserve exactly one paragraph. The mechanic is unchanged from Lecture 8, chapter 2: real claims go up, quoted as found and uncommented; before anything is taught about any of them, you write a one-sentence prediction; each claim is eventually run through the nine questions and ends in one of the rubric’s three verdicts — verified, plausible-but-unproven, refuted-as-stated — and a claim only comes down rated. None of that machinery gets re-derived here; last week’s chapter is the reference, and the digital board page in the course repository carries every claim with its source and its date. What is new today is the genre of the cargo.

First, though, the state of the wall — because a board that never resolves anything is decoration. Two claims came down in Lecture 8’s closing pass. “~95% on SWE-bench Verified” was rated plausible-but-unproven, and look again at what the rating said was pending: scaffold disclosure and cost disclosure — half of that verdict was already this lecture’s subject, filed in advance. “The best coding model in the world” went down refuted-as-stated: not a measurable sentence, just advertising wearing the grammar of one.

Claims 3 through 5 are still pinned, and today supplies the machinery for two of them. Claim 3, “#1 on LMArena”, gets rated live in chapter 5, once you know what an arena ranking actually measures and how long a #1 tends to survive. Claim 5, “80.3% on SWE-bench Pro”, reopens in chapter 7 under the cost lens: last week’s question was whether the number can be believed; today’s question is what the capability behind it would cost you to buy, per solved task. Claim 4, the ~16-hour time horizon, stays pinned — deliberately. It is a claim about autonomy, and autonomy is the finale’s subject, not today’s.

2.1 The new claims

Four new claims go up this morning, numbered in sequence with the first wave. Wave one was about capability — what a model can do. This wave is about money: what the capability costs, who sells it cheapest, and whether a seat pays for itself. All four are real, current, and quoted as found, June 2026, and you will recognize each genre on sight — the launch post, the price-war banner, the aggregator headline, the procurement pitch.

# Claim, as pinned Where it comes from The first question it raises
6 "The cheapest frontier-quality model on the market" Open-weights launch-post genre, June 2026 Frontier-quality by which measurement — and cheapest per token, or per solved task?
7 "90% cheaper than GPT" Price-comparison marketing for an open-weights API, June 2026 Which GPT, which tier, input or output tokens — and does the gap survive caching, tokenizer differences, and tokens-per-task?
8 "The most-used model in the world — #1 by token share on OpenRouter" Aggregator-rankings headline, June 2026 Is token share usage — and whose usage never flows through a router?
9 "One subscription seat pays for itself if it saves one developer-hour per month" Per-seat ROI pitch, June 2026 Saved how, measured on whose tasks — and what does a seat actually meter?

The fourth column is the same instrument as last week: not a rebuttal, just the first loose thread. Pull it, and the claim either tightens into something checkable or starts to unravel — and you cannot know which until you pull.

The ritual is also the same, and it is the teaching instrument, not a warm-up. For each of the four, write one sentence now, before any chapter has armed you: what does this claim actually assert, and would you bet €50 that it holds? The predictions are collected and come back when the claims are rated — the predict–observe–explain mechanics carry over from Lecture 8 unchanged.

But this wave carries a twist its predecessor did not. Capabilities drift on research timescales; prices are restated on marketing timescales — a vendor can cut, restructure, or re-tier a price overnight, so the “as of” stamp on a pricing claim dates both the claim and the evidence you will rate it against. A capability verdict from last week will still be roughly right at the exam.

A pricing verdict, by contrast, can be overtaken by a webpage edit: claim 7’s “90%” may be a different percentage by July without anyone having lied. The “as of June 2026” stamp was a courtesy on wave one; on wave two it is part of the verdict itself — and the same discipline binds the lecture materials, which is why every price you will see today was re-verified against the vendors’ own pages within the week before this session.

And notice — the same observation as last week, and it matters just as much here — that none of the four is an obvious lie. Each contains a real, checkable number or a real mechanism: there is a real price page behind the “90% cheaper”, a real ranking behind the “#1 by token share”, a real metering rule behind the seat pitch. What fails, when claims of this genre fail, is rarely what they state and usually what they imply: “cheapest” implies an axis of comparison, “90% cheaper” implies the discount survives the way you actually use a model, “most-used” implies that the traffic you can see is all the traffic there is. Invented strawmen would teach you to spot strawmen. These teach you to interrogate the real thing.

Before the schedule, read the wave once more as a set. The four claims are four different denominations: a market superlative (per model), a price ratio (per token), a traffic statistic (per token flowing), an ROI promise (per developer-month). Not one of them is denominated in the unit your decisions actually turn on — the solved task — and that absence is not an accident of phrasing; it is the genre. By tonight you will be able to re-denominate all four.

2.2 Where the wave gets rated

Like last week’s, the rating schedule is the lecture’s table of contents in miniature: each claim comes off the wall exactly when the day has supplied the vocabulary it needs, and watching where each one lands tells you what kind of claim it secretly is.

Claim 6 needs the most machinery, because both of its load-bearing words are doing work. “Frontier-quality” is a Lecture 8 question — which benchmark, whose scaffold, which of the nine questions it survives — but “cheapest” is not even well-typed until you can say cheapest per what. Chapter 3 maps the sellers, chapter 8 prices them, and chapter 7 supplies the axis the claim is hoping you will not ask about: cost per solved task, not cost per token. The rating itself happens in the lab (chapter 14), with the pricing table open on the desk.

Claim 7 lives or dies by arithmetic, which is why it waits for Block B. A per-token discount is not a per-task discount: whether “90% cheaper” survives caching (chapter 10), survives tokenizers that cut the same text into different token counts (chapter 6 names that trap, chapter 11 counts it), and survives the number of tokens each model actually burns to finish a task is precisely the computation Block B teaches you to run — and the lab lets you run it on your own Locust issue.

Claim 8 is the platform claim of the wave, and it gets rated live in chapter 5 — because what a token-share ranking measures, what it cannot measure, and whose usage never appears in an aggregator’s books at all is exactly what the platform tour exists to make visible.

Claim 9 is the slow burn. What a subscription seat actually meters — and it is not developer-hours — waits for chapter 13; the verdict itself is yours to issue in the lab (chapter 14), with your own estimated-versus-measured numbers as evidence.

Every one of the four names a seller or hides one. “Cheapest on the market” presupposes you know what the market contains; “90% cheaper than GPT” presupposes you know who sells GPT and in how many tiers; even the seat pitch presupposes a vendor whose metering rules you can state. So the rating work starts where any market analysis starts: with a map of who is selling what. That map comes with a known defect — it is guaranteed to expire — and learning to work with an expiring map is the actual point of chapter 3.

3. The Supplier Map: A Snapshot That Expires

Chapter 2 left the board four claims heavier, and notice what kind of claims they are: comparative ones. “The cheapest frontier-quality model on the market.” “90% cheaper than GPT.” A comparative claim cannot even be parsed, let alone rated, without a reference class — cheapest among whom? cheaper than which GPT? — and the reference class is the market itself. So before any rating can begin, you need to know who is actually selling tokens in June 2026, what each seller’s current line is, and what makes each one structurally different from its neighbours. This chapter draws that map. Then it spends its second half on the property of the map that matters more than any entry in it: the map is perishable, and the perishability is itself the lesson.

One deliberate omission before you read on: there are no prices in this chapter. Prices get a table of their own in chapter 8, where they can be handled with the care perishable numbers demand. Here the only price language permitted is qualitative — “the price floor,” “a cheap tier” — because this chapter’s job is structure: who sells, what they sell, and under which licence.

3.1 The supplier table

Eight suppliers matter for coding work as of June 2026. One row each: the current line, the licensing posture, and the one fact that distinguishes that supplier from the other seven. Every cell carries the same invisible footnote — as of June 2026 — and section 3.2 is about why that footnote is the most important thing on the slide.

Supplier Current line (as of June 2026) Weights The one fact to remember
Anthropic Claude Fable 5 (GA June 9, 2026); tiers Opus 4.8 / Sonnet 4.6 / Haiku 4.5 Closed API First Mythos-class public model: 1M-token context, adaptive thinking — the model chooses its own reasoning budget
OpenAI GPT-5.5 (API April 24, 2026); GPT-5.4 tier family Closed API Runs a dedicated agentic-coding line, GPT-5.3-Codex, beside the general flagship — one supplier, two products for "coding"
Google Gemini 3.1 Pro / 3.5 Flash; 2.5 Flash-Lite below them Closed API Holds the budget floor among the majors (2.5 Flash-Lite); Gemini 3.5 Pro (2M context, Deep Think) announced, June GA expected
Meta Llama 5 and Muse Spark (both April 8, 2026) Dual-track: open + closed The April pivot: open-weights Llama 5 ships alongside Muse Spark, Meta's first closed model (Meta Superintelligence Labs); Llama 4 Behemoth shelved, kept as a teacher model
DeepSeek V4 (April 24, 2026): V4-Pro, a 1.6T-parameter MoE with 49B active; V4-Flash at 284B/13B Open weights, MIT The frontier-scale price floor — flagship-class architecture, 1M context, and the weights are a download
Alibaba Qwen 3.6 family (open, Apache 2.0); Qwen 3.7 Max (closed API, 1M context) Both The dense Qwen 3.6 27B is the popular local-coding workhorse — small enough to run where the code lives
Mistral Medium 3.5 (128B dense, open); Large 3 (675B MoE, Apache 2.0); Small 4 Mostly open The EU option: a European supplier publishing open weights up to 675B parameters
xAI Grok 4.3 (April 30, 2026); grok-code-fast-1 Closed API Ships a deliberately cheap agentic-coding tier, grok-code-fast-1, beside the flagship

Three observations deserve more space than a table cell.

Meta’s row is a pivot, not a product. Until this spring, Meta was the open-weights standard-bearer among the giants. On April 8, 2026 it shipped two things at once: Llama 5, continuing the open line, and Muse Spark — its first closed-weights model, out of the newly branded Meta Superintelligence Labs — while Llama 4 Behemoth, long previewed, was quietly shelved into service as a teacher model for distillation. The company that made “open” a strategy now runs both stalls in the same market. That tells you something that matters for chapter 4: open versus closed is no longer an ideological divide between companies; it is a per-model business decision, made separately for each release, sometimes inside a single organisation on a single day.

DeepSeek’s row is the frontier-scale price floor wearing frontier anatomy. V4-Pro is a 1.6-trillion-parameter mixture-of-experts with 49 billion parameters active per token, a million-token context, and MIT-licensed weights anyone can download. Nothing about that description says “budget model” — the architecture is the same genre the closed flagships use — and yet this is the supplier that anchors the cheap end of the market’s frontier-class rows — the absolute budget floor, Gemini’s 2.5 Flash-Lite, is a lightweight tier, not a frontier product. When chapter 8 shows you how wide the price spread across this table really is, remember that the floor is not a toy; and when chapter 4 asks what it would take to host those weights yourself, remember the 1.6T.

And one discipline for every capability number that floats near this table. Third-party reports put a DeepSeek V4-Pro-Max variant at roughly 80% on SWE-bench Verified; Mistral’s own material claims 77.6% Verified for Medium 3.5. Both figures may be true. Neither is usable yet, because both arrive exactly the way Claim 5 arrived in Lecture 8 — vendor- or third-party-reported, scaffold and attempt count undisclosed, unreproduced — and question 9 of the checklist does not care how plausible a number feels. On this map, every score is a claim wearing its reporter’s name, or it is nothing. The map tells you who sells; it does not certify what they sell.

3.2 Teach the expiry, not the list

Now reread the table, but look at the dates instead of the names. Five of the eight rows carry a release date from April 2026 or later: Meta’s dual launch on April 8, GPT-5.5 and DeepSeek V4 on April 24, Grok 4.3 on April 30, Claude Fable 5 on June 9 — days before this lecture was finalised. And the churn does not pause politely between releases: while this lecture was being researched, the #1 slot on LMArena flipped within 48 hours of a release — a model entered the board on June 10, 2026 and took the top position — an incident chapter 5 tells properly, because the claim sitting on that #1 is pinned to the board and gets rated there.

So treat the table the way Lecture 8 taught you to treat a Claim Board verdict: it carries an as-of stamp, and the stamp is load-bearing. At the frontier, turnover is measured in weeks. A supplier map from January would miss Meta’s entire closed line, name the wrong OpenAI flagship, and place DeepSeek a generation back. The version above will fail the same way, on schedule. This slide is wrong by exam time — the skill is knowing where to refresh it. That sentence is this chapter’s actual learning objective. Memorising the table earns you nothing durable; what transfers is the refresh procedure, and the refresh procedure is chapter 5: the leaderboards for capability, the aggregators for price and speed, the routers for adoption — instruments that update in days or minutes while a lecture slide stays frozen.

What does survive the expiry is the structure the names are arranged in, because structure moves slower than products. Suppliers ship tiered families — a flagship, a mid-tier, a cheap tier — rather than single models. Several maintain a separate agentic-coding line beside the general flagship. Licensing postures — closed, open, dual-track — persist across releases even as version numbers roll. Those patterns will still describe the market when every model name in the table is a historical curiosity, and they are what you should actually encode.

Look down the table one last time before we move on. The capability differences between rows are real but fluid — they reorder themselves every few weeks, which is exactly why this chapter refuses to rank them. The qualitative price differences are wide, but chapter 8 will show you they are also negotiable, through caching, batching, and routing. The split that is neither fluid nor negotiable sits in the third column: the deepest divide in the supplier map is not capability and not price — it is who controls the weights. A closed API and an MIT-licensed download are not two prices for the same product; they are two different products with two different failure modes, and for a German employer the choice between them is frequently made before anyone reads a single benchmark. That choice is chapter 4.

4. Open Weights vs. Proprietary API: The European Question

Chapter 3 ended on the weights column — the one split in its table that is neither fluid nor negotiable — and this chapter exists because that binary decides who can take the model away from you, where your code is allowed to travel, and — in a German enterprise — whether you may use the model at all. Those are not benchmark questions. They are the questions that, in practice, get answered before anyone opens a leaderboard.

4.1 What open weights buy you

“Open weights” means the parameter files themselves are published under a real licence — MIT for DeepSeek’s V4 line, Apache 2.0 for Qwen 3.6 and Mistral’s Large 3, Meta’s own terms for Llama 5. What you buy when you take that download is a bundle of four guarantees, and it is worth seeing that each one is something an API contract structurally cannot offer:

That last guarantee sounds like the boring one. It is the one this chapter is named for.

4.2 Retired under you

Here is the proprietary risk in its concrete June 2026 form: Claude Sonnet 4 retires on June 15, 2026 — the very day Lecture 8 ran. By the time you sit in this lecture, an endpoint that real products were built on no longer answers. Anyone who shipped a feature tuned to that model’s behaviour — its tone, its tool-calling quirks, the prompt that finally worked after three weeks of iteration — now migrates on the vendor’s schedule, re-validates on the vendor’s timeline, and re-runs every evaluation they ever trusted, whether or not this quarter had room for it.

An API model can be retired under you; a weights file cannot be taken back. Notice what API “version pinning” actually is: you may pin a model identifier, but the pin defers retirement, it does not prevent it — a deprecation notice gives you a migration deadline, not a choice. Lecture 8 taught you that a benchmark score is a measurement of a system under specific conditions; retirement is the vendor deleting one of those conditions. Every number you measured against the retired endpoint — your pass rates, your regression suite of prompts, your Locust-issue results — refers to a system that no longer exists. Version pinning is not paranoia. It is the only reproducibility guarantee the API world cannot give you at any price, and it is the single strongest argument the open-weights column has.

4.3 The middle path — and the honest capability picture

Between “send your code to a foreign API” and “operate your own inference cluster” sits a middle path most European enterprises actually take: the frontier closed models served through regional cloud endpoints — AWS Bedrock and Google Vertex both offer EU regions — where API convenience comes with contractual data-residency commitments: your prompts are processed in-region, under terms your lawyers can read and your auditors can cite. Understand precisely what this buys. It answers the residency question, not the retirement one: a hosted model is still the vendor’s model, and it retires on the vendor’s schedule no matter whose region it runs in.

Whether the open road costs you capability is an honest question with a dated answer. As of June 2026 the open/closed gap has narrowed dramatically: open-weights models now claim SWE-bench-Verified-class scores around 80% — claims that arrive vendor- or third-party-reported, and which chapter 3’s discipline says you flag rather than endorse. The very top, though — the vendor-measured ~95% systems from Lecture 8 — is API-only territory. And owning the weights is not the same as being able to run them: DeepSeek V4-Pro activates 49 billion parameters per token, but all 1.6 trillion must be resident in memory to serve a single request. Self-hosting that is a multi-machine serving deployment with an inference stack to operate, capacity to plan, and engineers on call — a serious infrastructure project with a payroll, not a docker pull. The realistic local end of the open-weights world is the other end of chapter 3’s table: dense models like the Qwen 3.6 27B, popular for local coding precisely because they fit on hardware one developer controls.

The three postures, side by side — qualitative on purpose, because every quantitative cell would need chapter 8’s expiry stamps:

Dimension Open weights, self-hosted Regional endpoint (Bedrock / Vertex EU) Direct vendor API
Retirement risk None — the file is yours forever Vendor's schedule (hosting does not prevent retirement) Vendor's schedule
Data residency A property of your hardware Contractual, in-region The vendor's terms of service
Fine-tuning on private code Unrestricted, on-premises Only what the platform offers Only what the vendor offers
Capability ceiling (June 2026) ~80% Verified-class — vendor/third-party-claimed The closed frontier The closed frontier (vendor-measured ~95% systems)
Operational burden Yours — sized to the model, up to "serious infrastructure project" Mostly the cloud's The vendor's

Read the table column-wise and the trade becomes visible: moving left buys control and costs operations; moving right buys the frontier and costs guarantees. No column dominates — which is precisely why this is a decision and not a default.

4.4 The German employer reality

Now place all of this where most of you will actually encounter it: a German employer. Two institutions make the weights column decisive here long before any capability argument starts. The first is the works council (Betriebsrat): introducing an AI coding assistant is co-determination territory, because a tool that logs every prompt, every file touched, and every acceptance or rejection of a suggestion is — whatever its stated purpose — a technical system capable of monitoring employee performance. Expect the negotiation to turn on exactly the questions this chapter has been answering: where do the prompts go, who can read the logs, what leaves the building. The second is regulation by sector: an automotive supplier whose codebase embodies a customer’s IP under contract, or a BaFin-supervised financial institution with outsourcing and auditability obligations, does not treat data residency as a preference to be traded against a benchmark score. It is a hard constraint, and “the model is better” is not an argument against a hard constraint.

In a German enterprise, the first question is rarely “which model is best?” — it is “which models are we allowed to use?” Internalise that ordering, because it inverts how every launch post wants you to think: constraints filter first, capability ranks second among the survivors. The ordering is not a German peculiarity to be worked around; it is simply correct engineering under constraints, and it is why hard constraints first will reappear verbatim as step 2 of the decision framework in chapter 7.

Whichever side of the weights divide your employer lands on, the next move is the same: you hold a shortlist — two or three closed endpoints, or two or three open checkpoints — and you must compare them. The places where those comparisons happen, the leaderboards and arenas and aggregators and routers, are not neutral windows onto the market. They are instruments, each with a measurement procedure and each with a characteristic failure mode — and an instrument you cannot read critically will mislead you exactly as confidently as a launch post does. Reading those instruments, and knowing what each one cannot tell you, is chapter 5.

5. The Platform Tour: Where to Watch, and What Each Cannot Tell You

Chapter 3 handed you a supplier map with an expiry date, and chapter 4 added the licensing split that decides what you are allowed to run at all. What you still do not have is the instrument panel: the places you actually look when the map needs refreshing or a shortlist needs numbers. This chapter is the tour — and it is deliberately not a list of bookmarks, because the skill is not knowing that these platforms exist. The skill is knowing what each one can and cannot tell you, and Lecture 8 already gave you the method for finding out. A comparison platform is itself a measurement system, and it decomposes exactly the way a benchmark does: ask what it measures, who submits to it, and where its failure mode hides. That is the same three-part interrogation you ran against SWE-bench’s leaderboard, applied one level up — not to a single instrument but to the observatories that publish the readings.

Two of the Claim Board’s open items live or die in this chapter. Claim 3 — “#1 on LMArena” — has been pinned since Lecture 8 and today finally meets the platform that produced it. Claim 8 — “the most-used model in the world, #1 by token share on OpenRouter” — went up an hour ago. Both get rated live in section 5.4, once the tour has put the evidence on the table.

5.1 The platform table

Seven instruments cover, between them, almost every question an engineer actually asks about the market. No single one covers more than two.

Platform What it measures Trust it for Its failure mode
swebench.com % of real GitHub issues resolved Repo-level coding ability; the bash-only track for clean model-vs-model comparison Saturation at the top; scaffold confound on the open track
SWE-bench Pro (Scale AI) Held-out and commercial repos, graded SWE-bench-style The current "hard" coding number Young; far less community scrutiny than the original received
LMArena Human preference: Bradley–Terry ratings fit to blind pairwise battles (not true Elo) UX and instruction-following feel Style/verbosity bias; the Leaderboard Illusion (§5.2)
Artificial Analysis Independent re-runs: an averaged intelligence index plus measured price and speed The cost/performance frontier chart — the best single teaching artifact of this lecture One averaged index hides task-specific strengths
OpenRouter rankings Real token share across one router — revealed preference "What do practitioners actually run?" Tokens ≠ value (§5.3); blind to enterprise direct traffic
HELM (Stanford) Multi-metric academic harness: accuracy plus calibration, robustness, fairness The methodology gold standard Slow cadence; weak on agentic coding
Epoch AI Independent re-evaluations and long-run capability trends "Is the vendor number real?" Selective coverage — it re-runs what it chooses to

Now read the table down the who submits axis, because that single question sorts the seven into three families. The first family grades tasks and accepts submissions: swebench.com’s open leaderboard takes entries from anyone — which is exactly why its open track inherits the scaffold confound of Lecture 8, and why its bash-only track exists — and Scale AI’s SWE-bench Pro leaderboard runs the held-out evaluation itself, which is what makes it the current reference for “hard” coding work and also what makes it young: a private grader earns trust slowly, and Pro has not yet absorbed years of adversarial community audit the way the original did. LMArena’s submitters are the vendors themselves, in a way that matters enough to get its own subsection. The second family submits nothing and runs the models on its own account: Artificial Analysis re-measures price, speed, and an averaged capability index on its own hardware budget; HELM and Epoch re-evaluate on academic timelines. They are the reproduction tier — checklist question nine made into institutions. The third family is OpenRouter, which has no submissions at all: its rankings are telemetry, a side effect of routing billions of real, paid requests. Nobody can submit to it; you can only use it, which is precisely what makes its numbers interesting and precisely what limits them (§5.3).

One row deserves a flag before we move on: Artificial Analysis’s cost/performance frontier chart — every current model as one point in the price-versus-capability plane — is the single best teaching artifact of this lecture, and chapter 7’s shortlisting step is built on it. It is also an averaged index, and an average is a choice: a model that is mediocre on everything and a model that is exceptional at exactly your task can land on the same point.

5.2 The Leaderboard Illusion

LMArena’s mechanics first, because the press shorthand obscures them. You type your own prompt, two anonymous models answer side by side, you vote for the better answer, and only then are the identities revealed. Millions of such battles are aggregated with a Bradley–Terry model — a statistical fit over the entire battle matrix that estimates, for every pair of models, the probability one beats the other. Headlines call the resulting ratings “Elo,” but they are not: Elo is a sequential update rule from chess; Bradley–Terry is re-fit over the whole history, which is why every rating ships with a confidence interval. That detail will matter in §5.4, because it makes LMArena one of the few leaderboards that actually publishes its error bars.

Then came the audit. The Leaderboard Illusion (arXiv:2504.20879) documented two structural advantages available to large providers. First, private variant testing: 27 private Llama-4 variants were battle-tested on the arena, unpublished, before release — try many, publish the winner. You have seen this move before: it is best-of-\(N\) selection with the leaderboard itself playing the verifier — the same machinery Lecture 8 found inside vendor scaffolds, lifted from the harness to the instrument. Second, sampling asymmetry: models from the biggest providers appeared in up to 34% of daily battles, versus roughly 3% for small providers — more battles mean tighter ratings, faster iteration, and a far larger stream of user prompts flowing back to the vendor.

Underneath both findings sits a bias that needs no manipulation at all. A blind voter judging two answers side by side rewards what a side-by-side makes visible: confident tone, clean formatting, satisfying length. A blind voter cannot run your test suite, check the citation, or wait to see whether the confident answer was actually correct. Bradley–Terry aggregates preferences; it cannot tell you why a model wins. A rating of 1510 says “picked more often.” Whether it was picked for correctness, for markdown polish, or for never admitting uncertainty is information the instrument does not record — and therefore cannot report.

5.3 Tokens are not value

OpenRouter measures the one thing no grader and no voter can: what people actually pay to run, request by paid request. That is revealed preference, the economist’s favourite evidence, and it deserves real respect — a model that millions of real tokens flow through every day is passing a test no benchmark administers. But the same dataset contains its own refutation of the obvious reading. As of June 2026, Anthropic models account for roughly 12.3% of OpenRouter’s token share but about 46% of its dollar share. Volume and value rank the market differently inside one instrument’s own books. The mechanism is no mystery: cheap models absorb the bulk work — high-volume, low-stakes drafting, summarization, classification — while expensive models are reserved for the work people will pay a premium per token to get right, and per-token prices differ across this market not by percentages but by orders of magnitude (chapter 8 puts the exact number on the spread). So token share measures traffic; dollar share measures what users will pay for; and “most used” silently chooses between them. Add the second blind spot — a router only sees traffic that is routed through it, and the heaviest enterprise consumers contract with vendors directly, invisibly — and you have everything needed to rate Claim 8.

5.4 Two claims, rated live

The verdict vocabulary is Lecture 8’s rubric — verified, plausible-but-unproven, refuted-as-stated — and the method is the nine questions, bent slightly to fit instruments that grade votes and traffic instead of patches.

Claim 3: “#1 on LMArena.” Pinned in Lecture 8, days after a newly released model entered the Text board. Run the questions. Contamination and leakage (questions 1–2) lose their grip — there are no gold answers to leak into a preference battle. The grader (question 3) is the human voter, with every bias §5.2 catalogued. What exactly was measured (question 5) is, unusually, fully disclosed: blind pairwise preference votes, Bradley–Terry aggregation, one named board. Error bars (question 7) are published — rarer on leaderboards than it should be. Representativeness (question 8) fails for your purposes: chat battles are not repository work. The reporter (question 9) is the platform, not the vendor, and anyone with a browser can check the board. And the facts do check out: the model entered the Text board on June 10, 2026 and took #1 at a rating of 1510 ± 11, while Grok 4.1 Thinking fell to #29 at 1466 ± 3 — as of June 2026. Read those two numbers together: 44 rating points span the distance from #1 to #29. The top of the board is dense, and rank order amplifies preference differences that the ratings themselves report as small. The whole inversion took 48 hours from one release — the same incident chapter 3 used as expiry evidence, now sitting inside a verdict.

Verdict: verified — as a statement about blind preference votes, on one board, on one date. And now the lesson this chapter exists to teach. A claim can be verified and still be almost worthless as a decision input — verification and decision-relevance are different properties. Nothing in the verdict is sarcasm: the count is real, the error bars are honest, the platform is independent. But the thing verified is a perishable fact about votes — structurally tilted by §5.2’s sampling asymmetry, produced by a grader that cannot run a test suite, overturned in two days by a single release — and not a durable fact about how a model will handle your repository. Lecture 8’s rubric tests whether a claim is true as stated; chapter 7’s framework tests whether it should move a decision. Those are separate gates, and Claim 3 is the board’s first claim to pass the first gate and stall at the second. By the house rule — a claim only comes down rated — it comes off the wall today, verdict and dates stamped.

Claim 8: “The most-used model in the world — #1 by token share on OpenRouter.” Pinned an hour ago in chapter 2. Split it the way the rubric taught you. The measurable half — #1 by token share on OpenRouter — is checkable on the platform’s own published rankings, on the platform’s own terms, and we will take it as real. The superlative half fails twice, and both failures are supplied by the instrument itself. A router can only count traffic that flows through it: the largest consumers in the market buy directly from vendors, and their tokens are invisible to the rankings, so “in the world” claims a population the instrument never sampled. And within the router’s own dataset, “most-used” is ambiguous in exactly the way §5.3 quantified: ranked by tokens one provider leads; ranked by dollars the order changes — 12.3% against 46% is the proof, drawn from the same books the claim cites. Verdict: refuted-as-stated — the number is real; the superlative is not supported by the instrument that produced it. Note the family resemblance to Lecture 8’s “best coding model” verdict: once again the rating is a statement about the sentence, not about the model.

5.5 The trust heuristic

Compress the tour to one slide — one question, one instrument, one caveat each:

That heuristic routes every market question you will ask this semester to the instrument least likely to mislead you on it. But notice what every row of the platform table has in common: all seven are aggregators — they sit on top of, re-run, or collect votes about models that arrive on the market wearing a primary document written by the vendor itself. The headline numbers pinned to the Claim Board were not born on platforms; they were born in model cards, and they reached the launch post only after passing through a vendor’s choice of scaffold, attempt count, thinking budget, and subset — choices that live in the footnotes, when they are disclosed at all. The platforms tell you what others measured. The card tells you what the vendor measured, under conditions the vendor chose, disclosed at a level of honesty the vendor controls. Reading one is a skill of its own — footnotes first — and that is chapter 6.

6. How to Read a Model Card

Chapter 5 ended with an instruction this chapter now makes practical: when a platform number matters to you, walk upstream. Every stop on the tour was a secondary source — Artificial Analysis averages other people’s benchmarks into an index, OpenRouter counts other people’s tokens, LMArena counts other people’s votes — and aggregation, whatever its virtues, discards exactly the measurement conditions that Lecture 8 taught you are the measurement. The primary source is the model card: the vendor’s own document stating what the model is, what was measured, and — if you know where to look — how. (Its longer sibling, the system card, carries the safety evaluations and deployment detail; for benchmark claims, the model card’s table is where the action is.) This chapter teaches you to read one, and the lesson has a sharp center: the part of the card designed to be looked at is not the part that carries the information.

6.1 The anatomy of a model card

Across vendors, model cards converge on the same five-part skeleton, in roughly the same order:

  1. Description and knowledge cutoff — what the model is: context window, modalities, intended use, and the date after which the world is invisible to its weights.
  2. The benchmark table — the scores, usually with the vendor’s previous generation and one or two competitors as comparison columns. This is the part that gets screenshotted.
  3. Methodology footnotes — the asterisks under the table: attempt counts, thinking budgets, sampling settings, subset definitions, scaffold descriptions.
  4. Safety evaluations — organized at Anthropic under its Responsible Scaling Policy and at OpenAI under its Preparedness Framework, frameworks that tie measured capability thresholds to deployment safeguards; the detail lives in the system card, and one line is all we need here.
  5. Limitations — the paragraph the lawyers and the honest scientists co-wrote. Routinely skipped; routinely the most candid text on the page.

Now look at what that ordering does. The table is item two, and everything that determines what the table means is item three — set in smaller type, below the fold, written in a register that signals “technicality, skip me.” Lecture 8 spent an entire afternoon establishing that the conditions are not technicalities: a score without its measurement protocol is not a measurement, it is a number wearing one. On a model card, the protocol has a typographic address. The benchmark table is the marketing surface; the footnotes are the measurement protocol — so read the footnotes as the content, and the table as an index into them.

6.2 Footnote hunting

Five footnote species recur across vendors, and each one maps onto machinery you already own from Lecture 8 — which means you can decode all five today. The hunt itself has a fixed choreography, and it is the one you will use in the lab: find the benchmark table, scan it for markers — asterisks, daggers, superscript numerals, the word “with” inside a cell — then read the footnote block before you let any score register, and finally re-read the table with each cell mentally annotated by its protocol. The table below is your field guide — its left column quotes the genre, not any one vendor’s card: the wording is representative and the numbers are illustrative; the lab in chapter 14 puts two real model-card benchmark pages in your hands — printed, because asterisks are easier to hunt when the page cannot scroll away — and sends you hunting with exactly this guide.

The footnote, as printed What it actually tells you The Lecture 8 machinery it points to
"with parallel test-time compute" Best-of-N wearing a lab coat: the task was run many times and a selector picked the winner. The score is a generous-k point on a pass@k curve, printed next to competitors' single attempts. pass@k; test-time compute scaling
"with extended thinking (64K token budget)" A thinking budget is an inference budget — the third component of the system decomposition. Raise it and the score rises; the footnote rarely says whether the competitor columns got the same allowance. model + scaffold + inference budget
No sampling settings at all The dog that doesn't bark: a Pass@1 without temperature and sample count is an underdefined quantity — two labs can both honestly report "Pass@1" and have run different experiments. the underdefined Pass@1
"on a 477-instance subset" Twenty-three instances are missing. Which ones, chosen how, by whom? Dropping tasks a harness cannot run is defensible engineering; dropping tasks the model fails is score laundering — and the footnote's wording rarely lets you tell which happened. the task count moves the error bar
The scaffold goes unnamed Whose harness produced the number — the vendor's tuned, undisclosed one, or a public reference loop? As of June 2026, that one choice spans roughly 77% (bash-only public track) to ~95% (vendor-measured) on the same 500 Verified tasks. the scaffold problem; the two numbers

Walk the first row once, slowly, because it is the genre’s masterpiece. “Parallel test-time compute” sounds like an infrastructure detail — something about GPUs, surely, nothing to do with the score. What it describes is running the whole task many times and letting a verifier pick the best candidate: best-of-\(N\), exactly the machinery Lecture 6’s test-time compute scaling named and Lecture 8’s pass@k chapter quantified — a generous \(k\) being the single cheapest way to make a per-attempt probability look like a strong system. The number is real. Placed in the same table column as a competitor’s single attempt, it stops being comparable — and Lecture 8 put the cost of that machinery at 10–100× a single attempt, a multiplier the footnote concedes in principle and states nowhere.

Once you have decoded one row, notice that the five species are not five separate lessons. Each footnote names one component of Lecture 8’s system decomposition: the sampling-settings footnote (or its absence) configures the model, the unnamed harness is the scaffold, parallel test-time compute and the 64K thinking allowance are two faces of the inference budget, and the 477-instance subset is the data. A model card’s footnote block is the system decomposition, scattered into fine print — which is why hunting it is not pedantry: you are reassembling the experiment that produced the number, the same experiment the nine questions demand to see. And the subset species earns one extra beat of suspicion, because it is the only one that changes the task population rather than the measurement: twenty-three quietly absent instances move both the score and its error bar, and “excluded for infrastructure reasons” is unfalsifiable from the outside.

One fairness note before the lesson hardens into cynicism: the footnote itself is the vendor behaving well. Disclosure is precisely what the nine questions demand, and a card that prints its attempt counts, budgets, and subsets is strictly better than the many launch posts that do not. The failure happens downstream. The asterisk survives on the model card and dies in transit — stripped from the tweet, the keynote slide, the aggregator cell, the news headline — until the number circulates naked, meaning something materially different from what the card said. The footnotes are where the inflation lives — not because vendors hide the protocol there, but because every retelling of the number strips the protocol off, and the card is the last place it exists. Footnote hunting is the act of walking back to that last place: the same upstream move chapter 5 taught for platforms, taken one step further to the source.

6.3 The tokenizer footnote

One footnote species is missing from the field guide above because it deserves its own section — partly because it is the quietest of them all, living in migration guides and pricing pages rather than under the benchmark table, and partly because it is this lecture’s hinge from scores to bills. The species: the tokenizer changed between model generations. The current instance: as of June 2026, the tokenizer Anthropic introduced after Opus 4.7 yields roughly 30% more tokens than its predecessor for the same text.

Sit with what that does to a price comparison. A token is not an SI unit. Lecture 4 taught you it is the output of a learned segmentation, and segmentations differ — between vendors, and now demonstrably between generations of the same vendor. Every per-token price comparison silently assumes the denominator is constant; across a tokenizer change it is not, so a model that looks cheaper per token can bill more tokens for identical input, and the list-price discount shrinks — or inverts — on the invoice. The same trap operates between vendors, where tokenizers have always differed: Claim 7 on the board, “90% cheaper than GPT”, silently assumes both models would meter your workload in the same units. They will not, and the footnote that would tell you so does not exist, because no vendor footnotes the other vendor’s tokenizer.

A per-token price is denominated in a unit the vendor defines; comparisons across generations or vendors are only meaningful per task — tokens-per-task times price-per-token — never per token alone. That sentence is this chapter’s contribution to Block B: when chapter 8 puts the full price list in front of you, every comparison you do with it must carry a tokens-per-task denominator, and the chapters after it will teach you to estimate one.

There is a reassuring corollary hiding inside the warning. You do not need to model tokenizer differences analytically — you need to measure in the right unit. Any comparison that runs the same real task on both models and reads off the billed token counts, or simply the bill, has already normalized the denominators away; the trap lives exclusively in arithmetic done on list prices alone. That is one more quiet argument for running your own evaluation instead of comparing brochures — the next chapter’s framework makes it a mandatory step — and chapter 11 supplies the counting tools that make the measurement cheap.

And with that, Block A’s reading course is complete: map, constraints, platforms, and now the primary source, footnotes first. What none of those readings does on its own is decide anything — assembling them into a procedure, six steps ending on an honest confession, is chapter 7.

7. The Decision Framework: From Benchmark to Bill

This chapter exists because everything Block A has produced so far is inputs without a procedure. From chapter 3 you have a supplier map with an expiry date printed on it; from chapter 4, the constraints that settle make-or-buy before any benchmark gets a vote; from chapter 5, a platform for each question and a trust heuristic for each platform; from chapter 6, the primary source, read footnotes-first. Each of those is a reading skill, and reading skills accumulate into connoisseurship — the ability to say increasingly sophisticated things about model cards at parties. What your team lead will ask for is different: which model do we use, starting Monday? This chapter assembles the readings into the procedure that answers that question.

The procedure has six steps, and it is deliberately undramatic — every step is something a chapter of this course has already armed you for, placed in an order that prevents the most common failure mode in the wild: starting at a leaderboard and rationalizing backward from its top row. It also ends, honestly, on a confession. The decisive step requires arithmetic that this lecture has not yet given you, and naming that gap precisely is how Block A closes.

7.1 The six steps

  1. Classify the task construct — what kind of work is this, and which benchmark even measures it?
  2. Hard constraints first — filter on what must be true before anything is allowed to be ranked.
  3. Shortlist two or three models — public capability numbers crossed with public price/speed numbers.
  4. Run your own 20–50-task eval on your repo — public benchmarks predict ranking, not fit.
  5. Decide on cost-per-solved-task, not cost-per-token — the decision rule, treated in §7.2.
  6. Re-evaluate quarterly — the market your decision lives in expires faster than your fiscal year.

The order is the content. Each step shrinks the space the next one has to search: the construct decides which benchmarks are admissible, the constraints delete candidates wholesale, the platforms reduce what is left to a shortlist, and only then — fourth, not first — does any measurement of your work enter. Leaderboards appear in step 3 and nowhere earlier, which is exactly as much influence as Block A’s evidence says they have earned. One table to hold the whole procedure — each step, the failure it exists to prevent, and the chapter that armed you for it:

Step The failure mode it prevents Where you learned it
1. Classify the construct Quoting a function-synthesis score at an agentic-PR decision — the construct category error Lecture 8, chapter 3 ("can code" is not one thing)
2. Hard constraints first Carefully ranking models the works council, the regulator, or the budget would veto anyway Chapter 4
3. Shortlist two or three Treating a leaderboard as a verdict instead of a sieve Chapter 5 (the trust heuristic)
4. Your own 20–50-task eval Buying the familiarity discount: scores earned on famous repositories that do not transfer to yours Lecture 8, the familiarity effect
5. Cost-per-solved-task Optimizing for the cheapest tokens instead of the cheapest delivered work Lecture 8, the missing column → §7.2
6. Re-evaluate quarterly A decision made exactly once, on a market that has since turned over Chapter 3 (the map expires)

Step 1 is Lecture 8’s construct lesson, applied to procurement. Chapter 3 of that lecture established that “can code” is not one measurable thing — and neither, it follows, is “good enough.” Inline autocomplete is a latency-and-price construct: completions must arrive at keystroke frequency, which puts you in the cheap tier before quality is even discussed. A multi-file agentic PR is the frontier construct — SWE-bench Pro country, where the real capability separations live. And whole-repo comprehension is a third construct in which context length is the binding constraint: a model that cannot hold the repository is disqualified at any quality score. Classify first, because the construct determines which benchmark column is allowed to influence you at all — quoting a function-synthesis score at an agentic-PR decision is the category error Lecture 8 taught you to name.

Step 2 is chapter 4 run as a filter. Must the weights stay in the building — data residency, a works council agreement, a regulator with opinions? Is fine-tuning on private code required? Is there a hard budget cap? A hard constraint is a filter, not a weight: a model that fails it gets no second look regardless of what it scores, and in a German enterprise this step routinely decides more than the benchmarks do — the works council and BaFin do not read leaderboards, and chapter 4 explained why they do not need to.

Step 3 finally opens the leaderboards — used exactly as chapter 5’s trust heuristic prescribes: capability from SWE-bench Pro and Terminal-Bench, price and speed from Artificial Analysis’s frontier chart. The output is a shortlist of two or three candidates, not a ranking. The public numbers’ job is to make sure nothing obviously dominant is missing from your list and nothing obviously dominated is on it; at that point they have done everything Block A says they can reliably do.

Step 4 is where the decision becomes yours, and it is the step most teams skip. Public benchmarks predict ranking, not fit: Lecture 8’s contamination chapter showed the same agents scoring measurably better on issues from the famous benchmark repositories than on simpler tasks from newly added ones — the familiarity effect — and your repository is, with overwhelming probability, the less-famous kind. So build the small eval: twenty to fifty tasks from your own issue tracker, with pass/fail checks you define, run identically for each shortlisted model. Twenty to fifty is small on purpose — cheap enough to actually run and to re-run in step 6 — and Lecture 8’s error-bar arithmetic still applies at that size: gaps of a few points are noise, and what you are looking for is a clear separation on your task distribution, not a rank order. The Locust thread is this step in miniature: the fail-to-pass instance you built by hand for last week’s homework is row one of exactly such an eval, and the course finale assembles the class’s rows into a benchmark of its own. Step 5 is then the decision rule that reads the eval out, and it carries enough weight to get its own section — §7.2, next.

Step 6 is chapter 3’s lesson turned into a calendar entry. The supplier map expires; so does any decision derived from it. Quarterly, re-check prices and deprecation notices — chapter 4 taught you a model can be retired under you — and re-run the step-4 eval against whatever now looks like the frontier. This is why step 4 insisted on small and versioned: the framework is a loop, not a line, and a loop you cannot afford to traverse is a decision you made exactly once, years ago, on someone else’s numbers.

7.2 Cost-per-solved-task

Step 4 leaves you with two numbers per model: what it solved, and what running it cost. Step 5 divides them:

\[\text{cost per solved task} \;=\; \frac{\text{total spend across the eval}}{\text{number of tasks actually solved}}\]

Decide on cost-per-solved-task, not cost-per-token: total spend across your eval — failed attempts included — divided by the tasks it actually solved. The sting of the definition is its asymmetry: a failed attempt bills its tokens exactly like a successful one and contributes nothing to the denominator. Per-token prices live in a world where every token is equally useful; cost-per-solved-task lives in yours, where tokens are a means and a merged fix is the end.

This is why the price list — the artifact chapter 8 is about to hand you — cannot make the decision by itself. A model at a third of the per-token price that solves half as many of your tasks looks like a bargain on the price list and is not one on the invoice. The discount erodes twice before it reaches you: the cheaper model meters your workload in its own tokenizer’s units (chapter 6’s lesson), and its failed attempts — frequently longer than successes, because an agent flails before it gives up — pile into the numerator while adding nothing below the line. And the unsolved half of the eval does not vanish; it migrates to whatever more expensive channel actually resolves it, usually the frontier model or you. Price the whole workload and the cheap model is not cheaper — it is a discount on the tokens and a surcharge on the work.

One honesty clause before the rule hardens: the denominator is only as trustworthy as the word “solved,” and you now know exactly how heavy that word is. Your eval’s pass/fail checks are a grader, and everything Lecture 8 established about graders — weak tests waving wrong patches through, strict tests rejecting correct ones — applies to the benchmark you just built, too. You will audit your own grader in the finale; for now it is enough to define “solved” before the eval runs, not after you have seen which definition flatters which model.

What you have at the end of step 5 is Lecture 8’s missing column, turned from a complaint into a decision rule. The public leaderboard renders a one-attempt system and a best-of-\(N\)-with-verifier system as the same row; your eval cannot make that mistake, because you paid both bills. The citation to carry is Kapoor et al., AI Agents That Matter — the case that agent comparisons are only meaningful cost-controlled, as accuracy-versus-cost Pareto curves rather than single points. Lecture 8’s reading list called that paper the bridge into this lecture; step 5 is where you cross it.

7.3 Claim 5, revisited

Lecture 8 pinned Claim 5 — “80.3% on SWE-bench Pro,” vendor-reported, June 2026, while the best entries on the public Pro leaderboard sat at roughly 46–59% in the same month — and left it on the board. The nine questions have said what they have to say: vendor-measured, scaffold undisclosed, not independently reproduced, a twenty-point gap to the best public entry unexplained — plausible-but-unproven. We will not re-walk them. The cost lens adds one new sentence, and it cuts deeper than all nine: even granting that the number is real, no cost is disclosed anywhere near it, so you cannot compute cost-per-solved-task from a launch post — which means the claim cannot enter step 5 at all.

That is a stricter failure than unverifiability. A claim can be true and still be useless to the procedure: with no cost line — tokens per attempt, attempts per task, anything — the 80.3% has no admissible role in the one step where the decision actually happens, no matter how much you believe it. So the verdict stands, per the rubric, at plausible-but-unproven — but with a sharper reason than last week: not merely unfalsifiable as published, but undecidable-with. The claim stays on the wall: plausible-but-unproven is a holding state, not a final rating, and the one cost figure that would promote it may yet be published. And per Lecture 8’s habit of naming the missing disclosure: a single cost figure next to the score would promote this claim from decoration to evidence. Note that the same lens is already trained on the new wave from chapter 2: Claim 6’s “cheapest frontier-quality model” asserts a numerator and a denominator and discloses neither. The lab will treat it accordingly.

So here is where Block A ends, on the confession promised at the top. The framework is sound, and five of its six steps you could execute this afternoon. But the decisive step — the division that actually picks the model — needs a numerator you cannot currently produce in either direction. Before a run: what will a fifty-turn agent attempt on your Locust issue cost? You have no formula; you cannot even say whether the bill is dominated by the tokens the model writes or the tokens it reads — and the honest answer will surprise most of you. After a run: the invoice is one opaque number; you cannot say why it is that high, what share a cache would have given back, or whether switching models moves it by a factor of two or by two orders of magnitude.

The decisive step of the framework is a division you cannot yet perform — and Block B exists to fix exactly that. It builds the cost side of cost-per-solved-task, piece by piece: why agentic sessions multiply tokens (chapter 9), how caching re-runs the bill (chapter 10), how to predict a cost before you incur it (chapter 11), how to see what you actually spent (chapter 12), and when to step off per-token pricing altogether (chapter 13). In the room, this is the break between the two ninety-minute blocks. When we come back, we start where every market starts: the price list first — chapter 8.

8. The Pricing Table and the 200× Spread

Block B opens here, and it opens by paying a debt. Chapter 7 closed Block A one step short of done: the framework’s decisive step — decide on cost-per-solved-task, not cost-per-token — is arithmetic you could not yet do, because it needs three things nobody had handed you: a price list, a count of the tokens a real task actually consumes, and the relationship between the two. This chapter is the price list. Chapter 9 is the token count — and the relationship between them, it will turn out, is the least intuitive object in this lecture.

One definition before any numbers, because every figure in the rest of this lecture hangs on it. API prices are quoted per MTok — per million tokens — with input and output priced separately, at the standard pay-per-token tier. When you read “$3/$15”, that is $3 per million input tokens and $15 per million output tokens. Input is everything you send: the system prompt, the instruction files, the conversation so far, every tool result. Output is everything the model generates — including, on reasoning models, the thinking tokens you may never read. Two prices per model, then, and the gap between them is not decoration. §8.2 is about why it exists.

8.1 The table, as of June 2026

Here is the market: sixteen models from six vendors, every figure taken from the vendor’s own pricing page, current as of June 2026 — with everything that hedge implies.

Provider Model Input $ per MTok Output $ per MTok Context One note
AnthropicClaude Fable 5$10$501MGA June 9, 2026; adaptive thinking
AnthropicClaude Opus 4.8$5$251M—
AnthropicClaude Sonnet 4.6$3$15—the price point of chapter 9's worked example
AnthropicClaude Haiku 4.5$1$5—budget tier
OpenAIGPT-5.5$5$30~1.05Mabove 272K input tokens: surcharge of 2× input and 1.5× output
OpenAIGPT-5.3-Codex$1.75$14—the agentic-coding line
OpenAIGPT-5.4$2.50$15—mid-line tier
OpenAIGPT-5.4-mini$0.75$4.50——
OpenAIGPT-5.4-nano$0.20$1.25—cheapest OpenAI rung
GoogleGemini 3.1 Pro$2$12—$4 / $18 above 200K context
GoogleGemini 3.5 Flash$1.50$9——
GoogleGemini 2.5 Flash-Lite$0.10$0.40—the budget floor of this table
DeepSeekV4-Pro$0.435$0.871Mopen weights, MIT license
DeepSeekV4-Flash$0.14$0.281Mopen weights, MIT license
AlibabaQwen 3.7 Max$2.50$7.501Mclosed API
Moonshot AIKimi K2.6$0.95$4.00—official first-party price — see §8.4

Now the note that is as load-bearing as the table. Every price above was verified on June 12, 2026, against the vendors’ own pricing pages, and it drifts on chapter 3’s clock — so the version shown in the lecture hall is re-verified the week before this lecture is taught. That is an instruction the instructor follows, and it is also the instruction you inherit: if you are reading this later — for the exam, for a project, for an employer — re-verify the table the week you use it, against the pricing pages themselves, not against blog posts that quote them (§8.4 demonstrates exactly what goes wrong otherwise). What does not drift is the structure — the two-price asymmetry, the in-vendor ladders, the context thresholds, the spread — and structure is what the rest of this chapter teaches. Learn the structure; look up the numbers.

Four things to notice before the structural lessons, just from reading the columns. First, no vendor sells you a model; every vendor sells a ladder. Anthropic’s input rungs run $1, $3, $5, $10 — a tenfold spread inside one vendor — and OpenAI’s run from $0.20 to $5, a factor of twenty-five. The price differences inside one vendor’s catalogue are larger than most differences between vendors, which already tells you that “cheaper than GPT” (Claim 7, pinned in chapter 2) is underdefined before you even reach the methodology. §8.3 returns to it.

Second, the price of a token can depend on how many tokens you have already sent. GPT-5.5 adds a long-context surcharge above 272K input tokens — 2× on input and 1.5× on output — and Gemini 3.1 Pro switches from $2/$12 to $4/$18 above 200K of context, both as of June 2026. Pricing is becoming a function of context length, which matters for exactly the long agentic sessions chapter 9 is about to price out: the sessions that grow are the sessions that cross these thresholds.

Third, read the Context column with the right suspicion. Million-token windows are table stakes at the frontier as of June 2026 — Fable 5, Opus 4.8, the DeepSeek V4 family, Qwen 3.7 Max, and GPT-5.5 at roughly 1.05M all offer them. But a window is a capacity, not a budget: it tells you how many tokens a call may carry, and the price columns tell you what carrying them costs. A bigger window does not make tokens cheaper; it makes bigger bills possible — a distinction chapter 9 turns into arithmetic.

Fourth, notice what the table deliberately omits: cache and batch discounts (chapter 10), premium tiers (one of them returns in §8.3), and subscription seats (chapter 13). The list price is the baseline that every one of those modifies — which is why it is the thing to pin down first.

8.2 Why output costs more

Run your eye down the two price columns and one pattern repeats without exception: output costs more than input — typically three to six times more across this table — and the direction never once flips. The exact multiple varies (DeepSeek prices output at twice input; GPT-5.3-Codex at eight times), but every vendor, open or closed, charges a premium for generated tokens. Sixteen independent pricing decisions do not align like that by convention. They align because of hardware.

You already own the mechanism; it is Lecture 5’s KV-cache chapter wearing a price tag. When your request arrives, the entire prompt is processed in one prefill pass: every input token flows through the forward pass in parallel, the matrix multiplications run wide, and the accelerator does what accelerators are built for — saturated, compute-dense work. Generation is the opposite regime. Decode produces one token per forward pass, and each pass cannot begin until the previous token exists — sequential by construction, with no parallelism to harvest across the sequence — while the KV cache for everything processed so far must be held in accelerator memory for the entire duration of the generation.

You have felt this asymmetry every time you watched a model respond, without necessarily naming it. The long prompt you paste is digested almost instantly — that is prefill, one wide parallel pass. Then the answer streams out token by token at a visibly finite rate — that is decode, one sequential pass per token. Time-to-first-token and tokens-per-second are the latency signatures of the two regimes; the two price columns are their cost signatures.

So a million input tokens and a million output tokens occupy the same hardware for very different lengths of time, at very different utilization — and the invoice reflects it. Output costs three to six times more than input because decode is sequential and memory-bound while prefill is parallel and compute-dense: the price asymmetry on your invoice is Lecture 5’s hardware asymmetry, denominated in dollars. Nothing in that sentence is about model quality; it is pure mechanism, and it holds for every row of the table.

Keep the asymmetry in your head with the right polarity, because intuition tends to store it backwards. The expensive-looking thing — the model writing your patch — is the cheap column. The invisible thing — the model ingesting and re-ingesting what you sent — is the expensive one. And chapter 9 will show that in agentic work the input side is not just pricier per token than you expect; it is vastly larger in volume than anyone expects.

8.3 The 200× spread

Now read the table vertically, across vendors, for the widest gap. Near the bottom sits DeepSeek V4-Flash at $0.14 per million input tokens — open weights, MIT-licensed, served by its own first-party API, and the cheapest frontier-scale line on the table (the one row beneath it, Gemini 2.5 Flash-Lite at $0.10, is a lightweight budget tier, not a frontier product). At the top — above the table, in fact — sits GPT-5.5-pro, OpenAI’s premium tier, at $30 per million input tokens. That pair is a more-than-200× spread across one market in one month, as of June 2026.

One caution before you quote it, because the $30 figures in this chapter are a trap for fast readers: GPT-5.5-pro is a distinct premium product, deliberately not a row in the table above, and its $30 is an input price — while the base GPT-5.5 from the table charges $30 for output. Same number, different column, different model. Conflate them and you have manufactured a wrong fact out of two right ones; the fast reader who does so is precisely the audience §8.4’s stale-number lesson was written for.

Two hundred is not a rhetorical multiplier; it is a budget fact. The same million tokens — the same Locust issue text, the same tool results — costs more than two hundred times as much to push through one endpoint as another. Set that against every other lever this lecture will hand you: prompt caching (chapter 10) and session discipline (chapter 9) each save you a meaningful multiple, but nothing else in Block B moves a bill by a factor of two hundred. Model choice is the single biggest cost lever you have — bigger than caching, bigger than prompt engineering, bigger than everything else in this lecture combined.

And immediately, the discipline that keeps the lever from cutting your hand: 200× is a per-token statement, and chapter 7 has already told you that per-token is the wrong denominator for a decision — the division that matters puts solved tasks underneath. The framework’s order stands: shortlist on capability first, then let the table separate the shortlist. The table is the lever’s price axis, never its verdict.

This is also where two of the new Claim Board claims acquire their raw material. Claim 6 — “the cheapest frontier-quality model on the market” — splits in half on contact with the table: cheapest per token is now checkable (and already contested: the $0.14 row has a $0.10 row below it), while frontier-quality is a Lecture 8 question about which benchmark, which scaffold, which disclosure.

Claim 7 — “90% cheaper than GPT” — does not even parse until you answer four questions the headline skips: which GPT, out of five OpenAI prices spanning a factor of twenty-five on input alone? Input tokens or output? Served by which host (§8.4)? And counted in whose tokenizer — chapter 6’s cross-generation footnote showed how that alone can quietly bend a per-token comparison. Both claims get their full rating in the lab (chapter 14), with this table open on the desk and per-token explicitly demoted to what chapter 7 said it is: a price axis, not a decision.

8.4 Same model, different prices

One confound remains before the table is safe to use, and it lives in the table’s last row. Kimi K2.6 is an open-weights model, and an open-weights model can be served by anyone with the hardware — chapter 4’s structural point, now wearing prices. A proprietary model has exactly one price list, the vendor’s. An open-weights model has as many price lists as it has hosts, and the hosts compete.

Watch what that does to a simple question. “What does K2.6 cost?” has, as of June 2026, several simultaneously true answers: $0.95/$4.00 on Moonshot’s own first-party API, with cache-hit input at $0.16 (chapter 10’s mechanism, priced); the same $0.95/$4.00 on Fireworks; and roughly $0.67/$3.39 as OpenRouter’s cross-provider average — one model’s name stretched across many hosts with many margins.

And there is a fourth answer in circulation that is no longer true at all: $0.55/$2.65, a stale snapshot that still floats through comparison posts and will keep floating long after every host has moved on. It is exactly the kind of number this chapter exists to make you distrust: precise-looking, sourced from nowhere, dated never. For an open-weights model, a price is a property of a model–host pair — and a price comparison that does not name the host is underdefined. You have heard this sentence’s skeleton before: Lecture 8 taught you that a score belongs to a model–scaffold pair, not to a model. Same move, new market. Strip the context off a number and it stops meaning anything — while continuing to look exact.

So now you hold the price list: current for a season, structural for much longer. But a price list prices tokens, and your bill is a price times a token count. For a single completion the count is what you would guess — your prompt plus the answer. For an agentic session — the kind Claude Code runs against your Locust issue, the kind every vendor scaffold in Lecture 8 ran thousands of times — the count is governed by an architectural fact of the API that makes it grow with the square of the session’s length, and nobody’s intuition survives the first honest computation. That computation is chapter 9.

9. The Quadratic Context

This is the most important arithmetic of the week — said plainly, because the lecture means it literally: if exactly one computation from these two blocks makes it into your fingers and stays there, it must be this one. Chapter 8 handed you the price list and closed on a warning: a bill is a price times a token count, and for agentic work the token count is not what anyone expects. This chapter computes it honestly.

The result is worth previewing so you know what to watch for. A fifty-turn coding session whose visible transcript is on the order of a hundred thousand tokens will be billed for more than three million tokens of input — and the reason is not waste, not a bug, and not a vendor trick. It is one architectural fact about how every LLM API works, a fact you have already met in this course and probably never connected to money.

The chapter runs in four steps, deliberately slow: the architectural fact (§9.1), the formula it forces (§9.2), the canonical worked example that every later chapter of this lecture re-prices instead of inventing its own (§9.3), and what the result means for how you work (§9.4).

9.1 The stateless API

Here is the fact. When you work with a chat interface or an agent, it feels like the model remembers the conversation. It does not, and neither does the API. The API holds no conversation state: every call stands alone, and every call re-sends the entire sequence so far — the system prompt, the instruction files, every tool result, every intermediate answer, and only then the newest message. What feels like turn fifty-one of a dialogue is, to the API, a brand-new request whose prompt happens to contain fifty turns of history.

Why would anyone build it that way? Because the model itself leaves no alternative. Lecture 5 showed you what a forward pass is: a function from a token sequence to a probability distribution over the next token. Nothing persists between calls — no variables, no notes, no recollection of having seen you before. The only “memory” a transformer has is the context window in front of it right now, which means the only way to give it a past is to put the past back into the present input. What feels like the model remembering is the orchestration layer re-reading the minutes aloud before every meeting.

You have already seen this mechanically. Lecture 7’s orchestration layer maintains the message array, appends each tool result and each model reply to it, and ships the whole array — not the delta — on every round trip. In Lecture 7 that was bookkeeping, the data structure behind the loop. Today it acquires a unit: every token in that array is billed as input, at chapter 8’s input price, on every call in which it appears. A tool result added at turn 3 is paid for again at turn 4, at turn 5, and on every turn until the session ends.

Be careful to keep this apart from a mechanism it superficially resembles. Lecture 5’s KV cache saves compute within one generation: while decoding, the server does not recompute keys and values for tokens it has already processed in that same call. That is an inference-time optimization inside a single request, invisible to you — and it changes nothing about billing. Send 200,000 tokens of history and you are billed for 200,000 input tokens, whatever the server’s internals reused. The mechanism that changes what you are billed is prompt caching — a separate, explicit, priced construct, and chapter 10’s subject. Two caches; only one of them is about money.

Say the consequence once, cleanly, because the rest of this chapter is just this sentence with numbers attached: in an agentic session, every token ever added to the conversation is billed again on every subsequent call. The system prompt is billed once per turn; the first tool result on nearly every turn; the last one exactly once. Your input cost is not a function of what you said — it is a function of what the conversation had to carry.

9.2 The formula

Now build the bill from parts, gently, with three symbols you can estimate for any real session before it starts. Call \(P\) the fixed prefix: the system prompt, the project’s CLAUDE.md, and the tool definitions — the block that sits, byte-for-byte identical, at the front of every single call. Call \(d\) the per-turn growth: each round trip appends a tool result plus the model’s reply, on average \(d\) tokens. Call \(N\) the number of tool-call round trips in the session.

Two of the three symbols deserve a closer look before the algebra. \(P\) you can inspect directly: it is the part of the context you wrote, or installed — and it is the part you control completely. \(d\) is dominated not by the model’s prose but by tool results: one verbose test run, one large file read, one long stack trace can each contribute thousands of tokens, and every one of them gets carried for the rest of the session. When you estimate \(d\) for your own work in chapter 11, it is the tool side, not the reply side, that will set the number.

Watch the first turns go by. Turn 1 sends the prefix plus your task and the first results — roughly \(P + d\). Turn 2 must re-send all of that plus its own growth: roughly \(P + 2d\). Turn 3 carries \(P + 3d\). The pattern is an arithmetic series with the prefix glued to the front of every term, and arithmetic series have a closed form you learned in school: summing \(P + n \cdot d\) over \(n = 1, \dots, N\) gives \(N \cdot P\) from the prefix plus \(d \cdot N(N+1)/2\) from the accumulation — and since \(N(N+1)/2\) is essentially \(N^{2}/2\) at any session length worth pricing:

\[\text{total input} \approx N \cdot P + \frac{d \cdot N^{2}}{2}\]

Read the formula as two competing terms. The first is linear: you pay the full prefix once per turn, so every token of CLAUDE.md is not a one-time cost but a recurring charge, billed \(N\) times per session. The second is quadratic, and it carries an unintuitive corollary: a token’s total cost depends on when it entered the conversation. The tool result from turn 2 is re-sent on every remaining turn; the one from turn 49 is re-sent once. The earliest tokens of a session are its most expensive ones — and because the second term grows with the square of the session’s length, it always wins eventually.

One honesty note, in the spirit Lecture 8 trained into you: this formula is an estimator, not an invoice. Real sessions have lumpy growth — a 9,000-token test log here, a two-line reply there — and \(d\) is the average that smooths them. The formula’s job is to put you within a factor of two of the truth before you spend anything; the exact bill is what chapter 12’s tools read off afterwards, and chapter 11 builds the full estimate-first recipe around exactly these three symbols.

9.3 The worked example

Now the example this lecture is built around — the one canonical session that every later chapter re-prices rather than inventing its own. The numbers are deliberately realistic for a Claude Code bug-fix session, not adversarial: a fixed prefix of \(P = 15{,}000\) tokens (a system prompt plus a grown-up project CLAUDE.md — chapter 12’s /context will show you your own), per-turn growth of \(d = 2{,}000\) tokens (a test run’s output here, a file read there, the model’s own replies), and \(N = 50\) round trips — an afternoon-sized agentic task, the shape of your Locust homework.

Apply the formula. The prefix term: 50 × 15,000 = 750K tokens. The accumulation term, computed exactly: 2,000 × (50 × 51)/2 = 2,000 × 1,275 = 2.55M tokens (the \(N^{2}/2\) shortcut says 2.5M — close enough to justify using it). Total input: ≈ 3.3M tokens.

Pause on the size before pricing it. The session as you experience it — the transcript, the prefix plus each turn appended once — is on the order of 115,000 tokens (that is just \(P + N \cdot d\)). The other roughly 3.2 million tokens are pure re-sending: the same history, shipped again and again, billed at full input price each time. Nothing you can scroll through accounts for the bill; the bill lives in the repetition.

If you want a sanity check rather than a formula, reason like an engineer: fifty calls averaging 3.3M ÷ 50 = 66K tokens each. Plausible? The calls grow linearly from about 17K (first turn) to about 115K (last turn), and the average of a linear ramp is its midpoint — about 66K. The formula is nothing more mysterious than that midpoint times the number of calls.

Now price it, at chapter 8’s mid-tier price point of $3/$15 — Sonnet 4.6’s row, as of June 2026. Input: 3.3M tokens at $3 per MTok ≈ $9.90. And the output? Across all fifty turns the model actually generated only about 25K tokens — an average reply of five hundred tokens: short analyses, tool calls, and a final diff — which at $15 per MTok comes to ≈ $0.38. The same session, re-priced up one vendor’s ladder:

Model (price per MTok, as of June 2026) Input bill (≈ 3.3M tokens) Output bill (≈ 25K tokens) Session total
Sonnet 4.6 ($3 / $15) ≈ $9.90 ≈ $0.38 ≈ $10.30
Opus 4.8 ($5 / $25) ≈ $16.50 ≈ $0.63 ≈ $17
Fable 5 ($10 / $50) ≈ $33.00 ≈ $1.25 ≈ $34

Walk the first row again, slowly. Nine dollars ninety of input against thirty-eight cents of output — the input bill is roughly twenty-six times the output bill. Context re-sending, not generation, dominates agentic cost. Everything your intuition wants to call “the work” — the model writing the analysis, writing the patch — is the thirty-eight cents. The $9.90 is the model re-reading its own conversation, fifty times over. Your intuition prices the writing; the API bills the re-reading. And notice that chapter 8’s price asymmetry made this worse than it looks: input is the cheap token class, and it still dominates — by volume.

The second punchline hides in the formula’s exponent. Total input grows quadratically with turn count. Push \(N\) from 50 to 100 with the same prefix and the same per-turn growth: the quadratic term becomes 2,000 × 10,000/2 = 10M tokens, the total roughly 11.5M tokens, and the input bill roughly $34.50 on the same $3 model — all three numbers approximate, and all three falling straight out of the same formula. Double the turns, and the dominant term roughly quadruples while the bill more than triples. An agent that wanders — re-reads files it has already seen, retries failed tools, burns round trips — does not cost you linearly more. It costs you quadratically more.

The hundred-turn version also trips a wire you saw in chapter 8’s table. Its final calls each carry about 215K tokens of context (the prefix plus a hundred turns of growth) — past the 200K threshold where Gemini 3.1 Pro’s pricing steps from $2/$12 to $4/$18, and within sight of GPT-5.5’s 272K surcharge line. The quadratic does not merely grow the bill; long sessions push their late, largest calls across the context thresholds where the price per token itself steps up. The two effects compound exactly when you can least afford it.

Read the table’s three rows together and you see chapter 8’s lever acting on this chapter’s volume: the identical session costs about $10, $17, or $34 depending on which rung of one vendor’s ladder runs it. The two levers multiply — model choice scales the whole bill, session shape decides how large a bill there is to scale.

And one backward glance at Lecture 8 while the numbers are warm. When the scaffold problem told you that vendor harnesses run best-of-N at 10–100 times the inference cost of a single attempt, this is the arithmetic that multiplier multiplies: every additional rollout is another quadratic session like the one you just priced. The missing cost column from Lecture 8 was never going to contain small numbers — and now you can compute, rather than suspect, why.

A note for the exam, because this is precisely the style of the course’s calculation exercises: given \(P\), \(d\), \(N\), and a price pair, you should be able to produce the total input, the input cost, and the output cost on paper, no calculator heroics required. The lab worksheet (chapter 14) has you verify today’s arithmetic by hand and then run the same formula on your own estimates before your Locust attempt. Carry the formula, not the figures — the figures expire with chapter 8’s table; the formula does not.

9.4 What the quadratic means

What do you do about a quadratic? The formula’s three symbols are three levers, and each attacks a different term. Keep \(P\) lean: the prefix is a recurring charge, so a bloated instruction file is paid for \(N\) times in every session that uses it (chapter 11 shows you how to count it before you start). Keep \(N\) small: plan before acting, prefer a few substantial turns over many exploratory ones — \(N\) is the only symbol that enters the formula squared. And cap the accumulation: summarize instead of carrying everything, replacing old history with bounded summaries rather than dragging every tool result to the end of the session — an orchestration-layer discipline you will meet again later in Act 3.

Be honest about what the third lever costs, though, because it is not free the way the formula makes it look. A summary carries fewer tokens because it carries less information, and the model cannot ask a summary follow-up questions the way it can re-read a full test log. Compressing history is a judgment call about what the rest of the session will need — engineering, not bookkeeping — and getting it wrong buys you a cheaper session that fails. The cost side of that trade you can now compute; the quality side is a later lecture’s craft.

But the biggest single lever is none of these, and you can see it by staring at the 3.3M once more. Almost all of those tokens are the same tokens, re-sent: the prefix did not change once in fifty turns, and the history only ever grew at its end — everything before the newest turn is byte-for-byte identical to what the API processed one call earlier. Paying full input price to have the model re-read text it read seconds ago is exactly the kind of structural waste a market learns to price, and the vendors have: they will sell you the re-read of an unchanged prefix at a steep discount, if you structure your prompt to earn it. Re-running this chapter’s $9.90 bill under that discount is chapter 10’s opening move — and the result lands closer to one dollar than to ten.

10. Prompt Caching: Re-running the Bill

Chapter 9 closed by naming the waste in its own bill: nearly all of the $9.90 was the same tokens, re-sent — and from the provider’s side that waste is physical, GPU time spent re-running a prefill whose result it had computed and discarded one call earlier. Prompt caching exists because the waste was mutual: the provider keeps the compute it saves, and passes part of the saving back to you as a discount. This chapter is about that discount — how it is priced, what it does to chapter 9’s bill, and the small set of engineering rules that decide whether you actually receive it.

The mechanism needs no new machinery, because you have owned it since Act 2. Lecture 5’s KV cache keeps the computed key and value vectors of earlier tokens so that, within one response, each generation step extends the cache instead of re-projecting the entire sequence. Prompt caching — whose mechanics Lecture 6 §6.5 already walked through — is that idea promoted from within-one-generation to across API calls: the provider persists the KV state of a stable prefix after a response finishes and, when the next request arrives carrying a byte-identical prefix, reloads the stored state instead of re-running prefill.

That promotion changes the idea’s category. Inside one generation, the KV cache was an implementation detail you could ignore — automatic, invisible, free. Across API calls it acquires a price sheet, a time-to-live, and a set of rules about request layout that you can follow or violate. Prompt caching is the KV cache promoted from a per-request compute optimization to a cross-request billing construct — the same vectors as in Lecture 5, except that now whether they get recomputed is a line on your invoice. Lecture 6 taught you the mechanism so you would stop worrying about large instruction files. This chapter attaches the numbers Lecture 6 deliberately left out.

10.1 The price sheet: writes, reads, break-even

Caching is not free for the provider — parked KV state occupies memory on inference hardware that could serve other requests — so the pricing has two sides: a premium to write a prefix into the cache, and a steep discount to read it back. The multipliers, as of June 2026:

Provider Cache write Cache read How it is triggered
Anthropic 1.25× base input (5-minute TTL) or 2× (1-hour TTL) ~0.1× base input Explicit: you mark cache breakpoints in the request
OpenAI No write premium ~0.1× base input Automatic prefix caching
DeepSeek — ~1/50th of the input price Automatic; cache hits are the effective price

Run the arithmetic on the standard Anthropic tier and you see why the discount is structured as a bet you almost always win. Sending a prefix twice uncached costs 2.0 price units; writing it once at 1.25× and reading it once at ~0.1× costs 1.35 — the break-even arrives at the second request. An agentic session issues its second request within seconds of its first, comfortably inside even the five-minute window, and every request after that is nearly pure discount. The 2× one-hour tier exists for prefixes reused on a slower rhythm — a pipeline that fires every twenty minutes, a session you keep returning to between meetings — where the longer survival is worth the more expensive write. One caveat the multipliers do not show: a TTL is an upper bound on the cache’s patience, not a guarantee — Lecture 6 §6.5 already warned that vendors evict on idle or capacity pressure, so a hit is something you assume in the estimate and verify on the invoice.

OpenAI makes the whole decision for you: no markup to write, automatic detection of repeated prefixes, ~0.1× on hits. Comfortable — but automation also means the request layout of §10.3 is the only lever you hold, because you cannot mark what should be cached; you can only arrange the request so the automatic matcher finds it.

DeepSeek shows what caching looks like when it stops being a feature and becomes the business model: cache hits are billed at roughly one-fiftieth of an input price that already sits at the market’s frontier-scale floor (chapter 8). For an agentic workload, where most input tokens are re-reads of an unchanged prefix, caching is not a discount on DeepSeek’s price; it effectively is DeepSeek’s price — quote that provider’s headline number without the cache-hit rate and you have quoted almost nothing.

One more line on the price sheet, easy to miss because it has nothing to do with caching: Anthropic, OpenAI, and Gemini all offer batch APIs at −50% for asynchronous processing — submit a job, collect results later, no interactive latency guarantee. The discount stacks with caching. For an interactive coding session it is useless; for the eval-style workloads this arc keeps circling — costing a benchmark run, re-grading a few hundred instances overnight — it is the difference between an experiment you propose and an experiment you run.

A closing caveat that the hedging above already implied: the multipliers are pricing policy, not physics. The mechanism — stored KV state, prefix matching, eviction under pressure — is stable knowledge; the numbers 1.25, 0.1, and 1/50 are business decisions revisable any quarter, which is why every figure in this chapter carries its “as of June 2026” stamp and why chapter 3’s refresh discipline applies to this table exactly as it applies to the supplier map.

10.2 The cached re-run

Now re-run chapter 9’s bill with caching on. Same session: \(P = 15\text{K}\), \(d = 2\text{K}\), \(N = 50\), same $3-per-MTok mid-tier model at June-2026 prices — one change: the prefix and the accumulated history are marked cacheable. The shape of what happens is simple: nearly all of the 3.3M input tokens become ~0.1× cache reads, because every re-sent copy of the prefix and every re-read line of history now hits the cache, while each turn’s 2K of genuinely new tokens is written once at the write premium. Re-priced, the session lands at approximately $1.39 — an ~86% reduction on the $9.90 uncached bill.

Sit with that pair of numbers, because you have met this exact comparison before, in qualitative form. Lecture 6 §6.5 closed its caching section with a sentence built to reassure: prompt caching changes the economics of large instruction files, not the influence story. In May, that was an assertion you took on trust — caching makes the big prefix affordable, somehow, to some degree. What Lecture 6 could only state qualitatively, you can now compute: on one fifty-turn session, the economics of a large instruction file are $9.90 without caching and $1.39 with it. And the other half of Lecture 6’s sentence survives untouched: those cached tokens still occupy positions, still consume attention mass, still dilute. Caching changes what the prefix costs you, not what it does to the model — the reason to keep CLAUDE.md concise was never money, and it still isn’t.

The same cache hit that saves the money also saves the time. Prefill is work, and skipping it shows up not only on the invoice but in time-to-first-token: a session whose 15K-token prefix is reloaded instead of recomputed responds noticeably faster on every turn. Cost and latency move together here because they share a cause — both are prefill, and the cache removes prefill. That matters for the engineering instinct: when caching silently stops working, the first symptom you feel is often the latency, before any bill arrives.

One honest qualifier before the rules. A read at 0.1× is not a read at zero. The quadratic of chapter 9 is discounted, not repealed: the \(N^2\) term still grows with every turn, just on a ten-times-gentler slope, and a session that wanders long enough will eventually make even tenth-price re-reads the dominant line on the bill. Caching buys you headroom; it does not buy you immunity from turn count.

10.3 The caching rules

Whether you receive any of this depends on a single mechanical fact. A prompt cache is a prefix match: one changed byte invalidates everything downstream of it. The provider does not diff your request against the cached one and reuse whatever overlaps — it walks from the first token until the first mismatch, and everything after that point is recomputed and billed accordingly. From that one fact, the entire engineering discipline follows:

Notice that an agentic loop satisfies these rules almost by construction, which is why the discount lands so hard on exactly the workload chapter 9 priced: conversation history is append-only, so each turn’s request extends the previous request’s prefix rather than editing it, and each turn’s new tokens are written to the cache once and read cheaply ever after. The quadratic structure that made the uncached bill explode is the same structure that makes the cached bill collapse.

So caching changes what a session costs — by an order of magnitude, under rules you now know how to follow. What it does not do is tell you what a session will cost before you start it: everything in this chapter re-priced a bill that had already been run. But the decisions that matter — which model (chapter 7), caching on or off, whether the task is worth automating at all — all sit before the first token is sent, and they need an estimate, not a receipt. Producing that estimate — counting a prefix you can see, bounding a turn count you cannot — is chapter 11.

11. Predicting Cost Before a Task

This week’s diary prompt is an experiment on yourself: estimate the token cost of your last Claude Code session before running /cost, then record estimate against actual. The prompt is only fair if you have an estimation method, and that is what this chapter supplies. Notice what you already hold after chapters 8–10: the prices, the formula, and the caching rules. What you do not yet hold are the formula’s inputs for a session that has not happened — the prefix size \(P\), the per-turn growth \(d\), the turn count \(N\). The good news is that those three inputs are not equally unknowable: \(P\) is sitting on your disk and can be counted, \(d\) can be read off your own past sessions, and \(N\) can be bounded even though it cannot be known. The toolkit comes in three parts: the counting tools and their honest limits, one systematic bias that affects this room more than most, and a five-step recipe that turns all of it into a number you can write in the diary before any money moves.

11.1 The counting tools and their honest limits

Three instruments will count tokens for you, and each comes with a limitation you need to know before you trust it.

Anthropic’s count_tokens endpoint is the right instrument for Claude-bound text: it accepts the same request shape as a real call — system prompt, messages, and tool definitions — and returns a token count without running inference, free of charge. Free matters: you can count before every session at zero marginal cost. Counting the full request shape matters more, because tool definitions are tokens you never see in your editor but pay for on every single turn — for most students, the first count_tokens call on their actual setup is the moment the invisible part of \(P\) becomes visible. One discipline, though: Anthropic’s own documentation calls the returned number an estimate. Never write “exact” next to it — the vendor doesn’t.

OpenAI’s tiktoken is a local library: fast, offline, no API call, and faithful to the GPT tokenizers it ships. That last clause is the limitation. Applied to Claude-bound text it is the wrong instrument — wrong vocabulary, wrong merges, so the same string splits at different boundaries — and in practice it undercounts Claude tokens by roughly 15–20%. You know exactly why from Lecture 4: a tokenizer is a learned compression scheme, and two models trained with different vocabularies compress the same text differently. The error is at least directional — tiktoken on Claude-bound text reads low, so inflate accordingly.

The back-of-envelope heuristic — roughly 4 characters or 0.75 words per token for English text — is the crudest of the three and the only one that works on a napkin. It also connects the formula to things you can inspect with no tooling at all: by the heuristic, chapter 9’s 15K-token prefix is about 60KB of text on disk — a number ls -l hands you before any API enters the conversation.

Instrument What it is Cost The honest limit
count_tokens (Anthropic) API endpoint; counts the full request shape incl. tool definitions Free Returns an estimate — Anthropic's own wording; never call it exact
tiktoken (OpenAI) Local library, GPT tokenizers Free, offline GPT-only: undercounts Claude-bound text by roughly 15–20% (wrong vocabulary, wrong merges)
~4 chars / ~0.75 words per token Back-of-envelope heuristic Nothing English text only; crude — an order-of-magnitude tool, not a counter

Three instruments, three error profiles: an estimate the vendor refuses to call exact, a precise count of the wrong tokenizer, and a napkin rule. That is not a counsel of despair, because estimation does not need precision — it needs calibration. A rough count with a known bias direction is a usable budgeting instrument; a precise-looking count whose bias you cannot name is not. This is Lecture 8’s error-bar lesson transplanted from scores to budgets: the question is never “is the number exact?” but “how wrong can it be, and in which direction?”

11.2 The German premium

There is one bias that deserves its own section in this particular room. The tokenizers of every model you have used this semester were trained on English-heavy corpora — that is Lecture 4’s multilingual fairness gap, and in April it was a fairness observation. Today it is a line item: German text fragments into roughly 1.2–1.7× more tokens than English text carrying the same content. Be precise about the epistemic status of that range, in the spirit of Lecture 8’s checklist: it is a practical heuristic from working practice, not a published constant, and you should not cite it in a thesis. What is documented and citable is the phenomenon itself — tokenizer unfairness between languages, reaching up to ~15× for some languages (Petrov et al., arXiv:2305.15425). Cite the paper for the phenomenon; treat the German multiplier as a field estimate you can verify yourself in five minutes by running count_tokens on a paragraph and its translation.

Now put the premium where the money is. A German CLAUDE.md inflates \(P\): chapter 9’s 15K-token prefix, written in German, would weigh in somewhere around 18K–25.5K tokens by the heuristic range — same instructions, same influence on the model, more tokens on the meter. German issue text, German commit messages, and German documentation flowing back through tool results inflate \(d\) the same way. And the formula from chapter 9 does not pay those premiums once: a German CLAUDE.md pays its token premium on every single turn — the quadratic formula multiplies a premium on \(P\) by \(N\), and a premium on \(d\) by \(N^2/2\). A choice that looks like a style question — which language do we write our instruction files in? — is a cost-engineering decision compounded fifty times per session.

Nor does caching make the premium disappear. Chapter 10’s discounts are multiplicative — a cache read is ~0.1× of whatever the input was — so a prefix that is 1.2–1.7× larger stays 1.2–1.7× more expensive after every discount has been applied. Percentages do not erase ratios. The premium is one of the few cost levers in this lecture that is entirely free to pull: write the instruction files in English, and the discount applies before any provider is involved.

11.3 The pre-task recipe

Here is the toolkit assembled into the artifact of this chapter — five steps, run before the session starts, in the time it takes the coffee to brew:

  1. Count the prefix. count_tokens on the system prompt + CLAUDE.md + tool definitions, exactly as the request will send them. This is \(P\), and it is the one input you can know almost precisely in advance.

  2. Estimate the per-turn growth \(d\) from your own history. A typical turn adds a tool result — a file read, a test run, a diff — plus the model’s reasoning about it. Do not guess from intuition; look at what your past sessions’ tool results actually weighed. Your own logs are the dataset, and chapter 12 gives you the tools that read them.

  3. Pick \(N\) pessimistically. Agents wander: they re-read files, chase a wrong hypothesis for six turns, re-run the test suite after every edit. Budget for the 80th percentile of your past sessions, not the median — a budget that only covers the well-behaved case is not a budget, it is a hope.

  4. Apply the chapter-9 formula: total input \(\approx N \cdot P + d \cdot N^2 / 2\), then multiply by the input price and add the output term — which chapter 9 showed is usually a rounding error in agentic work.

  5. Decide caching and model before starting. The same estimate, priced three ways at June-2026 prices, is three different decisions: chapter 9’s session is ≈ $9.90 uncached or ≈ $1.39 cached on the $3-per-MTok mid-tier, ≈ $17 on Opus 4.8, ≈ $34 on Fable 5. If the cheap tier’s expected quality suffices (chapter 7, step 1), the frontier premium now needs a justification you can state in euros.

Run the recipe on the canonical example and you see it end to end: a 60KB prefix counts to ~15K tokens; your logs say a typical turn adds ~2K; your pessimistic turn budget is 50; the formula says ≈ 3.3M input tokens; the price sheet turns that into the menu in step 5. No number in that chain required a completed session — which is the entire point. The recipe is what makes chapter 7’s decisive step computable before any money is spent: cost-per-solved-task needs a cost estimate per attempt, and you can now produce one from a directory listing and your own logs. It is also exactly the arithmetic the Locust homework in chapter 14 asks you to commit to paper before you let an agent near your issue. And when the estimate comes out above what the task is worth, the formula tells you which knob to turn, because each variable is a lever: shrink \(P\) (trim the instruction files), cap \(N\) (set a turn budget and stop the wanderer), drop to a cheaper tier, or turn caching on — four interventions, each addressing a different term of the bill.

Two honest caveats on the recipe’s blind spots. First, if you work with a reasoning model, the output term stops being negligible without warning: thinking tokens are billed as output, and Lecture 6’s test-time-compute chapter showed you that the model may spend a large, task-dependent budget there — so your estimate needs a thinking-budget assumption stated next to it. Second, the recipe predicts the cost of an attempt, not of success — if the task takes three attempts, the bill is three estimates, which is precisely why chapter 7 divides cost by solved tasks rather than by API calls.

Hold the recipe’s output with the right epistemic grip. An estimate built from a counted prefix, a measured \(d\), and a pessimistic \(N\) is not a prediction — it is a hypothesis about a session that has not happened yet, and a hypothesis is only worth something if an observation can contradict it. A cost estimate is a hypothesis; this week’s diary entry — estimate first, /cost second — is its experiment. The measurement side of that experiment — /cost and /context, the local log analyzers, the telemetry exports, the spend limits that catch the estimate that was wrong by an order of magnitude — is its own discipline, and it is chapter 12.

12. Monitoring What You Actually Spent

Chapter 11 left you holding a number that does not exist yet: an estimate, assembled from a counted prefix, a guessed per-turn growth, and a pessimistic turn count. Treat that number the way this course has treated every prediction since Act 1 — as the first half of predict–observe–explain. You wrote down what you expected the tools to do before you touched them; you wrote down what the Claim Board’s numbers meant before the rubric reached them; and every time, the gap between prediction and observation was where the teaching happened. The same discipline now applies to your own money. An estimate you never check against a measured bill is not a cost model; it is folklore with better arithmetic. This chapter supplies the observation step: the instruments, at three altitudes — inside your own coding session, at the provider account that pays for it, and across an application that makes calls without you watching.

12.1 Inside Claude Code

Start at the altitude where you already work. Claude Code answers the three session-level questions — what did this cost, what is filling the window, what did the week look like — with two slash commands and one external tool.

/cost answers the first: it reports what the current session has spent so far. The habit that matters is when you run it — mid-session, not post-mortem. A cost trajectory you can still bend is information; one you can only regret is accounting.

/context answers the second, and it is chapter 9’s formula rendered live. The command shows what currently occupies the window: at the top the block that never changes — system prompt, tool definitions, your CLAUDE.md — which is exactly the fixed prefix \(P\) you pay to re-send every turn; beneath it the accumulated conversation and tool results, which are the per-turn growth \(d\) in mid-accumulation. Watching /context during a long agentic session is watching the quadratic happen: the fixed block never shrinks, the history only grows, and every turn re-submits both.

The third question outlives any single session, and for it there is ccusage (npx ccusage, nothing to install). It reads the JSONL session logs Claude Code already writes to your own disk — which has two consequences worth noticing: it works offline, and it needs no API key, because you are inspecting your own records rather than querying anyone’s service. You get daily reports, per-session breakdowns, and a blocks view that groups usage into roughly five-hour windows — a grouping that looks arbitrary today and will explain itself in chapter 13. One version note, because older write-ups still circulate: ccusage no longer ships a separate live-monitor mode; since version 18 that capability lives in its statusline command, which feeds a live usage readout into Claude Code’s own status line.

It is worth opening one of those JSONL files once, in the same spirit in which this course opened the transformer: a session log is just a record of API requests, each line carrying its token counts — which is the entire reason a third-party tool can reconstruct your bill from your own disk without asking anyone’s permission. There is no magic in cost monitoring. There is a price list (chapter 8), a count, and a multiplication; everything in this chapter is plumbing that performs the multiplication where you can see it.

All three of those instruments are personal. The moment a team needs the same visibility, Claude Code will export it: set CLAUDE_CODE_ENABLE_TELEMETRY=1, point the standard OTEL_* exporter variables at your collector, and cost and token metrics flow into whatever dashboard the team already runs. /cost is the developer’s view; OpenTelemetry is the same observation made organizational — how an engineering lead notices which repository’s sessions are the expensive ones before the invoice does.

This is also where this week’s diary deliverable lives — estimate before /cost, record both numbers, explain the gap; chapter 14 formalizes it. The step that matters is the third: a systematically wrong estimate is a measurement of your assumptions, and after two or three rounds the direction of the error tells you which one to fix — usually the per-turn growth \(d\), occasionally your optimism about the turn count.

12.2 Provider consoles

Session instruments observe; they do not protect. Everything in the previous section tells you what happened, increasingly close to real time — none of it can stop an agent loop that is happily grinding through a retry cycle at three in the morning. Protection lives one altitude up, in the provider consoles: both Anthropic’s and OpenAI’s give you the same three controls — a usage dashboard, hard spend limits, and budget alerts that fire at thresholds you choose.

The dashboards are the least important of the three. The advice that earns its bold is about the limits: set the limits below the pain threshold — an agent loop gone wrong should cost euros, not hundreds of euros. A limit is not a forecast of normal usage; it is a circuit breaker for abnormal usage, and a circuit breaker sized to your monthly budget protects the budget, not you.

The two controls have two different jobs, and you want both, staggered. An alert informs — it costs you nothing when it fires falsely, so it can afford to be paranoid: single-digit euros above a normal day’s spend is a fine first threshold. A limit halts — it will interrupt legitimate work if set too tight, so it sits higher, at the point where you would genuinely rather have the interruption than the bill. The gap between the two is your reaction window: the alert wakes you up while the limit is still far enough away that you can look at what is running and decide, instead of discovering both at once.

Chapter 9 tells you why the comfortable default — alert at 50%, 75%, 100% of a monthly cap — is mis-sized for this failure mode. A runaway loop does not burn money at a constant rate: the quadratic term means its spend accelerates, because every turn re-submits a longer history than the turn before — turn 100 costs more than turn 10. Intuitions built on linear burn (“at this rate I’ll hit the cap in three weeks”) fail in exactly the wrong direction: by the time a late alert fires, you are on the steepest part of the curve. Alerts must trigger early, at euros, while the quadratic is still shallow.

12.3 The application level

The third altitude is the one most of you will meet in industry: LLM calls inside a product or a pipeline, where “what did I spend” decomposes into “which API key, which feature, which customer, which request”. Per-session eyeballing does not survive that transition; attribution has to be built into the call path. Four tools cover the territory, as of June 2026:

Tool Where it sits What it gives you
LiteLLM A proxy between your application and the providers Per-key budgets and model routing — the spend limit of the previous section, but per team, per service, per customer
Langfuse An observability platform your code reports traces to Per-trace cost attribution: every request carries its own bill, so you learn which feature is expensive, not just which month was
Helicone A logging layer in front of the API Request-level logging and usage observability with minimal integration effort
OpenRouter The aggregator you met as a rankings platform in chapter 5 Unified billing across providers: one invoice, many vendors; no markup on inference itself, a ~5.5% fee on credit purchases (as of June 2026)

None of these adds capability. All of them add attribution — and attribution is what turns a bill from a monthly surprise into an engineering signal you can route, cap, and assign.

You will not need any of the four for this course’s labs; your own sessions and the consoles cover everything we measure. The reason they get a table anyway is that the transition happens fast in practice: the week your team’s prototype gets a second user is the week “what did I spend” becomes “what did we spend on what” — and retrofitting attribution onto a month of unlabeled API calls is a reconstruction job, not a query.

Notice, finally, what every instrument and limit in this chapter has quietly assumed: that you pay for tokens à la carte — that each request meets chapter 8’s price list and produces a dollar figure somewhere. For most students in this room, that assumption is false. You pay a flat monthly seat, your interface shows no dollars at all, and your sessions are metered in something other than money. That is not a loophole in this lecture; it is a different metering model with engineering consequences of its own — and it is chapter 13.

13. Subscription vs. API

Everything in Block B so far has priced tokens à la carte: chapter 8 gave you the menu, chapter 9 the formula for what a session orders, chapter 10 the discount, chapter 11 the estimate, and chapter 12 the measured bill — which closed by naming the students its assumption quietly excluded. This chapter exists for them, which is to say for most of you, because the arithmetic of the last five chapters does not stop applying when the bill disappears from view. It gets metered differently, and the metering changes how you should engineer.

13.1 How subscriptions meter

A flat-rate plan does not sell you tokens; it sells you allowances. Concretely — these are the mechanics as of June 2026 — subscription usage is metered by rolling windows of roughly five hours, with weekly caps layered on top: an overall weekly ceiling plus model-specific ones. Within a window you draw freely against its allowance; when the allowance is gone, you meet the message every heavy user eventually meets: usage limit reached. Read that message precisely. “Usage limit reached” means the window is exhausted: your work is paused, not lost. The session, your files, your context — all of it survives. You are waiting for a window to reset, not recovering from a failure.

The metering rules are themselves a moving target, and the most recent move is worth reading with Claim Board precision. On 2026-05-06 the plans changed: the five-hour allowances were doubled, and the peak-hour reductions were removed — but the weekly caps did not change. Notice what a headline like “limits doubled” invites you to skip: which limit? A user who reads “doubled” and plans a week of doubled throughput will hit the unchanged weekly ceiling and feel cheated by arithmetic that was published all along. A pricing page deserves the same reading discipline as a benchmark table — find the quantity behind the headline before you build on it.

And the window meter is not different physics, only a different unit. Underneath your seat, the same tokens flow, the same quadratic accumulates, the same caching applies. A wasteful session does not cost you dollars; it costs you window, exhausting the allowance sooner — so every economy this lecture taught, from trimming the prefix to caching the stable part to keeping the per-turn growth small, converts one-to-one from saving money into saving allowance. You even own the instrument already: chapter 12’s ccusage blocks view groups your local logs into exactly these five-hour windows. That “arbitrary” grouping was this chapter, waiting.

The window also imposes a shape on your working day that the API never does, and it is worth noticing consciously. A long agentic task started four hours into a window is a task that may pause at the worst possible moment; the same task started against a fresh window runs with the full allowance ahead of it. Heavy users learn to spend their windows the way they spend attention — the expensive, many-turn work early, the cheap questions whenever. And a pause landing mid-task is the small, daily version of a constraint this chapter closes with in §13.3: work that cannot survive an interruption is work you will eventually pay for twice.

13.2 When each wins

The comparison becomes honest the moment you price your own working pattern both ways. Take chapter 9’s session — the 50-turn agentic afternoon that comes to roughly $10.30 uncached in API terms ($9.90 of input, $0.38 of output, on the mid-tier $3/$15 model). Run one such session every working day and the month lands around $200–300 at June-2026 API list prices — one session per working day puts you at the band’s lower end; a second session on the busy days pushes toward its upper. Caching (chapter 10) pulls that down substantially, but the order of magnitude is the point: against a daily habit of real agentic work, a flat seat is not a convenience purchase — it is the rational one. Predictable, human-paced, interactive use is exactly the workload subscriptions are priced for.

Reverse every one of those adjectives and the API wins instead:

Workload Pay with Why
Daily interactive development — the chapter-9 session, every working day Subscription API-equivalent spend ≈ $200–300/month at June-2026 list prices; a window pause costs you minutes of waiting, never money
Bursty experimentation — a heavy eval week, then silence API You pay for the burst and nothing for the quiet weeks; a seat bills you for both
CI and per-PR automation API + caching + batch No human rhythm to subsidize; repeated prefixes cache well, and asynchronous jobs take the batch tier's −50% (which stacks with caching, as of June 2026)
Production features API behind a proxy Needs the attribution and per-key budgets of chapter 12, not a login tied to one human

The bottom three rows share one sentence: a pipeline does not need a seat; it needs a budget. A seat is priced around a human’s working rhythm — sessions, evenings, weekends, the five-hour pulse of attention. A nightly job has no rhythm to subsidize. What it needs is the API, chapter 10’s caching for its repeated prefixes, the batch tier for anything that can wait, and a chapter 12 spend limit standing guard while nobody watches.

In practice, then, the question for a team is rarely either/or, and the stable pattern as of June 2026 is both, split by workload: seats for the humans, metered keys for the machines. Every developer gets a subscription for interactive work, where the window meter is a tolerable inconvenience and the flat price beats the daily-session arithmetic above; everything that runs without a human — CI, evals, the product itself — goes through API keys behind chapter 12’s limits and attribution. The mistake worth naming is the crossover in either direction: a production feature wired through someone’s personal seat (no attribution, no limits, and it dies with their account), or a fleet of developers pushed onto raw API keys because “we already have billing” (every one of them now generates an unbounded line item that a seat would have capped).

One claim from chapter 2’s wave belongs to this chapter: Claim 9, “one subscription seat pays for itself if it saves one developer-hour per month.” The arithmetic direction is easy, and you do not need numbers from me to check it: look up the seat price, then ask what one hour of your time costs your employer fully loaded — salary, social contributions, overhead — and notice that for a German developer the seat is the cheaper of the two. If the premise holds, the conclusion follows. But the premise is the claim, and it is doing all the work: saves an hour measured how, against what baseline — the same developer without the tool, doing which tasks? — observed on whose work? As pinned, the claim is unfalsifiable: it has the grammar of a measurement and no measurement procedure, the failure pattern Lecture 8’s nine questions taught you to name — the spirit of question 5, what exactly was measured? We do not rate it here. You will rate it in the lab (chapter 14), and by then you will hold the one thing that can move it off “unfalsifiable”: a measured hour of your own.

13.3 The resumability constraint

One engineering consequence of window metering deserves its own section, because it changes how you build rather than how you pay. On the API, a job that is too big returns a large bill. On a subscription, a job bigger than one window does something more disruptive: it stops mid-flight, at a point you did not choose, and continues only when the window resets. The pause itself is harmless — §13.1: paused, not lost. What is not harmless is a workflow that cannot survive the pause.

If your job is bigger than one usage window, your workflow must be resumable: idempotent steps, a checkpoint after every completed unit, and a restart that continues from the last checkpoint instead of re-doing — and re-paying for — work that already succeeded. The three properties are worth one sentence each, because they are design decisions, not features you can bolt on afterwards. Idempotent means a step is safe to run twice — it checks whether its output already exists before paying to produce it, so a restart that overshoots costs nothing. A checkpoint means progress is recorded somewhere durable — the filesystem, a commit — rather than living only inside a conversation context that the pause may not preserve. And a restartable job is one whose next action can be computed from those checkpoints alone, so that “continue” requires no memory of the run that stopped. Nothing in that list is subscription-specific, either. On the API, the same properties protect you from a different enemy: a crash at turn 40 of a 50-turn session, restarted from scratch, re-pays the entire quadratic that chapter 9 priced. Resumability is cost engineering under both meters; the window meter merely makes it impossible to postpone.

And it is not hypothetical. This repository — the same one that builds the course website you are reading — contains a real system built around exactly that constraint, and it is the finale’s case study. That is all you get today; the course finale opens it up properly. For now, carry the constraint, not the implementation.

With that, the loop this lecture promised is complete: a price list you can read (chapter 8), a formula that predicts a session (chapter 9), a cache that re-prices it (chapter 10), an estimate before the fact (chapter 11), a measurement after it (chapter 12), and a deliberate choice of payment model (this chapter). What remains is to run the loop yourself, end to end, on a task you own — your Locust issue from last week. That is chapter 14.

14. Lab and Deliverables

Thirteen chapters, and every number in them was either somebody else’s price or a bill you only predicted. You watched the supplier map expire (chapter 3), toured platforms that each measure something adjacent to what you need (chapter 5), priced a fifty-turn session that never ran (chapter 9), and re-priced it cached (chapter 10). Lecture 8 said it about scores, and it holds word for word for bills: distrust of numbers you did not produce decays into cynicism unless you eventually produce your own. The cure is the same as last week’s, with one refinement that is this lecture’s signature move: estimate first, then measure, and treat the gap between the two numbers as the actual deliverable. A bill you merely read tells you what something cost. An estimate written down before the run tells you where your model of the system is wrong — which term you misjudged, which discount you forgot, which turn count you were optimistic about — and that is the knowledge that compounds.

So this chapter points last week’s lab machinery at money. Four in-class exercises walk from other people’s numbers to your own: a model card’s footnotes, a benchmark run’s price tag, your own logs, your own policy. The homework is step 2 of the Locust thread — the step every chapter of this lecture has been quietly arming. The reflection log records one estimate-versus-actual gap in the diary. The optional reading collects the primary sources, including the ones that are designed to expire.

14.1 In-class exercises

Four exercises run in the lab slot. The first two are paper, pencil, and shared arithmetic, in the same deliberately analog spirit as Lecture 8’s lab; the last two run against your own logs and your own future behaviour.

  1. Find the footnote (pairs, ~15 minutes). Each pair gets the printed benchmark pages of two real model cards — pulled this week, because chapter 3 is right that anything older is archaeology. Hunt the asterisks with chapter 6’s list in hand:

    • pass@1, or best-of-\(N\) wearing a politer name (“parallel test-time compute”)?

    • Is there a thinking budget in the fine print — and is the price of that budget anywhere on the page?

    • Which subset of which benchmark, at which sampling settings, inside whose scaffold?

    • And the quiet one: did the tokenizer change between model generations? Chapter 6 showed why that footnote silently corrupts every per-token price comparison across a model family.

    Then each pair runs one pinned claim from chapter 2’s wave — Claim 6 or Claim 7 are the natural candidates — through the nine questions plus the cost questions this lecture stacked on top of them (chapter 7): cheapest per what, frontier-quality by which measurement — and does “90% cheaper” survive tokenizer differences, caching, and tokens-per-task? Five minutes of share-out. The board rule has not changed since Lecture 8: a claim only comes down rated.

  2. Cost a benchmark run (worksheet). Suppose you wanted to reproduce a SWE-bench Verified number instead of believing one — all 500 instances, one attempt each. The worksheet has you price that run for two models from chapter 8’s table; the default pairing is the $3/$15 mid-tier against the $0.14/$0.28 floor, both as of June 2026 and re-checked against the vendor pages before you trust them. It does not hand you a flat per-instance token figure, because flat figures are how people fool themselves. You derive it:

    • The worksheet stipulates a per-instance profile — \(P = 15{,}000\), \(d = 2{,}000\), \(N = 20\): chapter 9’s session, shortened to a single benchmark instance. Run it through the formula yourself. First check: which term dominates even at twenty turns?

    • Multiply by 500 instances and price the total at both models’ input rates. Ignore output tokens on the first pass — chapter 9 showed why that approximation is honest — then add them back as a correction and confirm it barely moves.

    • Plot each model as one point: cost on one axis, the best public Verified score you can find for it on the other (chapter 5 told you where to look — and question 9 of last week’s checklist still applies to whatever you find there).

    • Follow-up question, if your table is fast: what does caching do to the ranking? Re-price the run with chapter 10’s read rates and check whether the gap between the two models widens or narrows.

    • Flag the worksheet’s own assumption the way you would flag any benchmark’s: the token profile is stipulated, not measured. The homework replaces it with a measured one.

    Two points, two axes — and the leaderboard’s missing column becomes visible: a leaderboard without a cost axis is half a Pareto chart (Kapoor et al., AI Agents That Matter — the paper behind Lecture 8’s question six). The debrief ties the plot back to Claims 6 and 7: “cheapest per token” survives it only if the cheap model’s accuracy and tokens-per-task cooperate. Cost-per-solved-task decides, not cost-per-token (chapter 7).

  3. The own-session audit. Run npx ccusage on your own machine — it reads the local JSONL logs Claude Code already writes, offline, no API key, no account. In order:

    • Before you run it, write two guesses on the worksheet: which session this month was your most expensive, and roughly what the month cost — in euros or, on a subscription seat, in windows.

    • Then look. The daily and session reports answer the first guess; the blocks view groups your usage into the five-hour windows of chapter 13, and the shape of your blocks — bursty, deadline-clustered, or steady — is the most informative picture of your own consumption you will get today. Most people miss their most-expensive-session guess by a factor, not a percentage. That is normal, and it is the point.

    • If ccusage shows you nothing because your usage runs through an API key rather than a seat, the provider consoles of chapter 12 are your equivalent — the exercise is the audit, not the tool.

    • The audit closes with the last claim of the wave: Claim 9 — one seat pays for itself if it saves one developer-hour per month. You now hold everything the rating needs: chapter 13 told you what a seat actually meters (rolling windows and weekly caps, not hours saved), and your blocks view just showed you what your own usage pattern is worth. Rate it in pairs, missing disclosures listed — saved how, measured on whose tasks, valued at what hourly rate — and bring the verdict to the share-out.

  4. The budget policy (short, written). If you drafted a context budget policy in Lecture 7’s orchestration lab, extend it; if not, start one now — half a page is enough. It needs four rules with numbers in them:

    • a maximum CLAUDE.md size in tokens, plus the counter you will measure it with (chapter 11);

    • a cache-discipline rule: stable prefix first, volatile content last, nothing time-stamped above the fold (chapter 10);

    • a routing rule naming which task classes go to the cheap tier and which justify the frontier (chapter 7, step one);

    • a monthly alert threshold, set where it stings before it hurts (chapter 12).

    A policy without numbers is a wish, and a policy no tooling reads is a memo. Give every rule a number and a place where the number is checked.

Four things leave the room with you: a rated claim or two, a worksheet plot with a cost axis, an audited month, and a policy with numbers in it. Keep all four. The policy, in particular, comes back: the context-engineering labs ahead will be measured against it.

14.2 Homework

One item — but it is the item this lecture exists for, and the second step of the thread that ends in the finale. Do it before the next session, and do parts (a) and (b) on different days if you can: an estimate made five minutes before the run is half a prediction, half a peek.

Locust thread, step 2: cost an agent attempt. Last week you built a benchmark instance by hand — a closed Locust issue with a fail-to-pass test you verified yourself. This week you run the economics on it, in the order the whole lecture insisted on: estimate first, measure second.

Two notes on spirit. First: a failed attempt is a data point, not a failed homework — a benchmark whose cost column only contains successes would flunk the very checklist you built last week, and Locust-bench needs its expensive failures on the record as much as its cheap wins.

Second: if your seat hits a usage window mid-attempt, record that too. Chapter 13 told you the work is paused, not lost — and the pause itself is part of what a seat costs.

Bring the record to the finale. The class will assemble everyone’s attempts into Locust-bench — a resolution rate, a cost spread, and the audit you already know how to run against your own grader. Locust-bench carries a cost column from day one — the column SWE-bench never had. You are not just consuming the missing column anymore; you are filling it in for a benchmark of your own.

14.3 Reflection log

One AI-diary entry for this lecture, with the prompt the arc’s device table has carried from the start:

Estimate the token cost of your last Claude Code session before running /cost. Record estimate vs. actual.

Be specific, and be honest about the direction of the miss. Name which term of chapter 9’s formula you got most wrong — the prefix \(P\), the per-turn growth \(d\), or the turn count \(N\) — and say whether caching (chapter 10) accounts for whatever is left over. Calibration, not accuracy, is what this prompt measures: a miss you can locate in one term is worth more than a lucky hit you cannot explain.

If you have no recent session to audit, run a small real one — the homework provides an obvious candidate — and estimate that instead. The diary’s estimate-versus-actual column gets its final pass in the last lecture, next to the Claim Board, and one honest entry now is what makes that pass worth reading.

14.4 Optional reading

Everything the lecture leaned on, grouped, one line each on why it earns your time — with a warning Lecture 8’s list did not need. Half of these links are perishable by design: the documentation pages below are not background reading but the places where you refresh chapter 3’s expiring map and chapter 8’s expiring table before you rely on either.

The platforms (chapters 3 and 5):

Pricing and counting (chapters 8–11):

Monitoring and metering (chapters 12–13):

The argument (chapters 7, 9, and 11):

The lecture ends here, but nothing it set in motion does. The board’s ledger after today: Claim 3 came down in chapter 5, the board’s first verified entry — and a lesson in how little that can mean for a decision; Claim 8 fell there too, refuted-as-stated. Claim 5 stays up, sharpened but unresolved. Claims 6 and 7 come down only if your rating survived the share-out, Claim 9 stands or falls on your audit — and the sixteen-hour claim from Lecture 8 stays pinned, waiting for the finale. The diary keeps collecting estimate-versus-actual gaps, one session at a time. And the Locust thread is now armed: when the class next assembles around it, every one of you carries a costed, measured agent attempt on a real issue — one row of a benchmark with a price tag built in.

What remains of the arc puts these numbers to work. Context engineering shrinks \(P\) and \(d\) on purpose instead of by accident. Validation decides what “solved” is allowed to mean. And the finale closes the loop — the system builds the system — with a bill you will know how to read before it arrives. Bring the row. Bring the gap.

© 2026 Dominik Müller   •  Powered by Soopr   •  Theme  Moonwalk