04 Tokenization: How Machines Read

April 2026 (27997 Words, 156 Minutes)

1. Introduction

Welcome to Week 4 — and to Act II: Understand the Model Behavior.

For three weeks you have been using AI tools to write code, restructure projects, and explore a real open-source codebase. You saw impressive results — and puzzling failures. You noticed patterns you could not explain. Act II is where we explain them. Not the full mathematical depth of transformer architectures — that belongs in a later course — but the practical, mechanistic understanding you need to work effectively with these systems.

Over the next four weeks we will open the black box, one layer at a time. By the end of Act II, every puzzling observation from Act I will have an answer.

2. What Powers Your AI Tools

Before we open that box, let us step back and ask a basic question: what are these tools, actually?

Every AI coding tool you used in Act I — GitHub Copilot, Claude Code, ChatGPT — runs on a Large Language Model. You heard this term in Lecture 1, where we traced the history from AlexNet through transformers to GPT. You know the word. But what does it mean concretely?

An LLM is, at its core, a next-token prediction machine.

You already know this in abstract from Lecture 1: “GPT reads text left-to-right and predicts the next token.” But let us make it concrete with something you have actually experienced. When you asked Claude Code to “write a function that validates an email address,” what happened?

The model did not “understand” the task the way you understand it. It did not plan the function, think about edge cases, then write code. Instead, it did something much simpler and much stranger: it looked at all the tokens in your prompt and predicted which token was most likely to come next. Then it appended that token to the sequence and predicted the next one. Then the next. And the next.

Token by token, character by character, a function emerged:

def validate_email(address: str) -> bool:

Here is what that looks like from the model’s perspective. After processing the entire prompt and generating def, the model computes a probability distribution over every token in its vocabulary — tens of thousands of candidates — and picks the most likely one:

For clarity: in this diagram we show full words as tokens — validate, check, verify. What a token actually is and how text gets split into tokens is exactly what the rest of this lecture is about. For now, think of tokens as the chunks of text the model works with. The precise splitting turns out to matter enormously, and we will study it in detail. But the core mechanism is what you see here: predict the next token, append it, repeat. That is all there is. No hidden magic, no secret reasoning engine behind the scenes — just this loop, iterated thousands of times.

validate wins at 82% — not because the model “knows” what validation means, but because in its training data, the token validate overwhelmingly follows def in contexts that mention email validation. The runner-up check at 9% would also be a reasonable function name. The long tail of alternatives (verify, parse, test) trails off rapidly. The model samples from this distribution, appends the chosen token to the sequence, and repeats.

Not because the model planned it. But because the statistical patterns of Python code in its training data made each successive token highly probable given all the tokens before it. def is likely after a request to write a function. validate is likely given the task description. _email is likely after validate. ( is likely after a function name. And so on. One token at a time, a complete function emerges.

Code generation is text prediction. Every line of code Copilot suggested, every function Claude Code wrote, every refactoring ChatGPT proposed — all of it was produced by this exact mechanism: predict the most likely next token, append it, predict again. The model was not coding. It was completing sequences in a way that statistically resembles code. There is no deeper secret.

This is why understanding LLMs matters. Every strength and every limitation of your AI tools traces back to how this prediction machine works. The mystery board we are about to build is not academic — it explains the practical tool behavior you experienced every day in Act I.

3. The Unsolved Mysteries of Act I

With that framing, let us be explicit about what we cannot yet explain. These are the observations you made during Lectures 1 through 3 that had no satisfying answer at the time. All of them trace back to the next-token prediction machine that powers your tools. We will collect them now, write them down, and track them across the next four lectures. By the end of Lecture 7, every one of them will be resolved.

We track these on the Mystery Board — a living page that shows which mysteries are solved, which are partially answered, and which remain open. Check it after each lecture to see the progress.

Here are the mysteries:

Mystery 1: Why did CLAUDE.md improve results? In Lecture 3, you added a 50-line text file to the Locust project and the model’s output quality improved dramatically. But CLAUDE.md is just a text file. It is not a configuration panel, not a settings dialog, not a plugin. How does a plain text file change the behavior of a neural network?

Mystery 2: Why were English prompts often better than German? Several of you noticed that the same question, phrased in English versus German, produced noticeably different quality results. The model speaks both languages. Why would one outperform the other?

Mystery 3: Why was curated context better than pasting everything? Intuition says: give the model more information and it should do better. But in Lecture 3, dumping an entire repository into the prompt produced worse results than a carefully selected subset. More was less. Why?

Mystery 4: Why did the AI hallucinate API endpoints? In the Stack Overflow Challenge, some of you received confident, detailed descriptions of API endpoints that simply do not exist. The model did not say “I don’t know.” It invented something plausible and presented it as fact. Why?

Mystery 5: Why did vague prompts create plausible nonsense? In Lecture 2, you compared Approach 1 (a single vague prompt) with Approach 3 (structured, specific prompts). Approach 1 produced code that looked right but was riddled with subtle bugs. Approach 3 produced code that actually worked. Why does specificity matter so much?

Mystery 6: Why doesn’t CLAUDE.md always work? During Lecture 3 preparation, a 615-line CLAUDE.md was loaded into a session. The model generated code with lint violations that the CLAUDE.md explicitly said to catch. When asked to fix the violations, the model fixed the code but skipped the diary entry that the CLAUDE.md also required. Why does the model follow some instructions and ignore others?

Write these down. We will return to them.

4. The LLM in One Picture

You have seen a version of this diagram in the slides during Act I. Now let us look at it properly, because every concept in Act II maps to a specific part of this picture.

Walk through it left to right:

On the left: Inputs. Your system prompt, your user prompt, your project context (CLAUDE.md), and any enabled tools. This is everything the model will work with — all of it is text.

Next: the text → tokens step. All that text must be converted into numbers before the model can process it. This is tokenization — the subject of today’s lecture.

In the center: the Transformer Model. This is the neural network that does the actual prediction — the next-token machine we just discussed. It is auto-regressive: it generates one token at a time, each conditioned on everything before it. For now, this box stays locked. We open it in Lecture 5.

On the right: Output. Either raw text (the model’s response) or a tool call (a request for the system to read a file, run a command, or search something). The output feeds back into the input for the next prediction, creating the loop you can see at the bottom — the token sequence growing with each iteration.

Notice something important: CLAUDE.md is listed in the inputs box. It enters the model the exact same way your prompt does — as text that becomes tokens. There is no special “instruction channel.” No configuration system. Just text, in the same pipeline. Keep that in mind.

Today we zoom into that text → tokens step. By the end of this lecture, you will understand exactly what happens there and why it matters for everything else.

5. Why Text Must Become Numbers

So the model predicts the next token. But what is a token? And why can’t the model just work with text directly?

You already have an intuition from Lecture 1, where we discussed context windows measured in tokens and the transformer’s attention mechanism. Now let us make it precise.

The answer is fundamental: neural networks do not understand text. They understand numbers.

Think about what happened when you asked Claude Code to generate that email validation function. The model predicted def as the next token, then validate, then _email, then (. But inside the transformer, there are no words. There are no letters. There are only tensors — multi-dimensional arrays of floating-point numbers. Every operation inside the model — every matrix multiplication, every attention computation, every activation function — takes numbers as input and produces numbers as output.

Text is symbolic. The letter “c” is a symbol. The word def is a sequence of symbols. These symbols have no inherent numerical meaning. You cannot multiply def by return and get something useful. You cannot compute the dot product of two function signatures.

The transformer requires numbers. Your prompt provides text. Something must bridge the gap.

That something is the tokenizer. The tokenizer sits between human language and machine computation. Its job: convert text into a sequence of integers — token IDs — that the model can work with. Those integers then get looked up in a massive table to produce dense numerical vectors that the transformer actually processes.

Here is the complete pipeline:

Every piece of text you send to an AI tool — your prompt, your CLAUDE.md, the model’s own previous output — passes through this pipeline. Without it, there is no language model. Tokenization is not preprocessing. It is not an optional step. It is the first layer of the architecture, and every decision made during tokenization carries forward into everything the model does.

Before we dive into the details of how tokenizers work, let us trace this entire pipeline end-to-end — from text all the way to a predicted token — using a toy example small enough that you can follow every step by hand.

6. The Complete Journey in Miniature

Before we dive into the details of tokenization, let us trace the entire prediction pipeline from start to finish — using numbers small enough that you can follow every step by hand. This is the complete journey from text to predicted token, in miniature. We will study each station in depth over the coming weeks. Today, we see the whole map.

6.1 A Toy Vocabulary

Imagine a tiny language model with a vocabulary of just 10 tokens — the kind of model you might build as a teaching exercise. The vocabulary looks like this:

Ten words. A fantasy world where dragons drink, wizards mix, and trolls smash. Absurdly small — a real model has 50,000 to 200,000 tokens. But the principles are identical. Let us feed this model a sentence and watch what happens.

Input: <BOS> dragon drinks

6.2 Step 1: From Text to Token IDs

Each token is looked up in the vocabulary and replaced by its integer ID. The result is a sequence of three numbers:

Three tokens, from a vocabulary of 10. So far, nothing the transformer can work with — these are just integers. Discrete numbers with no mathematical structure. You cannot compute a gradient through an integer lookup. The model needs something richer.

6.3 Step 2: The Embedding Table

This is where the embedding matrix comes in. It is a table with one row for every token in the vocabulary and a fixed number of columns — the embedding dimension d. In our toy example, the vocabulary has V = 10 tokens and we choose d = 6. That gives us a 10 × 6 matrix \(E \in \mathbb{R}^{V \times d}\), where each row is a vector of six learnable numbers:

When the model processes the input (0, 5, 6), it looks up rows 0, 5, and 6 in this table. That is the entire operation — a table lookup. Token ID 0 maps to the vector [0.12, -0.34, 0.56, -0.78, 0.91, -0.23]. Token ID 5 maps to [0.67, -0.11, 0.43, -0.52, 0.28, 0.85]. Token ID 6 maps to [-0.39, 0.74, -0.18, 0.61, -0.45, 0.33]. The result is a 3 × 6 matrix of dense vectors — one vector per token in the input sequence. This is what the transformer receives. Not text. Not integers. Dense numerical vectors.

What do values like 0.67 or -0.52 mean? The honest answer: nothing interpretable to a human. Each number is like height, weight, or age for a person — except abstract. You cannot look at a single value and say “this encodes verb-ness” or “this encodes the letter d.” Each dimension captures some aspect of the token’s meaning that the model discovered was useful during training. The meaning lives in the pattern of values across all dimensions, and in the relationships between different tokens’ vectors.

Look at the table carefully. Notice that dragon and fire have somewhat similar patterns — both have strong positive values in d₁ and d₆. The verbs mixes, drinks, and smashes share structural similarities too. These similarities are not programmed by a human. They emerge from the statistics of the training data. Nobody writes these numbers by hand — the embedding matrix starts as random noise and is gradually adjusted through training as the model learns which vectors contribute to accurate next-token prediction.

Mathematically, the lookup operation can be expressed as a matrix multiplication: the input IDs are converted to one-hot vectors, and multiplying that one-hot matrix by the embedding matrix selects the right rows. This formulation matters because matrix multiplication is differentiable — gradients can flow backward through the embedding table, allowing the model to learn better representations over time.

Note the distinction between two different “vocabularies.” The tokenizer’s vocabulary maps each token ID to the byte sequence it represents — token 256 stores b"th", token 257 stores b"the", and so on. This is what allows lossless decoding back to text. The model’s embedding table maps each token ID to a dense vector of learned floating-point numbers — token 256 maps to a 768-dimensional vector. These are two different lookup tables, serving different purposes, indexed by the same token IDs.

6.4 Step 3: The Transformer (Preview)

After step 2, we have a 3 × 6 matrix — three rows (one per token), six columns (the embedding dimension). This matrix is the input to the transformer. Watch what happens to its shape as it flows through the system:

The transformer takes this 3 × 6 matrix and transforms it into a new 3 × 6 matrix. Same shape — still three rows, still six columns — but different values. What changed? Each vector has been refined by the context of the other vectors around it. The embedding for drinks started as a generic representation of the word “drinks.” After the transformer, it has become a representation of “drinks in the context of dragon before it” — a much more specific and useful signal.

How does the transformer do this? Three operations, stacked into a layer that repeats many times:

Positional encoding (once, at the start): Each vector gets position information added, so the model knows that dragon is token 2, not token 3 — word order matters.

Self-attention: Each token looks at every other token and asks: how relevant are you to me? The token drinks attends strongly to dragon — because who is drinking matters for predicting what comes next.

Feed-forward network (FFN): The gathered information is processed and transformed.

The attention + FFN pair forms one layer. In GPT-2, this layer repeats 12 times. In GPT-3, 96 times. Each layer takes the same-shaped matrix from the previous layer, refines the values further, and passes the result on. Layer 1 captures simple patterns (which words are nearby). Layer 50 captures abstract relationships (what kind of action is happening). Layer 96 captures task-specific intent (what token should come next).

We will open this box fully in Lecture 5. For now, the key insight is: the matrix shape stays the same (3 × 6) through every layer, but the values change. Three raw embedding vectors go in; three context-aware vectors come out.

If you have trained neural networks before, something should feel wrong here. In a fully connected network, the weight matrix has a fixed input dimension. A layer that maps 784 inputs to 256 outputs only ever accepts exactly 784 inputs. If you try to feed it 785, it breaks. So how can the same transformer accept a 3 × 6 matrix now and a 4 × 6 matrix on the next forward pass?

The answer is a key design insight of the transformer: the weights are sized for one token, not for the whole sequence.

Think of a factory with a conveyor belt. The machines along the belt — the drill, the welder, the painter — are fixed. They do not change when you put more items on the belt. Each item passes through the same machines, one at a time. Whether the belt carries 3 items or 3,000 items, each one gets the same treatment.

The transformer works the same way. The FFN is a function \(f: \mathbb{R}^d \to \mathbb{R}^d\) — it takes one 6-dimensional vector and produces one 6-dimensional vector. The same learned weights are applied to each token position independently. Whether there are 3 tokens or 3,000 tokens in the sequence, each one passes through the same FFN with the same parameters.

The Q, K, and V projection matrices that drive attention are also \(d \times d\) — per-token operations. The only place where tokens interact is in the attention scores: each token’s query is compared against every other token’s key via a dot product. For 3 tokens this produces a 3 × 3 score matrix; for 4 tokens, a 4 × 4 matrix. But this score matrix is computed dynamically from the data — it contains no learned parameters whose shape depends on sequence length.

So the architecture is really \(f: \mathbb{R}^d \to \mathbb{R}^d\) per token, with dynamic coupling between positions through attention. The sequence length \(n\) never appears in any weight matrix. This is why the same model can process a 3-word prompt and a 3,000-word document — the weights are identical, only the number of items on the conveyor belt changes. We will make this fully precise in Lecture 5.

6.5 Step 4: From Vectors Back to Words

The transformer has produced three refined vectors — one for each token in the sequence. But we only need one prediction: what comes next? The model takes the last token’s vector — the one for drinks — because it is the only position where the model needs to predict a continuation. (The <BOS> and dragon positions already have known next tokens.)

This single vector has 6 dimensions. The vocabulary has 10 tokens. To produce a score for each possible next token, the model multiplies this 6-dimensional vector by a 10 × 6 output weight matrix — one row per vocabulary token. The result: 10 numbers, called logits. Each logit says: “How likely is this token to come next?”

The raw logits are just numbers — they can be negative, and they do not sum to anything meaningful. Softmax converts them into probabilities: all positive, summing to 1, with higher logits producing exponentially higher probabilities. The bar chart shows the result: potion dominates at 62%. fire is the runner-up at 18% — a dragon might drink fire, after all. The other tokens trail off into single digits.

<BOS> dragon drinks → potion. One forward pass, one predicted token. The model appends potion to the sequence, and the whole process repeats — now with a 4 × 6 matrix as input, predicting the fifth token. Then a 5 × 6 matrix predicting the sixth. Token by token, a story emerges.

This is exactly the next-token prediction loop from Chapter 2 — except now you can trace every step mechanically, watching the matrices change shape and value at each station.

6.6 Scaling Up

Our toy example has 10 tokens and 6 dimensions — 60 parameters in the embedding table alone. Real models are orders of magnitude larger:

For GPT-2, this single table has 50,257 rows (one for every token in the vocabulary) and 768 columns (the dimensionality of the embeddings). That is over 38 million parameters — just for this one lookup table, before the transformer has done any actual computation.

The numbers get larger for bigger models. GPT-3 uses 12,288-dimensional embeddings. Modern frontier models use even more. The embedding matrix alone can contain hundreds of millions of parameters. But the principle is identical to our toy example: each token ID maps to a row, and that row is a vector of learned numbers. The only difference is scale.

6.7 Embeddings Encode Meaning

In a trained model at scale, the patterns we glimpsed in our toy example play out across tens of thousands of tokens with much greater clarity. The vectors for “king” and “queen” are closer together than the vectors for “king” and “banana.” The vectors for “Python” (the programming language) and “Java” are closer together than the vectors for “Python” and “python” (the snake) — though all three share some similarity. These relationships are not programmed; they emerge from the statistics of the training data.

The embedding vectors are what the transformer actually thinks with. Every downstream operation — self-attention, feed-forward layers, output projection — takes these vectors as input. The tokenizer’s job is to produce the integer IDs that index into this table. The embedding matrix then translates those IDs into the rich numerical representations the model requires.

You have now seen the complete journey from text to prediction — in miniature. The central question for today: how should we split text into tokens? Our toy vocabulary used whole words — dragon, drinks, potion. Real models do something more sophisticated. To see why, let us start with a puzzle.

7. What the Model Cannot See

The treatment of tokenization in this and the following chapters draws on the excellent video “How AI Actually Reads Text | Tokenization” by Attention Visualized, which provides a clear visual walkthrough of these concepts. The narrative structure, toy example, diagrams, and pedagogical framing are original to this course.

In 2023, a simple question broke the internet: “How many r’s are in the word strawberry?”

Every major language model — GPT-4, Claude, Gemini, Llama — answered confidently: two. The correct answer is three: strawberry. A system that could write working code and translate between languages could not count to three.

If you try this today, most frontier models will get it right. They have been specifically trained on this example and similar letter-counting tasks. But the underlying problem has not gone away — it has just been papered over for the most famous cases. Try a less well-known word instead:

Ask any model: “How many s’s are in ‘mississippi’?” Or: “How many c’s are in ‘Connecticut’?”

The question is: why? These are next-token prediction machines that can reason about complex problems. Why would they fail at something as simple as counting letters?

The answer has to do with tokenization. Before the word “strawberry” ever reaches the neural network, the tokenizer splits it into chunks — perhaps str, aw, berry:

The model sees three tokens, not ten letters. The r’s are scattered across token boundaries. But why does the tokenizer split it this way? Why not one token per letter, or one token per word? What determines where the cuts fall?

Try one more thing. Ask the model: What is 1234 + 5678?

The correct answer is 6912. Many models get this wrong or right only inconsistently — even models that now handle the strawberry question. The reason is related: the digits get split across token boundaries in ways that destroy the place-value structure arithmetic depends on. We will return to both of these failures later in the lecture, once we understand the algorithms behind tokenization.

To answer why the tokenizer splits text the way it does, we need to understand the design space. What are the options? What works, what fails, and what tradeoffs does every approach involve? That is what the rest of this lecture is about.

8. Word-Level Tokenization

The simplest approach is to give every word its own token. Split on spaces, assign each word a unique integer, and look it up in the embedding matrix.

The sentence “The cat sat on the mat” becomes six tokens, each with its own ID. Clean, intuitive, easy to implement.

And fatally flawed.

8.1 The Vocabulary Explosion

English has over 170,000 words in active use. But that is just the starting point. Add technical jargon — Kubernetes, microservices, backpropagation. Add proper nouns — every person’s name, every city, every product. Add typos — teh, recieve, definately. Add inflections — run, runs, running, ran, runner, runners. Add compound words, neologisms, slang.

The vocabulary balloons past half a million entries. And every entry needs its own row in the embedding matrix. With 768-dimensional embeddings, that is:

Nearly 400 million parameters just for the vocabulary lookup table. For a larger model with wider embeddings, the cost is even higher. This is an enormous amount of memory dedicated to a single component.

8.2 The Unknown Word Problem

But the cost is not even the worst problem. The real killer is what happens when the tokenizer encounters a word it has never seen.

Suppose someone types unbreakable. If this word is not in the vocabulary — because it was too rare in the training data, or simply was not included — the tokenizer has no entry for it. The word gets replaced with a special [UNK] token, meaning “I have no idea what this is.” All meaning is lost. The model knows that something was there, but it does not know what.

This is catastrophic for a language model. Imagine reading a book where every unfamiliar word has been blacked out. You might infer some meaning from context, but you have lost information that can never be recovered.

8.3 No Morphological Awareness

There is a third problem, more subtle but equally damaging. The words run, running, runner, and runs are four completely separate entries in a word-level vocabulary. The model gets no hint that they share a common root. Each one must be learned independently from scratch.

For a human, the connection is obvious — they all derive from “run,” and the suffixes (-ning, -ner, -s) carry predictable meanings. But in a word-level vocabulary, these relationships are invisible. The model must discover them purely from context, which requires far more training data than if the morphological structure were explicit.

Word-level tokenization was the standard in early natural language processing. For modern language models, it is essentially obsolete. The vocabulary is too large, unknown words destroy information, and morphological relationships are invisible.

We need a different granularity.

9. The Subword Sweet Spot

If word-level tokenization produces vocabularies that are too large and character-level tokenization produces sequences that are too long, is there a middle ground?

9.1 The Character Extreme

Let us first see why the other extreme — one token per character — also fails. If every character is its own token, the vocabulary is tiny: about 100 entries for the letters, digits, and common punctuation of English. You will never encounter an unknown word, because every possible word is just a sequence of characters.

But the cost is devastating. The word “cat” becomes 3 tokens. The word “unbreakable” becomes 11 tokens. A modest paragraph of 500 words explodes into roughly 2,500 character tokens.

Why is this a problem? Because transformer attention scales quadratically with sequence length. If you double the number of tokens in the sequence, the attention computation becomes four times more expensive:

A 2,500-token character-level sequence costs 6.25 million operations in the attention layer. A 500-token word-level sequence costs only 250,000. That is a 25-fold difference in computational cost — for the same text.

Character-level tokenization trades one problem (vocabulary explosion) for another (sequence explosion). Neither extreme works.

9.2 The Sweet Spot

The solution is subword tokenization: keep common words whole, but decompose rare words into meaningful pieces.

Watch what happens to the word “unbreakable.” Instead of one token (word-level) or eleven tokens (character-level), subword tokenization splits it into three: un, break, able.

These three pieces are exactly the right granularity. Each one is a reusable building block:

• un appears in unhappy, undo, unfair, uncertain, unbreakable
• break appears in breaking, breakage, breakfast, unbreakable
• able appears in readable, doable, comfortable, unbreakable

The model learns these pieces once and can compose them in new combinations. If the model has seen un and break and able separately, it can understand unbreakable even if it has never seen that specific word as a whole. This is similar to how you, as a human reader, can understand a word you have never seen before by recognizing its parts.

Common words like “the,” “is,” “function,” and “return” stay as single tokens — they are frequent enough to earn their own vocabulary entry. Rare words get decomposed. The result is a vocabulary of typically 30,000 to 200,000 tokens — large enough to keep common words intact, small enough to fit in memory.

This is the approach behind every major language model today — GPT, Claude, Llama, Mistral, Gemini, all of them use subword tokenization. The only question is: how does the machine learn which pieces to use?

9.3 The Two Subword Problems

At the character extreme, tokenization is trivial: every character is a token, and encoding any text means splitting it into individual characters. At the word extreme, it is equally simple: split on spaces, look up each word. In both cases, the encoding function — how to turn text into tokens — is obvious and requires no thought.

Subword tokenization is different. The moment you allow tokens of varying sizes, two problems arise that must be solved together:

1. What should be in the vocabulary? Which subwords do we pick? We need a set of tokens V that balances vocabulary size against sequence length. But how do we discover the right pieces from data?

2. How do we encode text into those pieces? Given a vocabulary that contains t, h, th, and the, how does the tokenizer decide whether to split “the” into t | h | e, th | e, or the? All three decompositions use only tokens that exist in the vocabulary. Without a clear rule, the splitting is ambiguous.

The second problem is easy to miss, because it did not exist at the extremes. But it is critical: a subword vocabulary that includes all individual characters could — with a naive encoding — simply decompose everything into characters, wasting every subword entry. The vocabulary alone is not enough. You also need an encoding function that knows how to split text intelligently.

Tokenization is therefore two things: a vocabulary V and an encoding function f that maps text to sequences of tokens from that vocabulary. Any subword tokenization algorithm must produce both. As we will see, BPE elegantly solves both problems simultaneously — the merge rules it learns during training serve double duty as the vocabulary and the encoding function.

10. Finding the Right Pieces

We have established that subword tokenization is the right idea — keep common words whole, decompose rare words into reusable parts. But this raises a question that is easy to overlook: who decides what the pieces are?

10.1 The Vocabulary Design Problem

Consider the word “unbreakable.” We said it should split into un | break | able. That seems obvious — but only because we are English speakers who recognize prefixes and suffixes. What about “Donaudampfschifffahrtskapitän” in German? Or “令和” in Japanese? Or a Python traceback that mixes English words, camelCase identifiers, and file paths?

There are three approaches we might try:

Ask linguists. Linguists can identify morphemes — the smallest meaningful units of language. For English, they would tell us that un- is a prefix and -able is a suffix. But this requires separate rules for every language. English morphology does not help with Mandarin, Arabic, or Tamil. And linguistics has nothing to say about calculateAverageScore or NullPointerException. This approach is too slow, too language-specific, and does not scale to multilingual models.

Ask engineers. We could hand-design rules: split on underscores, split after capital letters in camelCase, split on common suffixes. This works for specific cases but is inherently arbitrary. Why split after “break” but not after “bre”? Any rule we write reflects our assumptions, not the actual structure of the training data. Different corpora have different patterns, and a rule-based system cannot adapt.

Let the data decide. Instead of imposing rules from above, we can start from raw characters and let frequency statistics reveal which character combinations deserve to become tokens. If t and h appear next to each other thousands of times, merge them into th. If th and e appear together constantly, merge them into the. The data tells us what the useful pieces are — no linguistic expertise required, no language-specific rules, no manual curation.

10.2 From Compression to Tokenization

This “let the data decide” approach is not new. In 1994, Philip Gage published a simple compression algorithm called Byte Pair Encoding (BPE). The idea was straightforward: find the most frequent pair of adjacent bytes in a file, replace them with a single new byte, and repeat. Each merge makes the file shorter. It was a niche compression technique — nothing to do with language.

Twenty-one years later, in 2015, Rico Sennrich and colleagues realized that this same algorithm could solve the vocabulary design problem for neural machine translation. Instead of compressing files, they applied it to text: start with characters, find the most frequent adjacent pair, merge, repeat. The most common character sequences naturally become tokens.

By 2019, byte-level BPE had become the standard. Today, GPT-4, Claude, Llama, and Mistral all use BPE or close variants.

The beauty of the algorithm is its simplicity. One rule, applied repeatedly: find the most frequent pair, merge it. No linguistic knowledge, no dictionary, no grammar rules. The frequency statistics do all the work.

11. Byte Pair Encoding

Let us see how BPE works in practice.

11.1 How BPE Works

The algorithm starts with a training corpus — a large body of text that represents the language the model will process. Let us walk through a toy example.

Our training corpus: “the cat in the hat”

Step 0: Initialize. Split every word into individual characters. This gives us the smallest possible starting vocabulary:

t  h  e     c  a  t     i  n     t  h  e     h  a  t

The starting vocabulary is just 7 unique characters: {t, h, e, c, a, i, n}.

Step 1: Count pairs and merge the most frequent. Look at every adjacent pair of characters in the corpus and count how many times each pair appears:

• (t, h) appears 3 times (in the, the, hat)
• (h, e) appears 2 times (in the, the)
• (a, t) appears 2 times (in cat, hat)

The pair (t, h) wins with the highest count. But what if there had been a tie? In practice, the implementation picks one deterministically — for example, whichever pair comes first in the scan order. A different tiebreaker would produce a slightly different vocabulary. On our toy corpus, ties are plausible. On real training data with billions of words, the counts are so large that ties are extremely rare.

We merge t and h. Every occurrence of t h in the corpus is replaced by the new token th:

th  e     c  a  t     i  n     th  e     h  a  t

The vocabulary grows from 7 to 8: {t, h, e, c, a, i, n, th}.

Step 2: Repeat. Recount all adjacent pairs. Now (a, t) appears twice (in cat and hat). Merge a and t into at:

th  e     c  at     i  n     th  e     h  at

Vocabulary: 9 tokens.

Step 3: Repeat again. Recount. (th, e) appears twice. Merge them — and just like that, the, one of the most common English words, has become a single token:

the     c  at     i  n     the     h  at

Vocabulary: 10 tokens.

Three merges, and the algorithm has already discovered a whole English word — purely from counting which pairs appear most often. No grammar rules. No dictionary lookup. Just frequency statistics.

11.2 From Toy Example to Real Model

In practice, the scale is entirely different. GPT-2 performs 50,000 of these merges. GPT-4 performs roughly 100,000. The training corpus is not five words — it is hundreds of gigabytes of text. But the algorithm is the same: start from the smallest units, count pairs, merge the most frequent, repeat.

Each merge compresses the text a little more, building up a vocabulary of increasingly large and meaningful pieces. Early merges create common character combinations (th, in, er). Middle merges create common subwords and short words (the, ing, tion). Late merges create common full words and phrases (function, return, import).

The key insight is that BPE is entirely data-driven. It requires no linguistic rules, no dictionary, and no knowledge of any particular language. It simply looks at what pairs occur most often and merges them. The structure of language — common words, frequent suffixes, recurring prefixes — emerges automatically from the statistics of the training corpus.

11.3 BPE Solves Both Problems

Remember the two subword problems from Section 9.3: what should be in the vocabulary? and how do we encode text into those pieces? BPE solves both simultaneously.

Training produces two outputs:

1. A vocabulary — the set of all tokens: the 256 base bytes (or characters) plus every merged token. After our three merges: V = {t, h, e, c, a, i, n, th, at, the} — 10 tokens.
2. An ordered list of merge rules — the merges in the exact sequence they were learned:
   • Rule #1: t + h → th
   • Rule #2: a + t → at
   • Rule #3: th + e → the

These merge rules are not just a record of how the vocabulary was built. They are the encoding function. When the trained tokenizer encounters new text, it does not re-run the training algorithm. It replays the merge rules in order on the new input.

11.4 Encoding New Text

Let us see this in action. Our trained tokenizer has vocabulary V = {t, h, e, c, a, i, n, th, at, the} and three merge rules. Now a new sentence arrives that was never part of the training corpus:

Encode: “the hat”

Start: Split into individual characters (always the starting point for encoding):

t  h  e     h  a  t

Apply Rule #1 (t + h → th): Scan for any adjacent pair t h. Found at position 1-2. Merge:

th  e     h  a  t

Notice: the h in “hat” is preceded by a space, not by t, so it does not match. Only the t h at the start merges.

Apply Rule #2 (a + t → at): Scan for a t. Found in “hat”. Merge:

th  e     h  at

Apply Rule #3 (th + e → the): Scan for th e. Found at the start. Merge:

the     h  at

No more rules. The final tokenization is: the | h | at — three tokens.

This example reveals something important. Our vocabulary contains all the original characters — t, h, e, a — and the merged tokens th, at, the. The tokenizer could have split “the” into three character tokens t | h | e and the result would still use only valid vocabulary entries. But it did not, because the merge rules enforce a specific decomposition. The merge rules are the encoding function. Without them, you have a bag of allowed strings and no way to deterministically split new text.

This also answers a natural question: is the tokenizer doing “longest match” — looking for the biggest token in the vocabulary that fits? No. It is replaying merges in the order they were learned. Rule #1 fires first because it was the most frequent pair during training. The merge order implicitly means that more frequent (and thus more useful) merges happen before less frequent ones. The result often looks like longest match, but the mechanism is fundamentally different — it is an ordered sequence of rewrite rules.

11.5 When to Stop

In our toy example, we stopped after 3 merges because the remaining pairs all have a count of 1 — there is no strong statistical signal left. But the algorithm does not decide when to stop on its own. In practice, the target vocabulary size is a hyperparameter chosen before training begins:

• GPT-2: 50,257 tokens (256 base bytes + 50,000 merges + 1 special token)
• GPT-4: approximately 100,000 tokens
• LLaMA 3: 128,000 tokens

The algorithm runs merges until the vocabulary reaches the target size, then stops. On billions of words of training data, there is always enough frequency signal to support tens of thousands of merges. The choice of vocabulary size is an engineering tradeoff — we will explore its consequences in Chapter 16.

You will experience this directly when you implement the encoder in the BPE notebook exercise.

12. Strawberry Explained

Now you understand subword tokenization and BPE. Let us return to the puzzle from Chapter 7.

Why does strawberry split into str, aw, berry?

Because the BPE algorithm, trained on billions of words, learned that these are efficient subword units. The pair (b, e) was frequent enough to merge into be. Later, (be, r) merged into ber. Then (ber, r) into berr. Then (berr, y) into berry. The same process built str from individual characters. Each merge was driven by frequency statistics — not by any understanding of spelling, morphology, or the English language.

The merge statistics determined these boundaries. And those boundaries have consequences.

12.1 Tokens Are Atomic

The model cannot reach inside a token to examine individual characters. Each token is an atomic, indivisible unit — the smallest piece the model can perceive. When the transformer processes str, aw, berry, it sees three opaque chunks, not ten letters.

Where are the r’s?

• str contains one r, buried at the end of a three-character chunk.
• aw contains no r.
• berry contains two r’s, buried inside a five-character chunk.

The r’s are scattered across tokens, hidden inside larger pieces. The model has no mechanism to “look inside” a token and count its characters. It can only reason about the tokens it sees — and it sees three tokens, not ten letters.

12.2 Perception First — But Not the Whole Story

Tokenization destroys the character-level information before the neural network ever gets involved. If the model cannot see individual letters, it cannot count them. That is a genuine failure of perception. Think of it this way: if someone showed you a sentence where every third word was blacked out, you would struggle to answer questions about those words — not because you cannot read, but because the information was removed before it reached you.

But here an honest student should push back: what if we fixed the tokenizer? Suppose we tokenized “strawberry” letter by letter — s, t, r, a, w, b, e, r, r, y — ten tokens, every r visible. Would the model now correctly answer “three”?

Probably not reliably. And the reason reveals a deeper truth about how these models work.

A next-token prediction machine does not execute procedures. It does not have variables, loops, or accumulators. To count the r’s, the model would need to:

1. Scan every token and identify which ones are r
2. Maintain a running count that increments with each match
3. Output the final number

A transformer does none of these things. It takes the input tokens, passes them through attention and feed-forward layers, and produces a probability distribution over the next token. The output “3” would have to emerge as the statistically most likely continuation of “How many r’s are in strawberry? → “, based on patterns the model saw during training. If the training data contained thousands of examples of “How many r’s in strawberry → 3”, the model would learn it as a memorized pattern — not derive it through a counting procedure.

This is exactly why recent frontier models now handle the strawberry question correctly: they have been trained on this specific example and many similar letter-counting tasks. Pattern memorization, not improved counting ability. Try a less famous word — German “Donaudampfschifffahrtskapitän”, or a made-up one — and the failure often returns.

So the lesson has two layers:

1. Tokenization makes letter-counting impossible — you cannot count what you cannot see.
2. Even with perfect tokenization, next-token prediction is the wrong architecture for procedural counting. Counting requires a mechanism the transformer does not have.

The same two-layer explanation applies to the arithmetic failure from Chapter 7 — 1234 + 5678. Tokenization destroys place-value alignment (we study this in Chapter 17), but even with digit-level tokenization, arithmetic is a procedure that next-token prediction does not naturally perform. It can only produce 6912 if the pattern is strong enough in training data to emerge statistically.

The deeper insight: the transformer can carry information across tokens — that is what attention is for. We open that box in Lecture 5. Only then will we understand both why some of these failures can be partially overcome through training, and why they remain fundamentally fragile.

13. Byte-Level BPE: How It’s Really Done

The BPE we built in Chapter 11 operated on characters — a friendly way to introduce the algorithm, but not how real tokenizers work. GPT-2, GPT-4, Claude, Llama, and Mistral all operate on bytes, not characters. This chapter explains why, how, and at what scale — and what it means to stand on infrastructure that was built decades before transformers existed.

13.1 The Unknown-Character Problem

Standard character-level BPE is already a big improvement over word-level tokenization. But “character” is a more slippery concept than it first appears.

English has just 26 letters. But Chinese has tens of thousands of characters. Add emoji (3,600 and counting), mathematical symbols (over 2,000), currency symbols, diacritics from dozens of scripts, and rare Unicode characters, and the character-level base vocabulary — before any merges even begin — can exceed 100,000 entries.

This defeats the purpose of subword tokenization. We wanted a small vocabulary. Starting from 100,000 characters is not small. And we still have the UNK problem lurking: a new emoji gets added to Unicode every year, and a tokenizer trained last week has no way to encode it.

13.2 Standing on the Shoulders of Unicode and UTF-8

Before we can explain GPT-2’s answer, we need to appreciate the infrastructure it inherited.

LLMs did not invent text-as-numbers. Decades before the first transformer, the computing world had already solved the hardest part:

• Unicode (first published 1991) assigned every character in every script — Latin, Cyrillic, Chinese, Arabic, emoji, mathematical symbols — a unique integer called a code point. About 150,000 code points are assigned today, and the standard reserves space for over a million.
• UTF-8 (1993, RFC 3629) encoded these code points as byte sequences of length 1 to 4. It is backwards-compatible with ASCII (pure English text uses one byte per character) and has become the de-facto encoding of the internet — over 98% of web pages today.

13.2.1 What is a byte, actually?

A quick detour for readers who haven’t worked at this level before. A byte is simply 8 bits — 8 binary digits, each 0 or 1. Eight bits give you \(2^8 = 256\) distinct combinations, numbered 0 to 255. That is all a byte is: a number between 0 and 255.

The same byte can be written in four notations that look very different but mean exactly the same thing:

A byte on its own has no inherent meaning — context decides how it is interpreted. The byte 72 is the number 72 when you do arithmetic, the letter H when you display it as ASCII text, and a gray pixel shade when you render it as an image. The Python call list("Hello".encode("utf-8")) returns [72, 101, 108, 108, 111] — these are the bytes as integers, which is how BPE code usually works with them. Hex notation is popular in specs because two hex digits map to exactly one byte.

13.2.2 How four bytes cover more than 150,000 characters

If one byte has only 256 possible values, how can UTF-8 possibly encode Unicode’s ~150,000 assigned code points — let alone leave room for the full reserved space of over a million? The answer is the clever trick at the heart of UTF-8: the first byte of a character announces how many bytes that character occupies.

The leading bits of the first byte are self-describing:

• 0… — standalone 1-byte character (pure ASCII)
• 110… — I am the start of a 2-byte character
• 1110… — I am the start of a 3-byte character
• 11110… — I am the start of a 4-byte character
• 10… — I am a continuation byte; I never start a character

With four bytes you get 21 payload bits — over 2 million possible values, which more than covers Unicode’s reserved range. ASCII stays cheap (one byte per English letter), non-ASCII text pays more only where it has to. This is why UTF-8 won the encoding wars of the late 1990s: efficiency and ASCII-compatibility in one design.

13.2.3 UTF-8 in practice

Putting that into concrete bytes:

• The letter H → 1 byte: 0x48 = 72
• The German umlaut ü → 2 bytes: 0xC3 0xBC = 195, 188
• The Chinese character 世 (“world”) → 3 bytes: 0xE4 0xB8 0x96 = 228, 184, 150
• The emoji 😊 → 4 bytes: 0xF0 0x9F 0x98 0x8A = 240, 159, 152, 138

Notice how the first byte of the multi-byte characters (0xC3, 0xE4, 0xF0) matches the headers from the bit-pattern table: 11000011…, 11100100…, 11110000…. UTF-8 is not magic — it’s disciplined bit layout.

This asymmetry — non-English text is already more bytes before any tokenization happens — is the root cause of the multilingual fairness gap we discuss in Chapter 18. It is also exactly why the byte-level approach works: Unicode and UTF-8 guarantee that every possible text maps to a sequence of bytes, where each byte is just a number between 0 and 255.

Key insight: LLMs do not have to invent a universal alphabet. They inherit one. Tokenizer design starts after text has already been reduced to bytes — a reduction that 30 years of standards work made possible.

13.2.4 Inspect the bytes yourself in Python

Everything in §13.2.1–§13.2.3 is inspectable with a few lines of Python. No libraries beyond the standard library — every tool you need is built into the language.

The b'...' notation you will see. When Python prints a bytes object, it uses a mixed notation: printable ASCII bytes appear as their ASCII letter, and non-ASCII bytes appear as hex escapes \xNN:

>>> "Hü".encode("utf-8")
b'H\xc3\xbc'              # byte 72 shown as 'H'; bytes 195, 188 shown as hex
>>> list("Hü".encode("utf-8"))
[72, 195, 188]            # same three bytes, as integers
>>> bytes([72, 195, 188]) == b'H\xc3\xbc'
True

b'H\xc3\xbc' and [72, 195, 188] are two displays of the same three bytes. Tokenizer code works with the integer form because that is what get_stats and merge operate on (§13.4).

Getting the bit patterns. Python’s format spec :08b means “binary, zero-padded to 8 digits” — exactly one byte’s worth of bits:

>>> for b in "ü".encode("utf-8"):
...     print(f"byte {b:3d}  (0x{b:02X})  =  {b:08b}")
byte 195  (0xC3)  =  11000011
byte 188  (0xBC)  =  10111100

Now you can see the §13.2.2 bit patterns literally. The first byte starts with 110… (header for a 2-byte character); the second starts with 10… (continuation). Peel off the headers, glue the payload bits together, and you recover the Unicode code point:

First byte:  11000011  →  110 | 00011     (header | 5 payload bits)
Second byte: 10111100  →  10  | 111100    (header | 6 payload bits)

Combined payload: 00011 111100  =  binary 00011111100  =  decimal 252
                                  =  U+00FC  =  "ü"  ✓

A reusable one-liner. This small function works for any character and makes the structure visible at a glance:

def show_utf8(char):
    bs = list(char.encode("utf-8"))
    print(f"{char!r}  (U+{ord(char):04X} = {ord(char)}, {len(bs)} byte(s))")
    for i, b in enumerate(bs):
        print(f"  byte {i}: {b:3d}  0x{b:02X}  {b:08b}")

show_utf8("H")    # 1-byte:  0xxxxxxx
show_utf8("ü")    # 2-byte:  110xxxxx 10xxxxxx
show_utf8("世")   # 3-byte:  1110xxxx 10xxxxxx 10xxxxxx
show_utf8("😊")    # 4-byte:  11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

Run it and compare the leading bits against the bit-pattern table in §13.2.2 — they match exactly. Every UTF-8 claim in this chapter is one f"{b:08b}" away from being verified.

13.3 The Universal 256-Byte Base Vocabulary

Once text is bytes, the base vocabulary becomes trivial. There are exactly 256 possible byte values (0 through 255), and that is GPT-2’s entire starting point:

That’s it. 256 tokens cover every written human language, every emoji, every scientific notation — because UTF-8 already did the hard work of reducing them to bytes. BPE then runs on top, learning merges that combine raw bytes into increasingly meaningful multi-byte tokens.

Contrast with character-level BPE: ~100,000 base entries, still unable to handle next year’s new emoji. Byte-level BPE: 256 base entries, no UNK token, forever.

A guarantee, not a suggestion. Because the 256 base tokens exist before any training runs, every possible UTF-8 byte sequence can always be encoded — in the worst case, one token per byte. The [UNK] token that haunted word-level systems (§8.2) cannot arise. We will see in §13.5 that this has one subtle consequence: rare characters are always representable even if the tokenizer never learned a good merge for them; they just end up using more tokens.

13.3.1 Why a new emoji does not require retraining

The guarantee above is abstract. Let’s make it concrete with a scenario that plays out every year: Unicode adds new emoji. How can a tokenizer shipped years ago still handle characters that did not exist when it was built?

The setup. It is 2027. GPT-4o shipped in 2024 with o200k_base — its vocabulary is frozen. Today, Unicode 17.0 is released, adding a brand-new emoji we’ll call 🆕-emoji. Its UTF-8 encoding is four bytes — for illustration, let’s say F0 9F AA 99, or as decimal integers, [240, 159, 170, 153]. This specific byte sequence has never appeared in any tokenizer’s training data.

Step 1 — Encoding. A user types the new emoji into a prompt. The OS encodes it as UTF-8. The tokenizer’s encode() function receives the four bytes and applies its learned merge rules in training order (§13.4). It finds no merge involving this specific byte sequence — it could not have, since the bytes did not co-occur in the training corpus. So the four integers pass through as four base-byte tokens. The emoji is represented as 4 tokens instead of the 1 token a freshly retrained tokenizer might assign it.

No error. No [UNK]. No code change. No retraining.

Step 2 — The LLM processes the tokens. The model’s embedding matrix has a row for every token ID from 0 to ~200,000. Rows 240, 159, 170, and 153 were there from day one — they are part of the universal 256-byte base vocabulary (§13.3), existing unconditionally. The model looks up those four embedding rows, feeds them into the transformer stack, and processes them exactly like any other input.

The model has never seen this emoji in training. It has no learned semantic association with the new symbol. But the pipeline works end-to-end: encoding succeeds, the forward pass succeeds, and eventually, decoding succeeds.

Step 3 — The model generates output. Can the model produce the emoji? Yes — by emitting the four base-byte tokens in the correct order. The embedding rows exist, the output softmax covers all 200,000 token IDs, so token 240 followed by 159 followed by 170 followed by 153 is a perfectly valid generation. Concatenate the bytes, UTF-8-decode, and the emoji reappears on the user’s screen.

Will it produce the emoji in the right context? That depends on in-context learning. Show the model one example (“This emoji 🆕-emoji means ‘sparkle’”), and it will often echo the bytes back correctly when similar context recurs. Without any context, the model cannot invent a meaning — but it can always reproduce the emoji when asked.

Contrast with older tokenizer designs:

The takeaway. UTF-8 is a forward-compatible standard: every character Unicode will ever add is just a new code point encoded as 1–4 bytes — and those bytes are already in the tokenizer’s base vocabulary. A byte-level tokenizer trained today will correctly encode and decode every character Unicode adds for the rest of time, without retraining.

The model’s semantic understanding of new symbols grows only with new training data. But the technical capability to process them is built in from day one. This is what “standing on the shoulders of Unicode and UTF-8” (§13.2) actually buys you: a tokenizer that does not age out of its input domain.

13.4 The Algorithm, at Byte Level

Conceptually, nothing changes from the character-level BPE you met in Chapter 11. Concretely, the “pairs” are now pairs of byte-level tokens, and the “new tokens” are multi-byte sequences. A working byte-level tokenizer has five functions, in this order:

1. get_stats(ids) — count the frequency of every adjacent token pair in the sequence.
2. merge(ids, pair, new_id) — replace every occurrence of pair with new_id.
3. train_bpe(text, num_merges) — repeat get_stats + merge for num_merges iterations, recording the merge rules in the order they were learned.
4. encode(text, merges) — apply the learned merge rules, in training order, to tokenize new text.
5. decode(ids, vocab) — look up each token ID’s byte sequence, concatenate, UTF-8-decode.

Merge order matters. Later merges may depend on tokens created by earlier merges. If merge 1 produces th (token 256) from the bytes t + h, and merge 2 produces the (token 257) from (256, e), then to tokenize the word “the” you must apply merge 1 first — otherwise merge 2 has nothing to find. The self-study notebook linked in §13.8 makes this concrete: implement these five functions yourself, and watch what happens when you apply merges out of order.

13.4.1 A worked example: “Müller”

Abstract function names are one thing; seeing the actual integers BPE operates on is another. Take the tiny corpus "Müller Müller Müller" — one German surname, repeated three times. Encoded as UTF-8, using the byte values from §13.2.3 (M=77, ü=195,188, l=108, e=101, r=114, space =32), the full byte sequence is:

[77, 195, 188, 108, 108, 101, 114, 32,
 77, 195, 188, 108, 108, 101, 114, 32,
 77, 195, 188, 108, 108, 101, 114]

23 integers, each between 0 and 255. The critical insight: BPE does not know these are bytes. As far as get_stats is concerned, this is just a list of integer token IDs. It counts adjacent pairs and produces:

Several pairs tie at 3. Suppose BPE picks (195, 188) and assigns it the new token ID 256. Applying merge replaces every occurrence of that pair, producing:

[77, 256, 108, 108, 101, 114, 32, 77, 256, 108, 108, 101, 114, 32, 77, 256, 108, 108, 101, 114]

The vocabulary entry for token 256 stores the bytes b'\xc3\xbc' — which, when UTF-8-decoded, is exactly "ü". BPE has “discovered” the umlaut without being told anything about Unicode. It merely noticed that bytes 195 and 188 always appeared adjacent in this corpus and collapsed them into a single unit.

Why this works universally. In valid UTF-8, the two bytes of ü (195 and 188) always appear adjacent, in that exact order — the encoding standard guarantees it. So the pair (195, 188) is guaranteed to be highly frequent in any text that contains ü. BPE merges it early. More generally, every multi-byte character in UTF-8 looks to BPE like a short chain of highly-correlated bytes that co-occur 100% of the time. The variable-length nature of UTF-8 isn’t a complication BPE has to handle — it’s a feature BPE exploits automatically.

13.4.2 Two representations, two files

The Müller example left one question implicit: once BPE produces token 256, what does the tokenizer actually store for it? There are two plausible designs — one obvious, one that turns out to be the right engineering choice — and real tokenizers ship with both, in separate files.

Option A: store the pair. A natural design would be to remember “token 256 was built by merging (195, 188)”:

vocab_recursive = {
    256: (195, 188),        # the pair of ids that merged into 256
    257: (256, 101),        # 256 is itself merged — recursive
    258: (116, 257),
    # ...
}

To decode, recursively expand until you reach base bytes:

def expand(token_id):
    if token_id < 256:
        return bytes([token_id])       # base case: a raw byte
    a, b = vocab_recursive[token_id]
    return expand(a) + expand(b)       # concatenate children

This works. It is logically equivalent to what production tokenizers do.

Option B: store the flat bytes. This is what §13.4.1’s worked example did and what every production tokenizer actually ships:

vocab = {
    256: b'\xc3\xbc',       # bytes, already concatenated
    257: b'\xc3\xbce',      # eagerly built from vocab[256] + vocab[101]
    258: b't\xc3\xbce',
    # ...
}

Decoding is then a flat lookup, no tree walk:

pieces = [vocab[i] for i in [77, 257]]        # [b'M', b'\xc3\xbce']
b"".join(pieces).decode("utf-8")               # → "Müe"

Why Option B wins. Three reasons, in order of importance:

1. Decode speed. Flat is O(1) per token; recursive is O(depth) lookups plus concatenations. At inference, every generation step decodes a token, so this path runs constantly.
2. You already computed it during training. The training line vocab[new_id] = vocab[a] + vocab[b] eagerly concatenates the bytes every time a merge fires (see §13.4.1). Throwing that result away and recomputing later would waste work already done.
3. Memory is trivially cheap at this scale. A 200,000-token vocab with an average of ~10 bytes per entry is ~2 MB — not a concern.

The twist: you actually have both. Option A’s information is not discarded. It lives in a second file that ships alongside the flat vocab. Every trained tokenizer (GPT-2, GPT-4o, Llama, StarCoder2) is two files:

Both come from the same training run. They expose different views of the same merge tree, because the two operations need different things:

• Encoding needs order and priority — “which merge fires first on this byte sequence?” → merges.txt, where rank 0 beats rank 1 beats rank 2.
• Decoding needs only the final byte sequence for each token ID → vocab.json, the precomputed flat answer.

Look at GPT-2’s merges.txt and vocab.json — it is literally these two files.

The design pattern. This is a classic engineering move: precompute at training time (happens once) what speeds up inference time (happens constantly). The merge tree is the training artifact; the flat vocab is the precomputed inference artifact. Both together is the tokenizer — encoding requires merges.txt, decoding requires vocab.json, and the model at the other end of the pipeline needs both.

13.4.3 Where meaning lives — and where it doesn’t

One question worth making explicit, because it trips almost everyone up: does token 256 “know” it represents ü?

No. Token 256 is literally just the integer 256. It has no metadata, no label, no memory of the bytes that went into making it. BPE does not know what ü is. It does not know what any character is. It shuffles integers.

The apparent “ü-ness” of token 256 is actually a chain of four completely separate layers, each defined by a different piece of infrastructure:

All four layers must line up to put ü on your screen. BPE is only the top two rows. Everything below is handled by standards that long predate LLMs. Token 256 could just as easily have meant b'th', or the first two bytes of an emoji, or any other byte pair. The integer itself tells you nothing — the vocab is the decoder ring, and only then does UTF-8 turn bytes into characters.

About \xc3. A quick clarification that is easy to get wrong: \xc3 alone is not “a piece of ü.” It is just a byte. It becomes the first half of ü only when paired with \xbc and interpreted as UTF-8. A single byte outside the ASCII range has no character meaning — it is an incomplete UTF-8 sequence. (See §13.2.4 for the b'...' notation, and §13.2.2 for why the leading bits of \xc3 = 11000011 announce “I am the start of a 2-byte character.”)

Encoding and decoding are the boundary of the system. Pulling this together with §13.4.2:

• Encoding is the entrance: text → UTF-8 bytes → token IDs (so the model can ingest it).
• Decoding is the exit: token IDs → vocab bytes → UTF-8 text (so humans or external systems can read it).
• In between: pure integer math. The tokenizer sees integers. The model also sees only integers — it runs embedding lookups, attention, MLPs, and output softmax entirely in integer-ID space from input to output. It never calls .decode("utf-8") at any point.

The “meaning” of a token is layered in only when you cross the boundary back to the human-facing world. Inside the pipeline, everything — BPE, the model, the inference engine — is agnostic about what the integers represent. That agnosticism is the whole reason byte-level BPE is so robust: the pipeline has no semantic commitments to defend, so it cannot “misunderstand” anything. It never tried to understand in the first place.

13.4.4 So what, really, is a token?

After all that machinery, we can finally answer the deceptively simple question this chapter opened with:

A token is an integer with meaning assigned by its vocabulary.

That single sentence is surprisingly precise. Unpack it:

• It is an integer. Operationally, what tokenizer code passes around is an integer ID. tokenizer.encode("Hello") returns [13225, ...] — a plain list of integers, not a list of (id, bytes) tuples. The vocab is a shared reference table; each token points to its vocab entry implicitly, not explicitly.
• The meaning is assigned by the vocabulary. The integer 3788 is not inherently “ software” — it is “ software” only because vocab[3788] = b' software'. Strip the vocab away and 3788 is just a number, no more meaningful than 42.
• …by its vocabulary. Tokens are only tokens relative to a specific vocabulary. The same text encoded by GPT-2 and Llama 3 produces completely different integer sequences, because the vocabularies are different. “Token 3788” in cl100k_base means something totally unrelated to “token 3788” in o200k_base.

The word “token” is also used at three different levels depending on context, all consistent with this definition:

All three are “the token” — different representations of the same logical unit, each appropriate to its level. In code, it’s the integer. On disk, it’s the vocab entry. Inside the transformer, it’s the learned embedding vector. The integer threads them all together.

A useful analogy: a house key isn’t “the metal shape + the description ‘front door’.” The key is the metal; the description lives in your key manager. Both together tell you what door it opens, but you carry only the key. The same logic applies here — tokens travel as integers, the vocab is the key manager.

13.5 Lossless vs. Efficient: Two Different Worries

The Müller example raises a fair worry: what happens if the data is unbalanced? Imagine a huge English-dominant corpus where exactly one German word with ü appears. The pair (195, 188) occurs once in billions of byte pairs, so it never wins the frequency contest against English pairs like (116, 104) (“th”, appearing millions of times). BPE never merges the umlaut.

Has ü been “lost”? The answer depends on which of two very different questions you are asking.

Worry 1: Can ü become unrepresentable?

No. Never. This is exactly what the 256-byte base vocabulary guarantees (§13.3). Tokens 195 and 188 exist unconditionally — they are in the vocabulary before training even starts. So "Müller" still encodes fine:

"Müller" → [77, 195, 188, 108, 108, 101, 114]
           M   (ü in 2 bytes)   l    l    e   r

Seven tokens instead of the three or four tokens a German-trained tokenizer might produce — but no information is lost. Decode that sequence, concatenate the bytes via b"".join(...), call .decode("utf-8"), and you get back "Müller" perfectly. Byte-level BPE is encoding-lossless, always.

Worry 2: Can ü become expensive?

Absolutely yes, and this is the real concern. When (195, 188) is rare in training, BPE never merges it. German text then costs more tokens per word than comparable English text. The same word “Müller” might cost 7 tokens, while the English “Miller” — which benefits from learned merges for Mi, ll, er — might cost just 2 or 3. A 3× price difference for equivalent content.

Same bytes, different groupings. This is the cleanest framing to carry forward. The tokenizer always stores UTF-8 bytes in its vocabulary — it couldn’t deviate from UTF-8 if it wanted to. What varies is how those bytes get grouped into tokens:

Both roundtrip to "ü" without error. The difference is purely one of efficiency, not correctness.

Why this distinction matters. The lossless vs. efficient split is the single most important idea in this chapter. Byte-level BPE guarantees losslessness by construction; efficiency depends entirely on the training-data mixture. This is why §13.7 (Where the Data Comes From) treats data curation as a first-order engineering problem, and it is the mechanism behind the multilingual fairness gap in Chapter 18. The algorithm treats every byte equally; the merges it learns reflect only the data it was fed.

13.6 Scale Reality: From Toy to Production

Our worked example in Chapter 11 used 20 merges on a 95-byte corpus. GPT-4o uses ~200,000 merges trained on multi-terabyte web data. The algorithm is identical; the scale is six orders of magnitude apart.

Three consequences flow from that scale:

Embedding-matrix cost. The tokenizer’s vocabulary size becomes the number of rows in the model’s input embedding matrix. For a model with 12,288-dim embeddings and 200,019 vocab entries, stored in 16-bit precision, the embedding table alone is roughly \(200{,}019 \times 12{,}288 \times 2 \approx 4.9\) GB — before the transformer itself uses a single byte. This is why vocabulary size is a first-order design choice, covered in more depth in Chapter 16.

Compression ratio improves, with diminishing returns. Going from 20 merges to 50,000 roughly halves the token count for English text. Going from 50,000 to 200,000 saves another ~10–15%. Going from 200,000 to 2 million would save almost nothing — the curve is flat by that point. The self-study notebook (§13.7) lets you observe this curve directly.

Vocabulary reflects training data. The merges a tokenizer learns are whatever patterns repeat most often in its training corpus. If the corpus is 90% English, the merges will be English-biased; German, Japanese, and Arabic text will cost more tokens per word. This is not a bug — it is the predictable consequence of the training-data mixture, and it is the root cause of the multilingual fairness gap explored in Chapter 18.

13.7 Where the Data Comes From

“Trained on the internet” hides a very involved pipeline. Understanding it is essential to understanding why real tokenizers look the way they do.

The raw starting material is typically Common Crawl, a public web scrape that produces tens of petabytes of HTML every month. Nobody trains on Common Crawl directly. The standard pipeline looks like this:

1. Extraction. Strip HTML, boilerplate navigation, ads, scripts. Keep the main article text.
2. Language detection. Classify each document by language (fastText, langdetect, CLD3). Most labs enforce a curated language mix rather than keeping whatever the crawl happens to sample.
3. Quality filtering. Drop documents with low-quality signals — too many repeated lines, unrealistic punctuation density, known spam patterns. Some labs train small classifiers to score “Wikipedia-like” quality.
4. Deduplication. Remove exact and near-duplicates (MinHash, SimHash). Duplicates are a huge problem: they bias merges toward repeated boilerplate and cause models to memorize rather than generalize.
5. Mixture construction. Combine filtered Common Crawl with curated sources: books, Wikipedia, forum archives, code repositories, academic papers. The mix — how much code, how many languages, how much math — shapes every merge the tokenizer learns.

The key insight for tokenization: the merges you get are a fingerprint of your data pipeline, not a property of the algorithm. Two teams running identical BPE code on different data produce visibly different tokenizers. The notebook in §13.8 makes this concrete: it trains BPE separately on English, German, code, and emoji sub-corpora and lets you see — same algorithm, four very different vocabularies.

13.7.1 What is actually documented — and what you can measure yourself

The pipeline above is general. What do we actually know about the specific pipelines that produced today’s frontier tokenizers? The short answer: less than you might hope.

OpenAI has deliberately stopped disclosing. The GPT-4 technical report (Brown et al., 2023, arXiv:2303.08774) states outright:

Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.

Earlier generations were more transparent. The GPT-3 paper (Brown et al., 2020, arXiv:2005.14165) published the model’s training mix with weights — roughly 60% filtered Common Crawl, 22% WebText2, 8% Books1, 8% Books2, 3% Wikipedia — though the tokenizer itself was inherited from GPT-2 essentially unchanged. For cl100k_base (GPT-3.5, GPT-4) and o200k_base (GPT-4o), OpenAI open-sources the tokenizer via tiktoken but does not disclose the data distribution that produced it.

What empirical research has measured despite that. Two widely-cited 2023 papers quantify the multilingual fairness gap directly:

• Petrov et al. (2023), “Language Model Tokenizers Introduce Unfairness Between Languages” (arXiv:2305.15425, NeurIPS 2023; interactive project page) measured tokenization length for the same text across many languages on GPT-2/3/4 and other tokenizers. The headline finding: some language pairs differ by up to 15× in token count for equivalent content. Even tokenizers explicitly trained to be multilingual show substantial gaps.
• Ahia et al. (2023), “Do All Languages Cost the Same?” (arXiv:2305.13707, EMNLP 2023) framed the same effect as an economic one: across 22 typologically diverse languages, speakers of many languages pay more per API call and receive worse output — often from regions where the API is already less affordable.

Common Crawl as a sanity check on raw inputs. Since most frontier tokenizers are trained on filtered Common Crawl plus curated supplements, knowing what raw Common Crawl looks like sets expectations. The project’s public language statistics show roughly 45% English by document count, with Russian, German, Japanese, and French each at a few percent, and a very long tail of 160+ other languages at fractional percentages. A tokenizer trained without explicit rebalancing will inherit exactly that skew.

Code-focused models take different paths. The picture for code is mixed:

• Code Llama (Meta) — reused Llama 2’s SentencePiece BPE tokenizer without retraining; added a handful of code-specific special tokens (infilling markers). The team decided the base vocabulary was adequate for code.
• StarCoder2 + The Stack v2 (Lozhkov et al., 2024, arXiv:2402.19173) — trained a custom tokenizer on The Stack v2: 67.5 TB of code across 619 programming languages, ~900 B training tokens, 10× larger than The Stack v1. The resulting vocabulary has visibly more indentation tokens, bracket patterns, and language-specific keywords.
• Codex (the original Copilot model) — reused GPT-3’s general-purpose tokenizer unchanged.

The pattern: specialist models retrain the tokenizer only when the base vocabulary clearly fails them. Most inherit from a general-purpose tokenizer.

You don’t have to take anyone’s word for it. The Tokenizer Playground notebook in §13.8 loads cl100k_base and o200k_base directly via tiktoken and lets you decode every one of the ~200,000 tokens in GPT-4o’s vocabulary. That is literally how the research above is produced. For questions like “how much Chinese was in GPT-4o’s tokenizer training?” the tokenizer itself is the most authoritative source available — and it is on your laptop.

13.8 Try It Yourself

Two companion notebooks turn this chapter’s theory into hands-on output you can observe.

Build BPE From Scratch — implement the five functions (get_stats, merge, train_bpe, encode, decode) yourself, in pure Python, inside your browser (JupyterLite, no installation). The notebook trains on a mixed multilingual + code + emoji corpus, so you will watch byte-level BPE discover German umlaut byte sequences, Python indentation patterns, and emoji byte prefixes — side by side with English merges. There is even a dedicated cell that runs the same algorithm on each sub-corpus independently so you can compare the vocabularies that emerge.

Tokenizer Playground — load real production tokenizers (tiktoken for GPT-4o, HuggingFace for LLaMA 3 and Gemma 2) and compare how they carve up the same text. This one runs in Google Colab (it needs Rust-backed packages), but the comparison is striking: the same German sentence can tokenize to 12 tokens in GPT-4o and 20 tokens in Gemma 2.

The lesson to carry forward: byte-level BPE is simple enough to implement in an afternoon, universal enough to handle every written language and every emoji, and battle-tested enough to power every frontier LLM. The infrastructure behind it — Unicode, UTF-8 — was standardized decades before transformers existed. The cleverness of GPT-2 was noticing that the pieces were already on the shelf.

14. Frequency vs. Likelihood: How WordPiece Differs from BPE

BPE is the dominant algorithm, but it is not the only one. Remember from Section 9.3 that any subword tokenizer must solve two problems: building a vocabulary and defining an encoding function. BPE solves both with ordered merge rules. WordPiece and Unigram take different approaches — but they must solve the same two problems.

WordPiece, the algorithm behind BERT, looks almost identical to BPE at first glance. It is bottom-up, iteratively merging pairs, building a vocabulary from small pieces. But there is one critical difference: how it chooses which pair to merge.

BPE merges the most frequent pair. WordPiece merges the pair with the highest likelihood ratio — a measure of how surprising it is that two pieces appear together, given how common each piece is individually.

14.1 Why the Difference Matters

Consider a corpus where the subword ##u (in WordPiece notation, ## marks a continuation piece) appears in almost every word. The pair (##u, ##g) might appear 20 times, making it the most frequent pair. BPE would merge it immediately.

But WordPiece asks a different question: “Is this pairing surprising?” Since ##u appears everywhere (36 times total) and ##g appears often (20 times), the fact that they appear together 20 times is not particularly surprising. The score is low:

Now consider the pair (##g, ##s), which appears only 5 times. That is far less frequent. But ##g (frequency 20) and ##s (frequency 5) rarely appear together. The 5 co-occurrences out of a possible 100 is more surprising:

WordPiece picks this rarer pair first, because the co-occurrence is more distinctive. It identifies bonds that are not just frequent but meaningful.

14.2 An Analogy

Think of the letters Q and U in English. Q appears in just 0.1% of text. U appears in 2.7%. If they were independent, you would expect QU about 0.003% of the time. But QU actually appears 0.1% of the time — 37 times more than chance predicts. That is enormous mutual information. WordPiece detects these distinctive bonds; BPE might miss them in favor of more frequent but less meaningful pairs.

14.3 WordPiece in Practice

BERT, published by Google in 2018, uses WordPiece with a vocabulary of 30,522 tokens. One practical difference: WordPiece marks continuation subwords with a ## prefix. The word “unbreakable” becomes un, ##break, ##able. No prefix means “this token starts a new word.” The ## prefix means “this token continues the previous word.” This makes it trivially easy to reconstruct the original text.

In summary: BPE builds its vocabulary by merging the most frequent pairs. WordPiece builds its vocabulary by merging the pairs whose co-occurrence is most surprising. Both are bottom-up, greedy, and iterative. Both produce subword vocabularies. The choice between them is a design decision, not a clear winner — but BPE has become the dominant standard for generative language models.

15. Start Big, Prune Down: Unigram and SentencePiece

There is a third approach that works in the opposite direction entirely.

BPE and WordPiece are bottom-up algorithms. They start with the smallest possible vocabulary (characters or bytes) and build up through merges. Unigram tokenization is top-down. It starts with a very large vocabulary — potentially hundreds of thousands of tokens — and iteratively removes the tokens that contribute least to the overall quality of the tokenization.

The quality metric is the likelihood of the training corpus under the current vocabulary. At each step, Unigram asks: “If I remove this token, how much worse does the tokenization become?” Tokens whose removal causes the least damage are pruned. The process repeats until the vocabulary reaches the target size.

15.1 SentencePiece: The Practical Framework

SentencePiece, developed by Google, is the framework that makes Unigram practical for real-world models. Its key innovation is that it treats the input as a raw byte stream — no pre-tokenization, no assumption that spaces separate words. This makes it naturally language-agnostic. Languages like Japanese and Chinese, which do not use spaces between words, work just as well as English.

SentencePiece can run either BPE or Unigram as its underlying algorithm. It is used by T5, mT5, and many multilingual models.

15.2 Three Approaches, One Goal

To summarize the landscape:

All three approaches solve the two subword problems from Section 9.3: they produce a vocabulary and an encoding function. BPE’s encoding function is its ordered merge rules. WordPiece’s is similar but with likelihood-ordered merges. Unigram’s is fundamentally different — it uses probabilistic segmentation, choosing the decomposition that maximizes the likelihood of the input under the learned vocabulary. The methods differ, but the goal is the same: finding the right granularity between words and characters.

In practice, BPE has won. Nearly every frontier model in 2024-2025 — GPT-4o, Claude, LLaMA 3, Mistral, DeepSeek, Qwen — uses byte-level BPE. WordPiece survives in BERT and its descendants, and SentencePiece/Unigram is still used by some models (notably Gemma with its 256K vocabulary), but BPE is the dominant standard. The table above might give the impression that these are equally popular alternatives. They are not. But understanding all three helps you recognize tokenization choices when you encounter different models in practice.

For this course, the important thing is not the algorithmic details but the shared principle: modern language models do not see words. They see subword tokens, learned from data, that balance vocabulary size against sequence length.

16. The Balancing Act: Vocabulary Size and Its Consequences

How big should the vocabulary be? This is not a trivial question — it is a genuine engineering tradeoff with real consequences on both sides.

16.1 Smaller Vocabulary, Longer Sequences

A smaller vocabulary means words get split into more pieces. The text is longer in token terms, which means the attention mechanism — which scales quadratically — costs more to run. Less text fits in the context window. The model thinks more slowly and sees less of your input.

16.2 Larger Vocabulary, More Parameters

A larger vocabulary means fewer tokens per sentence, which is good. More text fits in the context window. Attention is cheaper. But the embedding matrix grows linearly with vocabulary size, and the output projection matrix has the same shape. This adds parameters, memory, and cost.

The numbers are concrete. LLaMA 2 uses a vocabulary of 32,000 tokens. Its embedding matrix is 32,000 × 4,096 dimensions — 131 million parameters. LLaMA 3 quadrupled the vocabulary to 128,000. Now the embedding matrix alone is 525 million parameters, and the output projection has the same shape:

That is 13% of the entire 8-billion-parameter model devoted purely to vocabulary representation. A four-fold vocabulary increase directly added roughly a billion parameters.

16.3 The Industry Trend

The trend is clear: vocabularies are getting larger. Here is the progression:

The general principle: larger models with abundant training data benefit from bigger vocabularies, because the compression gains (shorter sequences, cheaper attention, more text in the context window) outweigh the parameter cost. But every token in the vocabulary needs enough training examples to learn a useful embedding. A massive vocabulary with insufficient data means many tokens will have poorly trained representations — which brings us to glitch tokens later.

17. Why LLMs Fail at Arithmetic

In Chapter 7, we saw that many models get 1234 + 5678 wrong. Now that you understand how tokenizers split text, you can see exactly why.

17.1 The Place-Value Problem

When you add two numbers by hand, you align digits by place value and work right to left — ones column, tens column, hundreds column, thousands column. Each column interacts with its neighbor through carry operations. The ones digit of 1234 is 4. The ones digit of 5678 is 8. Their sum is 12, so you write 2 and carry 1. This requires you to isolate individual digits and reason about their positions.

But the tokenizer does not split numbers digit by digit. It splits them by learned subword boundaries — the same BPE algorithm that works well for English words, applied to digit sequences.

The token 1234 might be split as 12 and 34. These are two opaque chunks. The model cannot reach inside a token to pull out individual digits. The token boundary cuts right across the place-value columns where digits need to interact. The ones digit (4) and the tens digit (3) are locked inside the same token as an indivisible unit.

17.2 Inconsistent Boundaries

The problem gets worse. Here are three consecutive numbers and how a typical tokenizer splits them:

Three nearly identical numbers, three completely different internal representations. The model has no consistent way to reason about digit positions because the tokenizer gives it no consistent structure to work with.

The boundary at 999 and 1000 is especially dramatic:

• 999 → one token: 999
• 1000 → one token: 1000
• 1001 → two tokens: 100 + 1

Adding just one to a number can completely change the model’s internal representation. This is why arithmetic accuracy collapses for larger numbers — the token boundaries become increasingly unpredictable.

17.3 The Fix Is in the Tokenizer

The good news: some models have addressed this. Gemma explicitly forces every digit to be its own token, regardless of what BPE merges would produce. The number 2164 becomes four separate tokens — 2, 1, 6, 4 — each perfectly aligned with its place value.

Research from ICLR 2025 goes further, showing that tokenizing digits from right to left — aligning boundaries with the direction of carry operations — improves arithmetic accuracy by over 22 percentage points.

The critical insight: the fix is not a smarter model. It is a smarter tokenizer. The model was never the bottleneck — its perception was.

18. The Multilingual Fairness Gap

Now let us solve another mystery from Act I: why English prompts often worked better than German ones.

18.1 The Token Tax

The phrase “artificial intelligence” in English costs just 2 tokens. How much does the same concept cost in other languages?

German costs roughly twice as many tokens as English for the same content. That means a German prompt consumes the context window roughly twice as fast. It costs twice as much per API call. And it leaves less room for the model’s reasoning.

For Hindi, the ratio is 5×. For Burmese, it can be 8×. The same model, the same context window, the same price — but 4 to 8 times less usable content for over a billion speakers of these languages.

18.2 Why This Happens

The root cause is training data. A typical tokenizer’s training corpus is approximately 85% English text, with the remaining 15% spread across all other languages. During BPE training, the algorithm sees millions of English words and learns highly efficient subword units for them. Common English words survive as single tokens. English morphology is well-represented.

German compound words, Hindi conjunct consonants, and Burmese script sequences are rare in this English-dominated training data. They never get merged into efficient tokens. They stay fragmented — split into individual bytes or small pieces that the model must reassemble.

18.3 The Practical Impact

This affects everything:

• Context window: A 200,000-token window fits roughly 150,000 words of English. For German, it fits roughly 100,000 words of equivalent content. For Hindi, it might fit 30,000 words.
• API cost: If an English API call costs $1, the same call in German costs ~$1.50-2.00. In Hindi, ~$4-5.
• Response quality: Longer token sequences mean more attention computation, more room for error, and more pressure on the context window.

18.4 Progress, But the Gap Persists

The situation is improving. LLaMA 3 halved the token count for Hindi compared to LLaMA 2, by training on significantly more multilingual data. GPT-4o expanded its vocabulary to 200,000 tokens, partly to better accommodate non-English languages. But the fundamental gap persists. This is not a model bias — it is a tokenizer bias, baked in at the very first step of the pipeline.

Mystery 2 — SOLVED. Why were English prompts better than German? Token efficiency. English text uses roughly one token per word. German uses roughly 1.3 to 1.5 tokens per word. The same prompt in German consumes more of the context window, costs more per API call, and leaves less room for the model’s reasoning. This is compounded by the training data being predominantly English.

You will verify this yourself in the lab exercise: tokenize the same sentence in English and German, count the tokens, and see the difference directly.

19. When the Map Has No Territory: Glitch Tokens

In January 2023, researchers discovered something bizarre. Certain tokens in GPT’s vocabulary cause the model to produce complete nonsense.

If you prompt the model with “Please repeat the following word: SolidGoldMagikarp,” the model does not repeat it. Instead, it might say “Distribute” or produce garbled output. The token TheNitromeFan causes similar breakdown. These are called glitch tokens — tokens that exist in the vocabulary but cause the model to malfunction.

19.1 Why They Exist

The explanation reveals an important architectural detail: the tokenizer and the model are trained on different datasets.

The tokenizer’s training data included raw internet text — Reddit posts, code repositories, forum discussions, web crawls. When BPE ran on this data, it encountered the Reddit username SolidGoldMagikarp thousands of times (because this user was extremely active on a specific subreddit) and gave it a dedicated token.

But the model’s training data was curated and filtered. Reddit usernames were cleaned out. The model never saw the token SolidGoldMagikarp during its training. The embedding for this token was initialized randomly — as all embeddings are — but it was never updated during training. It is pure noise.

When this token enters a prompt, its random embedding disrupts the transformer like a corrupted signal in a clean data stream. The model’s attention mechanism tries to make sense of a vector that encodes no meaning, and the result is unpredictable output.

19.2 The Reddit Connection

The connection is remarkable. Six of the original glitch tokens trace back to active users on r/counting — a subreddit dedicated to counting to infinity, one post at a time, for nearly a decade. Their usernames appeared so frequently in the tokenizer’s training data that BPE gave each one its own token. But when the model’s cleaner dataset excluded these posts, those tokens became orphans — entries in the vocabulary with no learned meaning.

19.3 The Scale

A 2024 study found roughly 7,900 glitch tokens across seven major models — about 4% of all tokens in a typical vocabulary. Most of these are harmless (rare tokens that the model has partially learned). But some cause genuine breakdown.

GPT-4 addressed this by rebuilding the tokenizer from scratch. SolidGoldMagikarp is no longer a single token — it now splits into subwords like Solid, Gold, Mag, ik, arp. But the fundamental issue — the tight coupling between tokenizer and model — remains a design tension in all current architectures.

The lesson: a token without a trained embedding is like a word in a dictionary with no definition. Worse than useless — actively misleading.

20. How Code Gets Tokenized

Everything so far has used natural language examples — strawberries, German sentences, arithmetic. But this is a course about AI-supported software development. Code has its own tokenization properties, and they directly affect how well your AI tools work.

20.1 Indentation Is Expensive

Python uses whitespace for structure. Every level of indentation costs tokens — and unlike the words they indent, those whitespace tokens carry structural meaning but no semantic content.

A flat function might cost 20 tokens. The same logic nested three levels deep might cost 30 tokens — ten tokens spent entirely on indentation. In a context window of 200,000 tokens, this feels negligible. But in a long coding session where Claude Code has read dozens of files, indentation whitespace across hundreds of functions adds up to thousands of tokens that could have been spent on actual content.

The practical implication: shorter, flatter code gives the model more room for reasoning. This is not just a style preference — it is a token budget decision.

20.2 Naming Conventions Affect Token Cost

How you name your variables and functions changes how the tokenizer splits them.

Consider a function called calculateAverageScore. In camelCase, the tokenizer might recognize the whole name as one or two tokens — common patterns like calculate and Average are well-represented in training data. But calculate_average_score in snake_case almost certainly splits at every underscore, producing more tokens.

The difference is not just cost — it affects perception. When a function name is one token, the model sees it as an atomic concept. When it splits into three or four tokens, the model must reconstruct the meaning from pieces. This is the same phenomenon as the strawberry problem: information that spans token boundaries is harder for the model to reason about.

You will verify this directly in the Tokenizer Playground notebook by tokenizing identifiers in different naming conventions.

20.3 Boilerplate Is Cheap, Novel Code Is Expensive

The tokenizer learned its merge rules from billions of lines of code. Common patterns were merged into efficient tokens. Rare patterns were not.

def __init__(self): is so common in Python training data that it compresses into very few tokens — perhaps 5 or 6. But a domain-specific signature like def calibrate_magnetometer(readings: list[SensorDatum]) might be 12+ tokens, because magnetometer and SensorDatum are rare in the training corpus.

This has a direct consequence for code generation: the model sees boilerplate with high resolution and novel code with low resolution. Each common pattern is a single, well-understood token. Each novel identifier is a sequence of subword pieces that the model must interpret in context.

This is one reason AI coding tools are excellent at generating boilerplate (__init__, __str__, standard CRUD operations) but struggle with novel domain logic — the tokenizer gives the model a clearer signal for the common patterns.

20.4 Programming Languages Differ

Java’s public static void main(String[] args) costs more tokens than Python’s equivalent. Every keyword (public, static, void) is a separate token. The type annotations, brackets, and semicolons all add up.

This is the programming language equivalent of the multilingual gap from Chapter 17. The same logic expressed in a verbose language consumes more of the context window than the same logic in a concise language. When the model generates Java, it has less remaining budget for reasoning compared to generating Python.

In the Tokenizer Playground notebook (Chapter 24), you already compared Python and Java function signatures. The broader implication: language choice affects how much context the model can hold — not because the model is biased, but because the tokenizer compresses different languages with different efficiency.

20.5 Comments as Context

Natural language comments and code are tokenized by the same tokenizer, but they play different roles. Comments consume tokens (cost) but provide the model with explicit descriptions of intent (value). This is the exact same tradeoff as CLAUDE.md: every token of guidance is a token not spent on code context.

A well-commented file might cost 30% more tokens than the same file without comments. But those comments tell the model what the code is supposed to do, reducing hallucination and improving generation quality. Whether the token cost is worth it depends on the task — for a simple rename, comments are overhead; for implementing a complex algorithm, they are essential context.

The rule of thumb: comment for the model the same way you would comment for a new team member. Explain why, not what. The model can see what the code does (it processes the tokens). It cannot see why you wrote it that way — unless you tell it.

20.6 Tokenizers Built for Code

Everything in §20.1–20.5 assumed a general-purpose tokenizer trained mostly on web text with code mixed in — cl100k_base (GPT-3.5/GPT-4), o200k_base (GPT-4o), or a Llama-family SentencePiece model. Code-focused labs sometimes take a different path. Here’s what changes when they do.

Two patterns in production. When building a code-specialized model, you can either inherit a general tokenizer or retrain one on code-heavy data. Both are represented among frontier code models:

• Code Llama (Meta) — reused Llama 2’s SentencePiece BPE tokenizer unchanged. The team added only a handful of fill-in-the-middle special tokens (<FIM_PREFIX>, <FIM_SUFFIX>, <FIM_MIDDLE>) so the model can complete text in the middle of a file, not just append to the end. The vocabulary stayed identical; the special tokens are pure additions.
• StarCoder2 (Lozhkov et al., 2024, arXiv:2402.19173) — trained a completely custom tokenizer on The Stack v2: 67.5 TB of source code across 619 programming languages. The resulting vocabulary has visibly more indentation tokens, bracket patterns, language-specific keywords, and common function names — directly optimizing for the §20.1–20.4 pain points.
• DeepSeek Coder similarly retrained for a code-heavy mix.
• The original Codex (the first GitHub Copilot model) reused GPT-3’s general tokenizer unchanged, despite being a code specialist — evidence that “retrain” is not automatic.

What changes when the tokenizer knows code. Running the same Python file through a general-purpose tokenizer and through StarCoder2’s code-specialized tokenizer exposes differences that map almost 1:1 to §20.1–20.5:

• Indentation becomes cheap. Four spaces of Python indent is one well-optimized token, not a byte pattern that got merged opportunistically.
• Common identifiers merge more aggressively. Names like calculate_average_score that fragment in a web-trained tokenizer often compress into fewer tokens when the tokenizer learned merges from a code-dense corpus.
• Language-specific syntax gets first-class treatment. StarCoder2’s vocabulary includes single tokens for common Rust (fn main(), impl), Java boilerplate (public static void main), and JSX patterns — constructs a web-trained tokenizer sees only sporadically.
• The programming-language gap from §20.4 shrinks because The Stack v2 exposes the tokenizer to hundreds of languages proportionally, not just the popular few.

Fill-in-the-middle: a special-token case study. FIM demonstrates how small vocabulary additions unlock new capabilities. When a code model is trained on inputs shaped like <FIM_PREFIX>code_before<FIM_SUFFIX>code_after<FIM_MIDDLE>, it learns to infill between two cursors — exactly what IDE copilots need. These three tokens are not BPE-learned merges; they are reserved IDs added to the vocabulary by hand. The algorithm is identical; only the special-token inventory changes.

What this means for you. The tokenizer behind your AI tool is one of its hidden performance levers:

• Claude Code, Cursor, GitHub Copilot run on frontier models (Claude, GPT-4o, etc.) with general-purpose tokenizers. Those models have seen enormous amounts of code in training, so the gap to a specialist is smaller than you might guess — but it exists.
• Code-specialist models (StarCoder2, DeepSeek Coder, Code Llama) compress code more aggressively and handle obscure languages better. In exchange, they compress natural language less well and are less flexible as conversational assistants.

No single tokenizer wins everywhere. The choice trades code-density against general-text fluency — the same lossless-vs-efficient tension from §13.5, applied to a different axis. Both tokenizers are lossless; each is efficient for different content.

You can verify all of this directly in the Tokenizer Playground notebook: run a representative code snippet through cl100k_base, then through StarCoder2’s tokenizer (HuggingFace bigcode/starcoder2-15b). The token-count delta is typically 20–40% — smaller than you’d expect, but measurable.

21. The Complete Journey: From Text to Transformer

Now let us put it all together. In Chapter 6, you saw the complete journey in miniature — from <BOS> dragon drinks to the predicted token potion, using a toy vocabulary of 10 words. Here is the same journey at real scale, with all six stations visible, for a single sentence.

Input: "The tokenizer processes text."

21.1 Station 1: Pre-tokenization

Before BPE runs, a regex pattern splits the text at word boundaries. This regex is smarter than simple whitespace splitting — it handles contractions (splitting “don’t” into “do” and “n’t”), preserves leading spaces as part of the following word, and separates punctuation.

Our sentence becomes five pieces: The, ` tokenizer, processes, text, and .` (where the leading space is part of the token, not a separate entity).

21.2 Station 2: BPE Segmentation

Each piece enters the BPE engine, which applies its learned merge rules. The word “tokenizer” is not common enough to survive as a single token — it splits into token and izer. But “processes” and “text” are frequent enough to pass through intact.

21.3 Station 3: Token ID Assignment

Each subword is looked up in the vocabulary dictionary and mapped to a unique integer:

The → 464 | token → 11241 | izer → 7509 | processes → 7767 | text → 2420 | . → 13

21.4 Station 4: Special Token Insertion

An end-of-text marker is appended: <|endoftext|>. This tells the model where the document boundary falls. Different models use different special tokens — BERT adds a classification token [CLS] at the start, T5 uses sentinel tokens for its training objectives — but the principle is the same: special tokens carry structural information that the text itself does not.

The sequence is now: 464 | 11241 | 7509 | 7767 | 2420 | 13 | <eot>

21.5 Station 5: Embedding Lookup

This is the critical bridge from integers to vectors. Each token ID indexes into the embedding matrix — that 50,257-row table holding 38.6 million parameters. Seven dense vectors emerge, each containing hundreds of dimensions of learned meaning.

The integer 464 becomes a 768-dimensional vector encoding the meaning of “The.” The integer 11241 becomes a vector encoding “token.” And so on. The original text is now gone — what remains is a sequence of numerical representations.

21.6 Station 6: Positional Encoding

A wave pattern overlays each vector, encoding that token’s position in the sequence. Without this, the model would see the same set of vectors regardless of word order — “The cat sat” and “sat cat the” would be indistinguishable.

After positional encoding, each vector encodes both what the token means and where it sits in the sequence.

21.7 The Transformer Receives

And now the final sequence — seven rich, information-dense vectors — flows into the transformer. This, not your text, is what the model actually processes. The self-attention mechanism will compare these vectors, the feed-forward layers will transform them, and the output layer will use them to predict the next token.

Every single thing the model knows about your input came through this pipeline. Every bias in the tokenizer, every blind spot in the vocabulary, every inconsistency in how numbers or non-English text are split — it all carries forward into every layer of the model.

Tokenization is not preprocessing. It is the first layer of the architecture.

22. What This Means for CLAUDE.md

Let us briefly return to Mystery 1, which we posed at the start of this lecture: why did CLAUDE.md improve results?

We cannot fully answer this yet — the complete answer requires understanding the attention mechanism (Lecture 5) and the orchestration layer (Lecture 7). But we can now say something important:

CLAUDE.md is text. Text becomes tokens. Tokens become vectors. Vectors enter the transformer.

When you wrote a 50-line CLAUDE.md for the Locust project in Lecture 3, that file was not loaded into some special configuration system. It was not parsed by a rule engine. It was converted into approximately 200-500 tokens through exactly the same BPE pipeline we just described. Those tokens were then injected into the same sequence as your prompt, processed by the same embedding lookup, encoded with the same positional information, and fed into the same transformer.

From the model’s perspective, there is no difference between CLAUDE.md tokens and your question tokens. Both are just text in the sequence. There is no special “instruction flag.” The distinction between “project instructions” and “user question” exists only in the software that manages the model — not in the model itself.

This is the first clue to Mystery 1. We will complete the answer over the next three lectures.

23. Key Takeaways

Five things to carry forward from this lecture:

Tokenization is the bridge between human language and neural computation. Text becomes integers. Integers become vectors. Vectors become the representations the transformer works with. Every model begins here. And tokenization is always two things: a vocabulary (the set of allowed tokens) and an encoding function (the algorithm that splits text into those tokens). The vocabulary alone is not enough — without the encoding rules, splitting is ambiguous.

Subword tokenization is the universal standard. It keeps common words whole while decomposing rare words into reusable building blocks. BPE is the dominant algorithm — it solves both the vocabulary and encoding problems simultaneously through ordered merge rules. WordPiece and Unigram are viable alternatives that solve the same two problems with different strategies.

The consequences are real and measurable. Arithmetic failures trace directly to inconsistent number tokenization. Non-English speakers pay 2-8× more tokens for the same semantic content. Glitch tokens emerge whenever tokenizer and model training data diverge. These are not model bugs — they are tokenizer bugs.

Vocabulary size is an engineering tradeoff. Smaller vocabularies mean longer sequences and more expensive attention. Larger vocabularies mean more parameters and higher memory cost. The industry trend is toward larger vocabularies as models grow.

Tokenization is architecture. It is not a detail you can safely ignore. It shapes what the model sees, how much it can hold in context, and precisely where it will stumble. The next time a language model surprises you — consider what the tokenizer did first.

And one forward-looking note: the field is actively exploring whether tokenization can be eliminated entirely. Meta’s Byte Latent Transformer already matches LLaMA 3’s performance while operating on raw bytes — no tokenizer at all. Whether this approach scales remains an open question.

24. Mystery Board Update

Let us check where we stand. The full Mystery Board tracks all mysteries across Act II — check it after each lecture to see the latest progress.

One mystery solved. Two partially answered. Three still open. Next week, we open the Transformer box and see how those token vectors actually influence the model’s output.

25. Lab: The Tokenizer Playground

Now it is your turn. The rest of this session is hands-on. You will work through interactive Jupyter notebooks that let you explore tokenizers, compare models, and discover the phenomena from this lecture empirically.

25.1 Notebook: Tokenizer Playground

Open the Tokenizer Playground notebook. This notebook requires tiktoken and transformers — open it in Google Colab using the badge inside the notebook. You will:

• Tokenize text with GPT-4o’s tokenizer and see exactly where the splits fall
• Compare tokenizers across frontier models (GPT-4o, LLaMA 3, Gemma 2)
• Measure the multilingual token gap between English and German
• Explore how code gets tokenized — indentation cost, naming conventions, boilerplate vs novel code
• Dig into vocabularies — find the longest token, hunt for glitch tokens
• Estimate context window budgets for your CLAUDE.md from Lecture 3

Work through the notebook from top to bottom. Each section has questions to answer — collect your findings for the deliverables below.

25.2 Notebook: Build BPE From Scratch (Self-Study)

The Build BPE notebook is a self-study exercise that runs directly in your browser (no installation needed). You will implement the BPE algorithm step by step — counting byte pairs, merging the most frequent, building an encoder and decoder. By the end, you will have a working tokenizer that you built yourself.

This notebook is optional but highly recommended. Building the algorithm makes everything in Chapters 10-11 concrete.

25.3 Deliverables

Document your findings from the Tokenizer Playground in an AI Diary entry:

• At least 5 surprising tokenization results with explanations
• The English-German token count comparison with your calculated ratio
• Your CLAUDE.md token budget estimate — how much of the context window does it consume?
• One thing you would change about your prompting approach based on what you now understand about tokenization