04 Lecture 4 Quiz: Tokenization - How Machines Read

An LLM is a next-token prediction machine: it computes a probability distribution over its vocabulary, picks a token, appends it, and repeats. There is no planning step (A), no retrieval from a stored answer database (B), and no global search over complete outputs (D) — just iterated local prediction. Every line of code and every paragraph emerges from this single loop.

Inside the transformer there are no words and no letters — only tensors of floating-point numbers. Every operation (matrix multiplication, attention, activation) takes numbers and produces numbers. Symbols like def have no inherent numerical meaning — you cannot multiply def by return. Compression (B) is a side effect at best; tokenization is not encryption (D); the universal-representation framing (C) is a consequence, not the cause.

The pipeline is text → tokenizer → integer token IDs → embedding lookup → dense vectors → transformer. The tokenizer first converts text into integers; those integers then index into the embedding matrix, producing the dense vectors the transformer actually processes. Option A reverses tokenization and embedding lookup. Options C and D place the transformer before the embedding lookup, which is impossible — the transformer requires the dense vectors as input.

The lecture lists three problems with word-level tokenization: vocabulary explosion, no morphological awareness (A), and the unknown-word problem (D) — and calls the last “the real killer.” Any unseen word becomes [UNK], and the meaning is gone forever. Morphological awareness (A) is a real concern but the model can partially compensate from context. Punctuation (B) and Chinese segmentation (C) are workable engineering problems; UNK is unrecoverable information loss.

Attention cost grows as O(N²) in sequence length. Doubling the tokens makes attention 4× more expensive. A 500-word paragraph becomes ~2,500 character tokens — a 25× cost increase versus word-level. Character-level tokenization trades vocabulary explosion for sequence explosion. Option A is false (characters do contribute to meaning when composed); C and D are simply factually wrong.

Tokenization is two things: a vocabulary (the allowed tokens) and an encoding function (the algorithm that splits text into those tokens). The vocabulary alone is not enough — if t, h, e, th, and the are all in V, multiple decompositions of “the” are valid (t|h|e vs th|e vs the). BPE solves both problems with its ordered merge rules. Vocabulary size (A) is a hyperparameter, not the second core problem; language selection (B) and the model training (C) are separate concerns.

BPE is purely frequency-driven. It scans the corpus, counts every adjacent pair, and merges the most frequent one — no linguistic knowledge required. The likelihood-ratio criterion (A) is exactly what distinguishes WordPiece from BPE. Expert rules (C) is the rule-based approach the lecture explicitly rejected. Randomness (D) would defeat the purpose of learning useful merges.

The trained tokenizer does not retrain (A). It replays the merge rules in training order — Rule #1 fires first (the most frequent pair during training), then Rule #2, and so on. This often looks like longest match (B), but the mechanism is fundamentally an ordered sequence of rewrite rules, which can produce a different segmentation than a greedy longest-match search. Whitespace splitting (D) is the word-level approach BPE replaces.

Tokens are atomic — the smallest unit the model can perceive. When “strawberry” is split into str, aw, berry, the model sees three opaque chunks, not ten letters. It has no mechanism to look inside a token and count characters. Training-data frequency (A) is irrelevant — the word is common. Counting cost (B) is not the bottleneck; even short tokenized inputs produce wrong counts. Safety refusal (C) is not happening — the model confidently produces the wrong answer.

A byte is 8 bits, so 2^8 = 256 possible values (0–255). UTF-8 reduces every Unicode character — every script, every emoji, every symbol yet to be added — to a sequence of these bytes, so the 256 base tokens cover every possible input by construction. ASCII has only 128 characters (A is wrong). GPU registers (C) are unrelated to vocabulary design. The choice is mathematical, not empirical (D).

This is the central lossless-vs-efficient distinction. Byte-level BPE is encoding-lossless by construction — the 256-byte base vocabulary covers every possible byte sequence. A new emoji simply uses 4 base-byte tokens instead of the 1 token a freshly trained tokenizer might assign it. No information is lost (so B and C are wrong); tokenizers are static artifacts that do not retrain themselves (D). The cost is efficiency, not correctness.

Both are bottom-up algorithms that iteratively merge pairs, so the difference is the merge criterion. BPE picks the most frequent pair; WordPiece picks the pair whose co-occurrence is most surprising relative to each piece’s individual frequency. WordPiece favors distinctive bonds (like Q-U in English). Top-down pruning (B) describes Unigram, not WordPiece. Both algorithms are unsupervised (D is false). Byte vs character (A) is a separate design axis.

The embedding matrix has one row per vocabulary entry, so vocabulary size determines its parameter count directly. LLaMA 3’s embedding matrix is 128,000 × 4,096 ≈ 525M parameters; with the output projection of identical shape, that is ~1.05 billion parameters — about 13% of the entire 8B-parameter model. Table-lookup cost (A) is O(1) and negligible. Overfitting (C) is not an automatic consequence. Training data (D) does need to be sufficient for rare tokens, but no fixed quadrupling rule applies.

The tokenizer splits numbers by frequency-based merges, not by place value. 2249 is 2+249; 2250 is 22+50; 2251 is 225+1 — three nearly identical numbers, three completely different internal representations. The token boundaries cut across the place-value columns that arithmetic depends on. Floating-point precision (A) is more than adequate. Arithmetic examples (B) are plentiful in training data. Safety refusal (C) is not happening — the model produces confidently wrong answers. Gemma fixed this by forcing one digit per token.

The cause is training-data composition, not the algorithm or the model. BPE merges patterns it sees often. With ~85% English training data, English patterns get efficient merges; German compounds, Hindi conjuncts, and Burmese scripts stay fragmented. The cost can reach 2× in German, 5× in Hindi, 8× in Burmese — same context window, same API price, far less usable content. Alphabet size (A) is a side issue compared to merge efficiency. Architecture bias (C) and UTF-16 storage (D) are simply incorrect.

The tokenizer and the model are trained on different datasets. Reddit content gave BPE thousands of occurrences of SolidGoldMagikarp (a username), earning a dedicated token. The model’s cleaner training set excluded that content, so the embedding row was initialized randomly and never updated. The token’s random embedding disrupts the model like noise. Honeypots (B), reserved instructions (C), and corrupted entries (D) are not how this works — the vocabulary entry is valid; only its embedding is untrained.

The tokenizer learned its merges from billions of lines of code. Frequent patterns (def __init__, common library calls) became efficient short tokens; rare domain-specific names (calibrate_magnetometer) did not. The model “sees” boilerplate at high resolution and novel code at low resolution. Keywords are merged by frequency, not granted special status (A). Indentation costs real tokens (B is false). Different naming conventions split very differently — snake_case typically fragments at underscores (D is false).

CLAUDE.md is text. Text becomes tokens through the same BPE tokenizer; those tokens become vectors through the same embedding lookup; those vectors enter the same transformer in the same sequence as the user prompt. There is no rule engine (B), no separate instruction channel (C), and no signature verification (D) — the distinction between “project instructions” and “user question” exists only in the software around the model, not in the model itself. This is the first clue to why CLAUDE.md works (and sometimes does not).