Home

04 Lecture 4 Quiz: Tokenization - How Machines Read

April 2026 (3154 Words, 18 Minutes)

quiz tokenization bpe embeddings byte-level unicode multilingual

New: Interactive Quiz Available!

Want to track your progress and get instant feedback? Try our new interactive version with progress tracking, immediate explanations, and score calculation.

Start Interactive Quiz →

Instructions

This quiz tests your understanding of key concepts from Lecture 4
Each question has 4 options with exactly one correct answer
Try to answer without referring to the lecture materials
Focus on understanding concepts, not memorizing details

Question 1: The Core Mechanism

At the most fundamental level, what does a Large Language Model (LLM) do when it generates text?

A) It parses the prompt into an abstract syntax tree, plans a response, then writes it out
B) It retrieves the closest matching answer from a compressed database of training examples
C) It predicts the most likely next token, appends it to the sequence, and repeats — one token at a time
D) It runs a search algorithm over possible outputs and returns the highest-scoring complete response

Show Answer

Correct Answer: C

An LLM is a next-token prediction machine: it computes a probability distribution over its vocabulary, picks a token, appends it, and repeats. There is no planning step (A), no retrieval from a stored answer database (B), and no global search over complete outputs (D) — just iterated local prediction. Every line of code and every paragraph emerges from this single loop.

Question 2: Why Tokenize

Why must text be converted into numbers before reaching the transformer?

A) Neural networks operate on numerical tensors — they have no mechanism to process raw symbols like letters or words
B) Numbers compress text much more efficiently than characters, reducing memory usage during training
C) Different languages use different alphabets, so numbers provide a universal representation
D) The transformer needs encrypted input, and tokenization is the standard encryption layer

Show Answer

Correct Answer: A

Inside the transformer there are no words and no letters — only tensors of floating-point numbers. Every operation (matrix multiplication, attention, activation) takes numbers and produces numbers. Symbols like def have no inherent numerical meaning — you cannot multiply def by return. Compression (B) is a side effect at best; tokenization is not encryption (D); the universal-representation framing (C) is a consequence, not the cause.

Question 3: The Full Pipeline

Which sequence correctly describes how text reaches the transformer?

A) Raw text → embedding lookup → token IDs → transformer
B) Raw text → token IDs → embedding lookup → transformer
C) Raw text → transformer → token IDs → embedding lookup
D) Raw text → token IDs → transformer → embedding lookup

Show Answer

Correct Answer: B

The pipeline is text → tokenizer → integer token IDs → embedding lookup → dense vectors → transformer. The tokenizer first converts text into integers; those integers then index into the embedding matrix, producing the dense vectors the transformer actually processes. Option A reverses tokenization and embedding lookup. Options C and D place the transformer before the embedding lookup, which is impossible — the transformer requires the dense vectors as input.

Question 4: Word-Level Tokenization Problems

What is the most catastrophic problem with word-level tokenization for modern language models?

A) Word-level tokenizers fail to capture morphological relationships between related words like run and running
B) Word-level tokenizers cannot process punctuation or whitespace correctly
C) Languages without space-separated words like Chinese cannot be tokenized at the word level
D) Unknown words must be replaced with an [UNK] token, which permanently destroys information

Show Answer

Correct Answer: D

The lecture lists three problems with word-level tokenization: vocabulary explosion, no morphological awareness (A), and the unknown-word problem (D) — and calls the last “the real killer.” Any unseen word becomes [UNK], and the meaning is gone forever. Morphological awareness (A) is a real concern but the model can partially compensate from context. Punctuation (B) and Chinese segmentation (C) are workable engineering problems; UNK is unrecoverable information loss.

Question 5: Why Not Character-Level

Character-level tokenization solves the unknown-word problem with a tiny vocabulary. Why do real models avoid it anyway?

A) Characters cannot capture meaning because they have no semantic content
B) Transformer attention scales quadratically with sequence length, so character-level sequences become prohibitively expensive
C) Many languages (like Chinese) do not have characters in the Western sense
D) Characters cannot be stored in standard floating-point format

Show Answer

Correct Answer: B

Attention cost grows as O(N²) in sequence length. Doubling the tokens makes attention 4× more expensive. A 500-word paragraph becomes ~2,500 character tokens — a 25× cost increase versus word-level. Character-level tokenization trades vocabulary explosion for sequence explosion. Option A is false (characters do contribute to meaning when composed); C and D are simply factually wrong.

Question 6: The Two Subword Problems

Section 9.3 of the lecture states that any subword tokenization algorithm must solve two problems together. What are they?

A) Choosing a base alphabet AND choosing a final vocabulary size
B) Compressing the training corpus AND selecting which languages to support
C) Training the tokenizer AND training the transformer to use it
D) Defining a vocabulary V AND defining an encoding function f that maps text to tokens from V

Show Answer

Correct Answer: D

Tokenization is two things: a vocabulary (the allowed tokens) and an encoding function (the algorithm that splits text into those tokens). The vocabulary alone is not enough — if t, h, e, th, and the are all in V, multiple decompositions of “the” are valid (t|h|e vs th|e vs the). BPE solves both problems with its ordered merge rules. Vocabulary size (A) is a hyperparameter, not the second core problem; language selection (B) and the model training (C) are separate concerns.

Question 7: How BPE Picks Merges

During training, how does standard Byte Pair Encoding (BPE) decide which pair to merge next?

A) It merges the pair with the highest likelihood ratio relative to individual frequencies
B) It merges the most frequent adjacent pair in the corpus
C) It merges pairs based on linguistic rules supplied by experts
D) It merges pairs randomly until the vocabulary reaches the target size

Show Answer

Correct Answer: B

BPE is purely frequency-driven. It scans the corpus, counts every adjacent pair, and merges the most frequent one — no linguistic knowledge required. The likelihood-ratio criterion (A) is exactly what distinguishes WordPiece from BPE. Expert rules (C) is the rule-based approach the lecture explicitly rejected. Randomness (D) would defeat the purpose of learning useful merges.

Question 8: Encoding New Text with BPE

Once BPE has been trained, how does the tokenizer encode a new sentence it has never seen?

A) It re-runs the full BPE training algorithm on the new sentence
B) It performs longest-match lookup, picking the largest vocabulary token that fits at each position
C) It applies the learned merge rules in the exact order they were learned during training
D) It splits on spaces and looks up each word in the vocabulary

Show Answer

Correct Answer: C

The trained tokenizer does not retrain (A). It replays the merge rules in training order — Rule #1 fires first (the most frequent pair during training), then Rule #2, and so on. This often looks like longest match (B), but the mechanism is fundamentally an ordered sequence of rewrite rules, which can produce a different segmentation than a greedy longest-match search. Whitespace splitting (D) is the word-level approach BPE replaces.

Question 9: The Strawberry Problem

Why do most LLMs historically fail at counting the r’s in “strawberry”?

A) The model’s training data did not include enough examples of strawberries
B) Counting is computationally too expensive for the transformer’s forward pass
C) The model deliberately refuses to count letters as a safety measure
D) The tokenizer splits the word into chunks (like str, aw, berry) that the model cannot peek inside to inspect individual letters

Show Answer

Correct Answer: D

Tokens are atomic — the smallest unit the model can perceive. When “strawberry” is split into str, aw, berry, the model sees three opaque chunks, not ten letters. It has no mechanism to look inside a token and count characters. Training-data frequency (A) is irrelevant — the word is common. Counting cost (B) is not the bottleneck; even short tokenized inputs produce wrong counts. Safety refusal (C) is not happening — the model confidently produces the wrong answer.

Question 10: The 256-Byte Base Vocabulary

Why does byte-level BPE start with exactly 256 base tokens and need no [UNK] token?

A) 256 is the number of characters in the ASCII standard
B) A byte is 8 bits, giving 2^8 = 256 values; combined with UTF-8 this guarantees every possible text can be encoded
C) 256 is the maximum vocabulary size that fits in a single GPU register
D) Experiments showed 256 produces the best multilingual performance

Show Answer

Correct Answer: B

A byte is 8 bits, so 2^8 = 256 possible values (0–255). UTF-8 reduces every Unicode character — every script, every emoji, every symbol yet to be added — to a sequence of these bytes, so the 256 base tokens cover every possible input by construction. ASCII has only 128 characters (A is wrong). GPU registers (C) are unrelated to vocabulary design. The choice is mathematical, not empirical (D).

Question 11: Lossless vs. Efficient

A new emoji is added to Unicode after a byte-level BPE tokenizer has been trained. What happens when a user types it?

A) The emoji’s UTF-8 bytes pass through as several base-byte tokens — lossless but using more tokens than a freshly trained tokenizer would
B) The emoji is replaced with an [UNK] token and the model produces garbled output
C) The tokenizer fails with an error because the bytes are unknown to its merge rules
D) The tokenizer automatically retrains itself to incorporate the new emoji

Show Answer

Correct Answer: A

This is the central lossless-vs-efficient distinction. Byte-level BPE is encoding-lossless by construction — the 256-byte base vocabulary covers every possible byte sequence. A new emoji simply uses 4 base-byte tokens instead of the 1 token a freshly trained tokenizer might assign it. No information is lost (so B and C are wrong); tokenizers are static artifacts that do not retrain themselves (D). The cost is efficiency, not correctness.

Question 12: WordPiece vs BPE

What is the key algorithmic difference between WordPiece (used in BERT) and BPE?

A) WordPiece operates on bytes; BPE operates on characters
B) WordPiece merges top-down (pruning); BPE merges bottom-up
C) WordPiece picks the pair with the highest likelihood ratio; BPE picks the most frequent pair
D) WordPiece is supervised by linguistic experts; BPE is unsupervised

Show Answer

Correct Answer: C

Both are bottom-up algorithms that iteratively merge pairs, so the difference is the merge criterion. BPE picks the most frequent pair; WordPiece picks the pair whose co-occurrence is most surprising relative to each piece’s individual frequency. WordPiece favors distinctive bonds (like Q-U in English). Top-down pruning (B) describes Unigram, not WordPiece. Both algorithms are unsupervised (D is false). Byte vs character (A) is a separate design axis.

Question 13: Vocabulary Size Tradeoff

LLaMA 3 quadrupled its vocabulary from LLaMA 2’s 32,000 to 128,000 tokens. What is the main cost of doing this?

A) The model becomes slower to query because lookups take longer in larger tables
B) The embedding matrix and output projection both grow linearly, adding hundreds of millions of parameters
C) Larger vocabularies always reduce model accuracy due to overfitting
D) Quadrupling the vocabulary requires quadrupling the training corpus size

Show Answer

Correct Answer: B

The embedding matrix has one row per vocabulary entry, so vocabulary size determines its parameter count directly. LLaMA 3’s embedding matrix is 128,000 × 4,096 ≈ 525M parameters; with the output projection of identical shape, that is ~1.05 billion parameters — about 13% of the entire 8B-parameter model. Table-lookup cost (A) is O(1) and negligible. Overfitting (C) is not an automatic consequence. Training data (D) does need to be sufficient for rare tokens, but no fixed quadrupling rule applies.

Question 14: Why Arithmetic Fails

Why do LLMs often fail at simple multi-digit arithmetic like 1234 + 5678?

A) Floating-point precision is insufficient for integer arithmetic in the transformer
B) The training data did not contain enough arithmetic examples
C) Transformers are explicitly designed to refuse arithmetic operations as a safety measure
D) BPE splits numbers along learned subword boundaries that cut across place-value columns, so individual digits cannot be reliably isolated

Show Answer

Correct Answer: D

The tokenizer splits numbers by frequency-based merges, not by place value. 2249 is 2+249; 2250 is 22+50; 2251 is 225+1 — three nearly identical numbers, three completely different internal representations. The token boundaries cut across the place-value columns that arithmetic depends on. Floating-point precision (A) is more than adequate. Arithmetic examples (B) are plentiful in training data. Safety refusal (C) is not happening — the model produces confidently wrong answers. Gemma fixed this by forcing one digit per token.

Question 15: The Multilingual Fairness Gap

Why do non-English languages typically cost more tokens than English for equivalent content?

A) Non-English alphabets generally contain more distinct characters, which inflates the per-word token count
B) BPE’s training corpus is English-dominated (~85%), so it learns efficient merges for English and leaves other languages fragmented
C) The transformer architecture is internally biased toward English word order
D) Non-English text is stored in UTF-16 internally, doubling the byte count

Show Answer

Correct Answer: B

The cause is training-data composition, not the algorithm or the model. BPE merges patterns it sees often. With ~85% English training data, English patterns get efficient merges; German compounds, Hindi conjuncts, and Burmese scripts stay fragmented. The cost can reach 2× in German, 5× in Hindi, 8× in Burmese — same context window, same API price, far less usable content. Alphabet size (A) is a side issue compared to merge efficiency. Architecture bias (C) and UTF-16 storage (D) are simply incorrect.

Question 16: Glitch Tokens

Why do “glitch tokens” like SolidGoldMagikarp cause unpredictable model behavior?

A) The tokenizer’s training data contained these strings, but the model’s training data did not — so their embeddings were never trained beyond random initialization
B) These tokens are deliberately inserted as honeypots to detect prompt injection attempts
C) These tokens trigger reserved system instructions inside the transformer
D) Glitch tokens are corrupted entries left over from earlier tokenizer versions

Show Answer

Correct Answer: A

The tokenizer and the model are trained on different datasets. Reddit content gave BPE thousands of occurrences of SolidGoldMagikarp (a username), earning a dedicated token. The model’s cleaner training set excluded that content, so the embedding row was initialized randomly and never updated. The token’s random embedding disrupts the model like noise. Honeypots (B), reserved instructions (C), and corrupted entries (D) are not how this works — the vocabulary entry is valid; only its embedding is untrained.

Question 17: Code Tokenization

Which statement about how code gets tokenized is correct?

A) Code tokenizers always use language-specific keywords as atomic tokens regardless of frequency
B) Indentation whitespace is free — it does not consume any tokens
C) Common boilerplate (like def __init__(self):) compresses into few tokens, while novel domain-specific identifiers fragment into many
D) snake_case and camelCase identifiers always tokenize to the same number of tokens

Show Answer

Correct Answer: C

The tokenizer learned its merges from billions of lines of code. Frequent patterns (def __init__, common library calls) became efficient short tokens; rare domain-specific names (calibrate_magnetometer) did not. The model “sees” boilerplate at high resolution and novel code at low resolution. Keywords are merged by frequency, not granted special status (A). Indentation costs real tokens (B is false). Different naming conventions split very differently — snake_case typically fragments at underscores (D is false).

Question 18: CLAUDE.md from the Model’s View

From the transformer’s point of view, what is special about the CLAUDE.md file when it is included in a request?

A) Nothing is special — it is converted into tokens through the same BPE pipeline as your prompt and processed as part of the same sequence
B) It is parsed by a rule engine that applies its instructions deterministically before the model runs
C) It is loaded into a separate “instruction” embedding channel with higher attention priority
D) It is encoded with cryptographic signatures the model verifies before processing

Show Answer

Correct Answer: A

CLAUDE.md is text. Text becomes tokens through the same BPE tokenizer; those tokens become vectors through the same embedding lookup; those vectors enter the same transformer in the same sequence as the user prompt. There is no rule engine (B), no separate instruction channel (C), and no signature verification (D) — the distinction between “project instructions” and “user question” exists only in the software around the model, not in the model itself. This is the first clue to why CLAUDE.md works (and sometimes does not).

Scoring Guide

16-18 correct: Excellent! You have a strong grasp of the core concepts
12-15 correct: Good understanding! Review the areas you missed
8-11 correct: Fair understanding. Revisit the lecture materials for key concepts
Below 8: Please review the lecture carefully and reach out if you need clarification

Key Takeaways to Remember

LLMs are next-token prediction machines — every output emerges from iterated next-token prediction, not from planning or retrieval.
Tokenization bridges text and numbers — neural networks operate only on numerical tensors, so text must be converted to integers and then to embedding vectors before the transformer can use it.
Tokenization is two things: a vocabulary AND an encoding function — the vocabulary alone is ambiguous; encoding rules are needed to deterministically split text.
Subword tokenization is the modern standard — keeping common words whole while decomposing rare words into reusable pieces avoids both vocabulary explosion (word-level) and sequence explosion (character-level).
BPE merges the most frequent pair, iteratively — purely data-driven, no linguistic rules; the ordered merge rules serve as both vocabulary builder and encoding function.
Byte-level BPE is lossless forever — 256 base byte tokens combined with UTF-8 mean every possible text (including future emoji) can always be encoded, no [UNK] token possible.
Lossless ≠ efficient — byte-level BPE always preserves information, but rare patterns (non-English text, new emoji) cost more tokens than well-merged English text.
Tokens are atomic — the model cannot look inside a token to examine individual characters; this is why letter counting and digit-level arithmetic fail.
The multilingual fairness gap is a tokenizer bias — English-dominated training data produces tokenizers that fragment other languages, making non-English usage 2–8× more expensive in tokens.
CLAUDE.md is just tokens — from the model’s perspective there is no “instruction channel”; project instructions enter the transformer through exactly the same pipeline as the user prompt.