04 Notebook: Build BPE From Scratch (Self-Study)

April 2026 (158 Words, 1 Minutes)

Goal

Implement the Byte Pair Encoding algorithm from scratch. Build your own tokenizer step by step — counting byte pairs, merging the most frequent, and creating an encoder and decoder that can tokenize new text using the merge rules you discovered.

This notebook is optional but highly recommended. Building the algorithm yourself makes everything in Chapters 10-11 of the lecture concrete.

Open the Notebook

Open in JupyterLite — runs directly in your browser, no installation required.

This notebook uses only pure Python, so it works in JupyterLite (unlike the Tokenizer Playground). Your progress is saved in your browser — export the notebook if you want a permanent copy.

What You Will Build

1. Pair counter — scan a corpus and count adjacent byte pairs
2. Merge step — replace the most frequent pair with a new token
3. Training loop — repeat merges until you reach a target vocabulary size
4. Encoder — given a trained tokenizer, split new text into tokens by replaying merge rules in learned order
5. Decoder — convert a token ID sequence back to the original text

Why This Matters

Writing the algorithm yourself reveals the mechanics that are easy to miss when reading about them:

• Merge order is the encoding function. When you implement the encoder, you will see that applying merge rules in the order they were learned is what produces deterministic tokenization.
• Ties exist. On small corpora, you will observe multiple pairs with the same count — and see how your tiebreaker affects the resulting vocabulary.
• Encoding is not “longest match.” Your encoder applies rules sequentially, not greedily. The result often looks like longest match but the mechanism is different.

Inspiration

This exercise is inspired by minbpe by Andrej Karpathy — a minimal, clean implementation of BPE in Python. If you want to go deeper after finishing this notebook, minbpe is an excellent next step.

Reference

• Lecture 4 — Tokenization: How Machines Read
• Chapter 11 — Byte Pair Encoding