04 Notebook: Tokenizer Playground

April 2026 (141 Words, 1 Minutes)

Goal

Explore tokenizers empirically. Run this notebook during Lecture 4 as each section is introduced, or work through it as a complete lab afterwards. By the end, you will have verified every major tokenization phenomenon from the lecture by measuring it yourself.

Open the Notebook

Open in Google Colab — recommended, no setup required.

This notebook needs tiktoken and transformers, which have native C/Rust backends. Colab has them pre-installed. The notebook will not run in JupyterLite (browser-based Jupyter), because those extensions are not WebAssembly-compatible.

For LLaMA 3 and Gemma 2 tokenizer comparisons (Section 3), you additionally need:

1. A free HuggingFace account
2. Accepted licenses for Meta LLaMA 3 and Google Gemma 2

If you skip the HuggingFace login, those tokenizers are automatically skipped — the rest of the notebook still works.

What You Will Do

1. First contact — tokenize text with GPT-4o and see exactly where splits fall
2. Compare tokenizers — GPT-4o, GPT-4, LLaMA 3, Gemma 2 side by side
3. The multilingual gap — measure English vs German token cost
4. Arithmetic breakdown — see why 1234 + 5678 is hard
5. Code tokenization — indentation cost, naming conventions, boilerplate vs novel code, Java vs Python
6. Vocabulary archaeology — find the longest token, check for glitch tokens like SolidGoldMagikarp
7. Token budget estimation — how much of the context window does your CLAUDE.md consume?

Deliverables (AI Diary entry — notebook §9)

• At least 5 surprising tokenization results with explanations grounded in the lecture
• English-German token count comparison with calculated ratio
• Your CLAUDE.md token budget estimate — how much context window does it consume?
• One concrete change to your prompting approach based on what you now understand

Reference

• Lecture 4 — Tokenization: How Machines Read
• Chapter 25 of the lecture frames the notebook as the hands-on complement to the theory