05 Notebook: NumPy for Transformers

April 2026 (201 Words, 2 Minutes)

Goal

Build a working scaled dot-product attention from scratch in pure NumPy. The notebook is also a NumPy crash course — if you have never typed np.zeros before, the first six subsections walk you in from the start. By the end you have a 12-line attention function plus a multi-head wrapper plus a feed-forward network.

Open the Notebook

Open in JupyterLite — runs directly in your browser, no installation required.

The notebook uses only NumPy and matplotlib, both pre-installed in JupyterLite. Your progress is saved in your browser; export the notebook if you want a permanent copy.

What You Will Do

1. NumPy primer — array creation (np.zeros, np.eye, np.arange, np.diag), slicing 1-D and 2-D, the difference between A @ B and A * B
2. Implement dot_product and my_matmul — two stub-and-test pairs that build the atomic operations attention runs on
3. Embedding lookup with array indexing — turn token IDs into vectors via E[token_ids]
4. Broadcasting and the axis= argument — a per-row LayerNorm in three lines
5. Implement layernorm as a function — wrap LN for reuse
6. Implement split_heads and merge_heads — the multi-head reshape/transpose dance
7. Implement causal_mask — foreshadows the next notebook
8. Capstone — reproduce the lecture’s [12%, 40%, 16%, 11%, 20%] cat-sat-on-it weights, then run multi-head attention on the dragon-drinks toy
9. Implement my_gelu before adding an FFN to your block

Deliverables (AI Diary entry — notebook §12)

• The shapes of Q, K, V, S, A and the attention output for the dragon-drinks toy
• What happens to softmax when you remove the √d_k scaling, and why the lecture calls this saturation
• Multi-head parameter accounting — does your number match drill chapter 6 of the markdown drill book?
• Why FFN is “embarrassingly parallel” across tokens but attention is not
• One prediction for what notebook 06 will show about permutation equivariance, causal masking, prefix invariance, or positional encodings

Reference

• Lecture 5 — Inside the Transformer
• Chapter 7 — Understanding Attention
• Chapter 9 — Multi-Head Attention
• Companion drill book: prompts/lecture-ideas/lecture_5/exercises/basic_matrix_drills.md