video

Andrej Karpathy: Let's Build GPT: From Scratch, in Code, Spelled Out

by Andrej Karpathy

Karpathy writes a working GPT from an empty notebook in two hours — character-level tokenizer, self-attention, multi-head attention, residual connections, layer norm — training it on Shakespeare until it produces plausible pseudo-Elizabethan text. Assumes some Python and PyTorch, and repays the effort with an understanding of transformers that no diagram can give you.