Attention Is All You Need
The Transformer replaced recurrence entirely with self-attention, so every position can look at every other position in one step and the whole sequence can be processed in parallel. Multi-head attention, positional encodings, and the encoder-decoder layout defined the architecture behind every modern language model.
Key ideas
- Attention is a soft lookup: queries score keys, scores weight values
- Multiple heads let the model attend to different relations at once
- Without recurrence, position has to be injected explicitly
Why read it now
This is the architecture the rest of the track builds on. Read it slowly; the scaled dot-product attention equation is the one formula worth memorising.
Question to keep in mind
Where does the quadratic cost come from, and what would you have to give up to remove it?