BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
BERT pre-trains a Transformer encoder on unlabeled text with a masked-token objective, then fine-tunes the same weights for each downstream task. Bidirectional context and a single shared model pushed nearly every language benchmark forward at once.
Key ideas
- Masked language modelling lets an encoder see both directions
- One pre-trained model, many small task heads
- Next-sentence prediction turned out to matter less than masking
Why read it now
It established the pre-train then fine-tune workflow that dominated applied NLP for years and still shapes how embeddings are built.
Question to keep in mind
Why can a decoder-only model not use the masked objective directly, and what does it use instead?