Language Models are Few-Shot Learners
A 175-billion-parameter decoder-only Transformer, trained only to predict the next token, could perform new tasks from a handful of examples placed in the prompt. The paper measured how this in-context ability grows with model size and set the template for evaluating large language models.
Key ideas
- In-context learning emerges with scale without any gradient updates
- Zero-, one-, and few-shot settings measure the same capability at different strengths
- Data contamination and evaluation hygiene were already a concern
Why read it now
This is where prompting became an interface. Read it to see what few-shot behaviour looked like before instruction tuning made it feel ordinary.
Question to keep in mind
What evidence in the paper separates genuine in-context learning from memorised training data?