Scaling Laws for Neural Language Models
Language model loss follows smooth power laws in parameters, data, and compute over many orders of magnitude, and architecture details matter far less than scale. The paper turned model planning into arithmetic: given a compute budget, the curves say how big to go.
Key ideas
- Loss is predictable from scale across seven orders of magnitude
- Larger models are more sample-efficient
- Early stopping well before convergence is compute-optimal
Why read it now
Every later decision about model size and training length refers back to these curves, including the correction Chinchilla made to them.
Question to keep in mind
Which of the three scaling variables does the paper recommend growing fastest, and why did that later turn out to be wrong?