Training Compute-Optimal Large Language Models
Re-running the scaling analysis with learning-rate schedules matched to each run showed that parameters and training tokens should grow together, roughly one to one. A 70-billion-parameter model trained on four times more data beat a 280-billion-parameter model trained on the same compute.
Key ideas
- Compute-optimal training doubles data whenever it doubles parameters
- Most earlier large models were undertrained
- Inference cost favours smaller models trained longer
Why read it now
It corrected the field's most-cited scaling result and explains why later models are smaller and trained far longer.
Question to keep in mind
If you had a fixed budget for training and a much larger budget for inference, how would you deviate from the compute-optimal point?