Deep Double Descent: Where Bigger Models and More Data Hurt
Test error can rise and then fall again as model size, training time, or data increase, with a peak near the point where the model just fits the training set. The paper unifies these under an effective model complexity and shows the effect across architectures.
Key ideas
- The interpolation threshold is where things get worst, not best
- Label noise sharpens the peak
- More data can hurt a model sitting at the threshold
Why read it now
It explains behaviour that contradicts the classical U-shaped curve and informs how to read scaling results honestly.
Question to keep in mind
What would you change in an experiment to tell double descent apart from ordinary overfitting?