Understanding Deep Learning Requires Rethinking Generalization
Standard networks can perfectly fit training sets with random labels, so classical capacity measures cannot explain why they generalise on real labels. The experiments separate what regularisation actually does from what theory assumed it did.
Key ideas
- Memorisation capacity is not the constraint
- Explicit regularisers help but are not necessary for generalisation
- The optimiser and the data, not just the model class, decide what is learned
Why read it now
It set the agenda for generalisation research and is the fastest way to unlearn the textbook bias-variance picture.
Question to keep in mind
If the network can memorise noise, what property of real data lets the same network find a generalising solution first?