Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Normalising each layer's inputs over the mini-batch, then letting the network learn a scale and shift, made deep networks train faster with larger learning rates and less careful initialisation. The original explanation, internal covariate shift, was later questioned, but the effect on the loss landscape is real.
Key ideas
- Normalise, then learn to undo it where useful
- Batch statistics at training time, running averages at inference
- Smoother optimisation, not covariate shift, is the better explanation
Why read it now
Normalisation layers are everywhere; this paper shows the mechanics, including the train-time versus test-time statistics gap that still causes bugs.
Question to keep in mind
What breaks when batch size is very small, and which normalisation variant fixes it?