Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Randomly dropping units during training forces the network to learn redundant, robust features, and at test time the full network approximates an average over exponentially many thinned networks. The paper reports consistent gains across vision, speech, and text tasks.
Key ideas
- Noise during training discourages co-adaptation of features
- Weight scaling at test time approximates the model average
- The right drop rate depends on layer width and data size
Why read it now
It is the cleanest example of a regulariser with both an intuitive story and an ensemble interpretation, and it is still in most Transformer blocks.
Question to keep in mind
Why does dropout hurt when the model is already underfitting, and how would you detect that situation?