ImageNet Classification with Deep Convolutional Neural Networks
A deep convolutional network trained on two GPUs cut the ImageNet top-5 error rate far below the best hand-engineered pipelines. The recipe was mostly scale: more layers, more data, ReLU activations for faster training, dropout against overfitting, and aggressive data augmentation.
Key ideas
- ReLU units train several times faster than saturating activations
- Dropout on the fully connected layers was the main regulariser
- Splitting one model across two GPUs was a memory workaround that later became standard
Why read it now
This is the paper that made scale plus GPUs the default strategy. Read it to see how few ingredients the modern deep learning stack started with.
Question to keep in mind
Which of the tricks in this paper are still in every training run today, and which have quietly disappeared?