The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Dense networks contain sparse subnetworks that, when trained in isolation from their original initialisation, match the full network's accuracy. The paper finds them by iterative pruning and reset, and argues that overparameterisation helps by including many such tickets.
Key ideas
- The winning subnetwork depends on its initial weights, not just its structure
- Iterative magnitude pruning finds much sparser tickets than one-shot pruning
- Overparameterisation is a search advantage
Why read it now
It links initialisation, pruning, and why big models train more easily than small ones, which matters for both theory and deployment.
Question to keep in mind
Why does re-initialising the pruned subnetwork randomly destroy the effect?