Adam: A Method for Stochastic Optimization
Adam keeps running estimates of the gradient's mean and uncentred variance and uses them to give each parameter its own step size, with a bias correction for the early steps. It combined the strengths of momentum and RMSProp into one optimiser that works with little tuning.
Key ideas
- Per-parameter step sizes from first and second moment estimates
- Bias correction matters because the moving averages start at zero
- The default hyperparameters were chosen to be robust rather than optimal
Why read it now
Almost every model you will read about after this was trained with Adam or a close variant. Understanding the two moment estimates explains most optimiser folklore.
Question to keep in mind
Why does the second-moment estimate make the effective learning rate shrink for parameters with noisy gradients?