Learning representations by back-propagating errors
A three-page paper that popularised backpropagation: compute the error at the output, and push its derivative backwards through the network with the chain rule so every hidden weight gets a gradient. Hidden units then learn internal representations that were not hand-designed.
Key ideas
- The chain rule applied layer by layer gives every weight a gradient
- Hidden layers learn features rather than being programmed
- Gradient descent on a non-convex loss still finds useful minima
Why read it now
Automatic differentiation frameworks are this algorithm generalised. Understanding the chain of local derivatives explains vanishing gradients and why depth was hard.
Question to keep in mind
Where in the backward pass do gradients shrink, and what architectural choices later fixed that?