Long Short-Term Memory
Recurrent networks could not learn long dependencies because gradients through time either vanish or explode. The LSTM adds a memory cell with gates that let error flow unchanged across many steps, so information can be stored and retrieved over long spans.
Key ideas
- Constant error flow through a linear cell state avoids vanishing gradients
- Input and output gates decide when to write and read the cell
- Truncated gradients kept training tractable on 1997 hardware
Why read it now
Gating is the idea to extract: multiplicative control over what is kept and what is forgotten. It reappears in GRUs, highway networks, and attention itself.
Question to keep in mind
Why does a gate implemented as a multiplication protect the gradient better than a carefully chosen activation function?