A Mathematical Theory of Communication
Shannon defines information as the reduction of uncertainty, measures it with entropy, and proves that a channel has a capacity that cannot be exceeded but can be approached with suitable codes. Cross-entropy loss, perplexity, and bits-per-token all come straight from here.
Key ideas
- Entropy is the expected surprise of a source
- A model that assigns probabilities is also a compression scheme
- Channel capacity bounds what any code can achieve
Why read it now
Every loss function in the track is an information quantity. Reading the source makes perplexity and KL divergence feel obvious rather than memorised.
Question to keep in mind
Why is minimising cross-entropy the same as building the best compressor for the training data?