Papers
36 landmark papers in three tracks, sequenced so each one prepares you for the next. Read in order, write a summary in your own words, and mark what you finish.
AI research
From the first deep convolutional net to preference optimisation, in the order the ideas were needed.
- 01
ImageNet Classification with Deep Convolutional Neural Networks
This is the paper that made scale plus GPUs the default strategy. Read it to see how few ingredients the modern deep learning stack started with.
- 02
Adam: A Method for Stochastic Optimization
Almost every model you will read about after this was trained with Adam or a close variant. Understanding the two moment estimates explains most optimiser folklore.
- 03
Deep Residual Learning for Image Recognition
The skip connection is the single most reused architectural idea in the field. Transformers, diffusion U-Nets, and modern vision models all depend on it.
- 04
Attention Is All You Need
This is the architecture the rest of the track builds on. Read it slowly; the scaled dot-product attention equation is the one formula worth memorising.
- 05
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
It established the pre-train then fine-tune workflow that dominated applied NLP for years and still shapes how embeddings are built.
- 06
Language Models are Few-Shot Learners
This is where prompting became an interface. Read it to see what few-shot behaviour looked like before instruction tuning made it feel ordinary.
- 07
Scaling Laws for Neural Language Models
Every later decision about model size and training length refers back to these curves, including the correction Chinchilla made to them.
- 08
Training Compute-Optimal Large Language Models
It corrected the field's most-cited scaling result and explains why later models are smaller and trained far longer.
- 09
Training language models to follow instructions with human feedback
This three-stage recipe is what turned raw language models into assistants, and it is the reference point for every later alignment method.
- 10
LoRA: Low-Rank Adaptation of Large Language Models
It made fine-tuning large models practical on ordinary hardware and is the basis of most adapter ecosystems today.
- 11
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
It is the clearest example of treating a model as a systems problem. Longer contexts and cheaper training both trace back to this kernel.
- 12
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
It replaced a fragile RL pipeline with one loss function and is now the default starting point for preference tuning.
Systems
The papers behind the storage, coordination, and streaming systems this course designs.
- 01
Time, Clocks, and the Ordering of Events in a Distributed System
Every discussion of consistency, causality, and versioning in this course stands on this paper. It is short and every paragraph earns its place.
- 02
The Log-Structured Merge-Tree (LSM-Tree)
RocksDB, Cassandra, and most modern write-heavy stores are LSM-trees. The course's storage-engine module builds one from scratch.
- 03
The Google File System
It is the template for object and blob storage: a metadata service, dumb data nodes, and replication chosen for the workload rather than for generality.
- 04
MapReduce: Simplified Data Processing on Large Clusters
Counting, aggregation, and analytics pipelines in this course are all descendants of this model, whether they run on Hadoop, Spark, or a warehouse.
- 05
Bigtable: A Distributed Storage System for Structured Data
It defined the wide-column model that HBase and Cassandra copied, and shows how range partitioning and sorted storage fit together.
- 06
Dynamo: Amazon's Highly Available Key-value Store
It is the origin of most techniques in the NoSQL toolbox and the clearest worked example of a deliberate consistency trade-off.
- 07
Paxos Made Simple
Consensus is the foundation of every coordination service and replicated log. Read this before Raft so you can see what Raft was simplifying.
- 08
The Chubby Lock Service for Loosely-Coupled Distributed Systems
It explains why coordination is offered as a service rather than a library, and it shaped ZooKeeper and etcd.
- 09
ZooKeeper: Wait-free Coordination for Internet-scale Systems
Kafka, HBase, and many in-house systems used it for membership and leader election, and its recipes are still how coordination problems are decomposed.
- 10
In Search of an Understandable Consensus Algorithm
etcd, CockroachDB, and most new replicated logs implement Raft. This is the version of consensus you will actually read code for.
- 11
Spanner: Google's Globally-Distributed Database
It shows what it costs to get strong consistency at global scale, and why bounded clock error is a systems property worth paying for.
- 12
Kafka: a Distributed Messaging System for Log Processing
The partitioned log is the backbone of event-driven architectures. The course's queue and stream modules design one from this paper's ideas.
ML math
The results that explain why training works, from information theory to double descent.
- 01
A Mathematical Theory of Communication
Every loss function in the track is an information quantity. Reading the source makes perplexity and KL divergence feel obvious rather than memorised.
- 02
Learning representations by back-propagating errors
Automatic differentiation frameworks are this algorithm generalised. Understanding the chain of local derivatives explains vanishing gradients and why depth was hard.
- 03
Gradient-Based Learning Applied to Document Recognition
Convolution as weight sharing is the mathematical idea behind every vision backbone. Reading the original makes the parameter counting concrete.
- 04
Long Short-Term Memory
Gating is the idea to extract: multiplicative control over what is kept and what is forgotten. It reappears in GRUs, highway networks, and attention itself.
- 05
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
It is the cleanest example of a regulariser with both an intuitive story and an ensemble interpretation, and it is still in most Transformer blocks.
- 06
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Normalisation layers are everywhere; this paper shows the mechanics, including the train-time versus test-time statistics gap that still causes bugs.
- 07
Auto-Encoding Variational Bayes
The evidence lower bound and the reparameterisation trick are the two tools that connect probability to backpropagation; diffusion models are their descendants.
- 08
Generative Adversarial Nets
Adversarial training reframed generation as a game and introduced instabilities that a decade of papers then tried to tame. It is short and worth reading for the proof alone.
- 09
Understanding Deep Learning Requires Rethinking Generalization
It set the agenda for generalisation research and is the fastest way to unlearn the textbook bias-variance picture.
- 10
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
It links initialisation, pruning, and why big models train more easily than small ones, which matters for both theory and deployment.
- 11
Deep Double Descent: Where Bigger Models and More Data Hurt
It explains behaviour that contradicts the classical U-shaped curve and informs how to read scaling results honestly.
- 12
Denoising Diffusion Probabilistic Models
Diffusion is the dominant generative approach for images and audio, and its derivation ties together the VAE bound, score matching, and Langevin dynamics.