Kafka: a Distributed Messaging System for Log Processing
Kafka treats a message queue as a partitioned, append-only log that consumers read at their own offsets. Sequential disk IO, batching, and leaving position tracking to consumers gave it throughput that traditional brokers could not match.
Key ideas
- A log is the simplest durable queue: append, then read by offset
- Partitions give parallelism and ordering within a key
- Consumers own their offsets, so replay is free
Why read it now
The partitioned log is the backbone of event-driven architectures. The course's queue and stream modules design one from this paper's ideas.
Question to keep in mind
What ordering guarantee does a partitioned log actually give, and what does that mean for a consumer that needs global order?