MapReduce: Simplified Data Processing on Large Clusters
Two user-supplied functions, map and reduce, are enough to express many large-scale batch computations, and the framework handles partitioning, scheduling, and re-running failed tasks. The insight is that restricting the programming model makes fault tolerance automatic.
Key ideas
- A restricted model makes retries and stragglers easy to handle
- Move computation to the data, not data to the computation
- Deterministic tasks can be re-executed anywhere
Why read it now
Counting, aggregation, and analytics pipelines in this course are all descendants of this model, whether they run on Hadoop, Spark, or a warehouse.
Question to keep in mind
Why must map and reduce functions be deterministic for the fault-tolerance story to hold?