MapReduce and the Spark Model
How MapReduce splits work into map, shuffle and reduce, shown with a word count you can run in plain Python, then how Spark keeps the same idea but holds data in memory and builds a plan lazily.
MapReduce processes data too large for one machine by running a map function on each chunk in parallel, grouping the emitted pairs by key in a shuffle, and running a reduce function per key. Interviewers expect the comparison: Spark keeps the same model but holds intermediate data in memory, builds a lazy plan, and recomputes lost partitions from lineage.
MapReduce processes data too large for one machine by running a map function on every chunk in parallel, grouping the results by key (the shuffle), and running a reduce function per key. Spark keeps that model but holds intermediate data in memory and builds a lazy execution plan, which is why it is much faster for multi-step jobs.
- MapReduce, introduced in a Google paper in 2004, runs a job as map in parallel, then a shuffle that groups pairs by key, then reduce in parallel.
- The shuffle is the most expensive MapReduce step, because all values for one key must travel over the network to the same reducer.
- A MapReduce combiner pre-aggregates map output locally before the shuffle and works only for associative, commutative operations such as sum or max.
- Classic Hadoop MapReduce writes each job's output to HDFS before the next job reads it, so disk traffic dominates multi-step pipelines.
- Spark transformations such as map, filter and groupBy are lazy and run only when an action such as count or collect asks for a result.
- Spark recovers a lost partition by recomputing it from its lineage, the recorded list of steps that produced it, rather than restoring a copy.
01The problem it solves
When a dataset is spread across many machines, you cannot just loop over it. You need a way to run the same logic on each piece, then combine the pieces, without every programmer handling network failures and machine crashes by hand.
MapReduce, introduced in a Google paper in 2004, fixed that by restricting the program shape. You write two functions and the framework does the rest: splitting the input, scheduling, moving data between machines and re-running failed tasks. The restriction is what makes the automation possible.
02Map: emit key-value pairs
The map function sees one record at a time and knows nothing about the others, so thousands of copies can run in parallel on different chunks. For word count, each word becomes the pair (word, 1).
docs = ["the cat sat", "the dog sat", "the cat ran"]
def map_fn(doc):
for w in doc.split():
yield (w, 1)
pairs = [kv for d in docs for kv in map_fn(d)]
print(pairs[:4], len(pairs))This prints [('the', 1), ('cat', 1), ('sat', 1), ('the', 1)] 9. Nine words in, nine pairs out.
03Shuffle: group by key
from collections import defaultdict
groups = defaultdict(list)
for k, v in pairs:
groups[k].append(v)
print(dict(groups))The result is {'the': [1, 1, 1], 'cat': [1, 1], 'sat': [1, 1], 'dog': [1], 'ran': [1]}. In a real cluster this is the expensive step, because pairs travel over the network so that all values for one key land on one machine. Which machine is chosen by a partitioner, typically a hash of the key modulo the number of reducers. The same key must always go to the same reducer, or the counts would be split.
04Reduce: combine the values
def reduce_fn(k, vals):
return (k, sum(vals))
print(sorted(reduce_fn(k, v) for k, v in groups.items()))The output is [('cat', 2), ('dog', 1), ('ran', 1), ('sat', 2), ('the', 3)]. Because each key is handled separately, the reduce calls also run in parallel across machines. The whole job is: map in parallel, shuffle, reduce in parallel.
05Combiners reduce network traffic
A combiner is a mini-reduce that runs on each mapper's own output. For the line "to be or not to be", the mapper would emit 6 pairs. After combining locally it sends only 4:
from collections import defaultdict
ps = list(map_fn("to be or not to be"))
c = defaultdict(int)
for k, v in ps:
c[k] += v
print(len(ps), len(c), dict(c)) # 6 4 {'to': 2, 'be': 2, 'or': 1, 'not': 1}On real text with many repeated words, the saving is large. It only works when the reduce operation is associative and commutative, like sum or max. An average cannot be combined this way directly. You would send (sum, count) pairs instead.
06Where MapReduce hurts and Spark helps
Classic Hadoop MapReduce writes the output of each job to disk (HDFS) before the next job reads it. For a pipeline of many steps, such as iterative machine learning, that disk traffic dominates the run time. Fault tolerance comes from replication of the stored data.
Spark keeps the same split-the-work idea but works with in-memory distributed datasets (RDDs, and the DataFrames built on them). Two ideas matter. Transformations such as map, filter and groupBy are lazy: Spark records the plan and runs nothing until an action such as count or collect asks for a result, which lets it optimise the whole plan. And lost partitions are recomputed from their lineage, the recorded list of steps that produced them, rather than restored from a copy.
We did not run Spark code in this article, so no numbers are quoted for it. The point to carry away is the model: a shuffle between stages is still the costly operation, and avoiding unnecessary shuffles is the main performance skill in Spark.
07Common interview questions
Q: What are the phases of MapReduce?
A: Map, shuffle and sort, then reduce.
Q: What is the shuffle?
A: The step that moves all values for the same key to the same reducer, usually across the network. It is the most expensive part.
Q: What does a combiner do?
A: It aggregates map output locally before the shuffle to reduce data transfer. It requires an associative, commutative operation.
Q: Why is Spark faster than Hadoop MapReduce for iterative work?
A: It keeps intermediate data in memory instead of writing to disk between stages, and it optimises a lazy plan.
Q: Transformation versus action in Spark?
A: Transformations build the plan and are lazy. Actions trigger execution and return a result.
Q: How does Spark recover from a lost partition?
A: By recomputing it from lineage.