spark/core
intermediate
Connecting…What a shuffle actually does
A shuffle moves records so equal keys co-locate. It is disk + network + a stage boundary.
Explain it at my level
- Map write
- Hash
- Network
- Reduce
Step 1 of 4
Map-side partitions
P0 · A,C250 MB
P1 · B,A240 MB
P2 · D,C250 MB
P3 · B,D240 MB