DataYatriSpark, seen under the hood
Simulate
spark/core
intermediate
Connecting…

What a shuffle actually does

A shuffle moves records so equal keys co-locate. It is disk + network + a stage boundary.

Lesson 13 of 29 · Spark path

Explain it at my level

  1. Map write
  2. Hash
  3. Network
  4. Reduce
Watch the canvas:map partitionA, B → reducer 0C, D → reducer 1Live simulation
Map side — mixed keysP0map partitionACP1map partitionBAP2map partitionDCP3map partitionBDACBADCBDReducer 0waitingReducer 1waitinghash(region) % 2 · A,B → R0 C,D → R1 · still on diskNetwork + disk + serialization — this is the billComplete keys. Aggregation is now a local CPU loop.
Watch the keys move. Shuffle exists only because they start mixed.