What a shuffle actually does
groupBy region takes 50 minutes on 80 GB
Symptoms
- One region is 70% of rows
- Rest of tasks finish in 2 minutes
What a shuffle actually does interview questions
Interactive Spark interview questions on What a shuffle actually does. Same topic as /learn/spark/shuffle. A shuffle moves records so equal keys co-locate. It is disk + network + a stage boundary.
Question 1 of 3
beginner
What a shuffle actually doesWhat does a shuffle do?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat does a shuffle do?
What a shuffle actually does · tap to open the answer
Short: Moves records so equal keys land on the same reducer. Disk + network + a stage boundary.
Detailed: Map tasks write shuffle files. Reducers fetch their slice. groupBy, distinct, join (non-broadcast), and repartition all shuffle.
Common mistake: Thinking shuffle is 'Spark being slow' rather than a specific Exchange.
Follow-up: Which Spark UI numbers prove a shuffle?
intermediateWhy can a shuffle take longer than the compute?
What a shuffle actually does · tap to open the answer
Short: Network fetch, disk spill, and waiting on the slowest reducer.
Detailed: Shuffle write/read bytes, fetch wait, spill. A skewed key makes one reducer read most of the data. Compression and AQE help only if the plan is sane.
Common mistake: Adding CPU cores when fetch wait is the bottleneck.
Follow-up: What is a shuffle block?
seniorHow do you remove a shuffle from a join?
What a shuffle actually does · tap to open the answer
Short: Broadcast the small side, or co-partition/bucket both sides on the join key.
Detailed: BroadcastHashJoin keeps the fact table in place. SortMergeJoin shuffles both unless already partitioned the same way. Prove with Exchange absence in explain().
Senior: When both sides are large and unsorted. Then make it balanced — not broadcast a 30 GB dimension.
Common mistake: spark.sql.shuffle.partitions as the join strategy.
Follow-up: When is a shuffle the right answer?