Shuffle and Result — keys move, then we aggregate
Write succeeded, collect for 'QA' killed the job
Symptoms
- Delta table is fine
- QA cell collect()'d gold
Shuffle and Result — keys move, then we aggregate interview questions
Interactive Spark interview questions on Shuffle and Result — keys move, then we aggregate. Same topic as /learn/spark/shuffle-and-result. Hash partitioning, local shuffle write, network fetch, reduce tasks, and the rows that come back to show().
Question 1 of 3
What comes back to the driver after a shuffle?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat comes back to the driver after a shuffle?
Shuffle and Result — keys move, then we aggregate · tap to open the answer
Short: Only what the action asked for — a few show() rows, a count, or nothing for write.
Detailed: Shuffle moves data between executors. It does not dump the table on the driver. write() commits files; the driver stays small. collect() is the action that pulls.
Common mistake: Believing every shuffle returns the full dataset to the notebook.
Follow-up: Does saveAsTable send rows through the driver?
intermediateWhy is shuffle both a network problem and a disk problem?
Shuffle and Result — keys move, then we aggregate · tap to open the answer
Short: Map tasks write shuffle files locally; reducers fetch over the network; spill hits disk again if a reducer partition is fat.
Detailed: Spark UI: shuffle write, shuffle read, fetch wait, spill memory/disk. A healthy shuffle is balanced. A sick one has one reducer reading 10× bytes.
Common mistake: Only looking at CPU when a stage is an Exchange.
Follow-up: What is shuffle partition size aiming at?
seniorspark.sql.shuffle.partitions = 200 on a 4 TB agg. What goes wrong?
Shuffle and Result — keys move, then we aggregate · tap to open the answer
Short: Each reducer partition is huge — spill, OOM, or hour-long tasks.
Detailed: 200 is a default, not a law. Target ~128–256 MB shuffle partitions. AQE coalesce helps small data; large data needs more partitions or a better plan (pre-agg, broadcast).
Senior: It can coalesce empty/small partitions after seeing map-side sizes — it will not save you from one skewed key.
Common mistake: Leaving 200 forever because 'that's Spark'.
Follow-up: How does AQE change this number at runtime?