Interview/Spark

Spark interview questions: What a shuffle actually does

Interactive Spark interview questions on What a shuffle actually does. Practice with the matching lesson. A shuffle moves records so equal keys co-locate. It is disk + network + a stage boundary.

Lesson · Simulation

What does a shuffle do?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

What a shuffle actually does

groupBy region takes 50 minutes on 80 GB

Symptoms

  • One region is 70% of rows
  • Rest of tasks finish in 2 minutes

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

What does a shuffle do?

What a shuffle actually does · tap to open the answer

Say this first: Moves records so equal keys land on the same reducer. Disk + network + a stage boundary.

Why: Map tasks write shuffle files. Reducers fetch their slice. groupBy, distinct, join (non-broadcast), and repartition all shuffle.

Common mistake: Thinking shuffle is 'Spark being slow' rather than a specific Exchange.

Follow-up they will ask: Which Spark UI numbers prove a shuffle?

Lesson · Simulation

intermediate

Why can a shuffle take longer than the compute?

What a shuffle actually does · tap to open the answer

Say this first: Network fetch, disk spill, and waiting on the slowest reducer.

Why: Shuffle write/read bytes, fetch wait, spill. A skewed key makes one reducer read most of the data. Compression and AQE help only if the plan is sane.

Common mistake: Adding CPU cores when fetch wait is the bottleneck.

Follow-up they will ask: What is a shuffle block?

Lesson · Simulation

senior

How do you remove a shuffle from a join?

What a shuffle actually does · tap to open the answer

Say this first: Broadcast the small side, or co-partition/bucket both sides on the join key.

Why: BroadcastHashJoin keeps the fact table in place. SortMergeJoin shuffles both unless already partitioned the same way. Prove with Exchange absence in explain().

Senior depth: When both sides are large and unsorted. Then make it balanced — not broadcast a 30 GB dimension.

Common mistake: spark.sql.shuffle.partitions as the join strategy.

Follow-up they will ask: When is a shuffle the right answer?

Lesson · Simulation

intermediate

A stage reads a 40 GB table and the Spark UI reports 300 GB of Shuffle Write. How is the shuffle bigger than the input?

What a shuffle actually does · tap to open the answer

Say this first: Shuffle Write measures the serialized rows leaving the map side after your transformations, not the bytes you read.

Remember these

  • Shuffle Write is serialized post-map output, not input bytes
  • Compare Shuffle Write Records to Input Records to separate width from multiplication
  • collect_list and exact distinct skip map-side partial aggregation
  • Projecting away unused columns before the Exchange shrinks the shuffle directly

Why: Three things commonly inflate it. First, Parquet input is compressed and column-pruned on disk, while shuffle rows are full materialized records, so decompressed-and-widened data is simply bigger. Second, anything that multiplies rows before the Exchange — an explode, a many-to-many join, a cross-product of flags — ships the multiplied result. Third, aggregations that cannot do a map-side partial merge, like collect_list or an exact count distinct over a high-cardinality key, send every raw row to the reducer instead of a partial aggregate. Compare the stage's Input Size / Records with its Shuffle Write Records: if the record count grew, you have a row-multiplication problem; if only the bytes grew, you have a width and serialization problem and the fix is to select fewer columns before the Exchange.

(orders.select("customer_id", "amount")   # drop 30 unused columns first
   .groupBy("customer_id")
   .sum("amount"))

Common mistake: Comparing Shuffle Write to the compressed table size on storage and concluding Spark is broken.

Follow-up they will ask: Which aggregate functions can do a map-side partial merge and which cannot?

Lesson · Simulation

advanced

A task fails with FetchFailedException. Whose failure is that, and what does Spark do next?

What a shuffle actually does · tap to open the answer

Say this first: A reducer could not fetch map output, so Spark treats the map stage as lost and reruns the missing map tasks before retrying the reduce stage.

Remember these

  • The exception surfaces on the reducer but the cause sits on the map side
  • Spark reruns the missing map tasks, then retries the reduce stage
  • spark.stage.maxConsecutiveAttempts bounds the retry loop
  • External shuffle service keeps map output alive after executor loss

Why: FetchFailedException is raised on the reduce side but the cause is almost always on the map side: the executor that wrote those shuffle blocks is gone, its local disk filled or failed, the node was decommissioned or reclaimed as a spot instance, or the fetch timed out under load. The DAGScheduler responds by marking the map stage's output for that executor as unavailable, resubmitting only the missing map tasks, and then rerunning the reduce stage — which is why you see a stage 'retry 1' with far fewer tasks than the original. This repeats up to spark.stage.maxConsecutiveAttempts before the job fails outright. Running an external shuffle service decouples shuffle files from executor lifetime so losing an executor no longer loses its map output, and on Kubernetes where there is no shuffle service the equivalent is dynamic allocation shuffle tracking, which keeps executors alive while their shuffle data is still referenced.

Senior depth: Keep the driver and a core group of executors on on-demand capacity and let only the expansion capacity be preemptible, enable the shuffle service so a reclaimed node does not drag its map output with it, and shorten the shuffle stages — the less map output alive at any moment, the less a reclaim costs. Raising retry counts only makes the failure take longer.

Common mistake: Raising network timeouts first, when the executor that held the blocks is simply gone.

Follow-up they will ask: You are on spot instances and this happens nightly. What is your first structural change?

Lesson · Simulation

advanced

One task shows Spill (Memory) of 40 GB and Spill (Disk) of 6 GB. Which number should worry you, and what do they actually measure?

What a shuffle actually does · tap to open the answer

Say this first: They are the same data measured twice — deserialized in memory versus serialized on disk — and the disk number is the IO you actually pay for.

Remember these

  • Spill Memory is deserialized size; Spill Disk is serialized bytes written
  • Their ratio is the serialization factor, not an error
  • Disk spill is the number that costs wall clock through write, read, and merge
  • Spill on one task only is a skew signal, not a memory-config signal

Why: When a task's execution memory fills, the sorter serializes its in-memory records and writes them to local disk. Spill (Memory) is the size of those records as live JVM objects before the write; Spill (Disk) is the compressed, serialized bytes that landed. The ratio between them is your serialization and compression factor, usually somewhere around 5-10x for typical row shapes, so a wide gap is normal and not itself a bug. The number that costs wall clock is the disk one, because it is real write-then-read IO plus the merge pass afterwards. The real signal, though, is that spill is happening at all on one task: it means that partition does not fit in the execution memory available per task, so the fix is more partitions, less data per key, or fewer concurrent tasks per executor — not a bigger spill buffer.

Senior depth: Under unified memory management, execution can evict cached storage blocks down to the storage reservation, but storage cannot evict execution. So a job that both caches aggressively and shuffles heavily will quietly lose its cache first and start recomputing — which looks like a mysterious slowdown rather than a memory error. Check the Storage tab fraction-cached alongside the spill metrics before changing any memory fraction.

Common mistake: Reporting the memory spill number as 'we need 40 GB more executor memory'.

Follow-up they will ask: Execution memory and storage memory compete. Which one gives way first?

Lesson · Simulation

senior

How does the external shuffle service change what dynamic allocation is allowed to do?

What a shuffle actually does · tap to open the answer

Say this first: It serves shuffle blocks from a process outside the executor, so idle executors can be released without destroying their map output.

Remember these

  • Shuffle files outlive the executor only when a separate daemon serves them
  • Without it, scaling down an executor invalidates its map output
  • spark.shuffle.service.enabled is the YARN-era companion to dynamic allocation
  • shuffleTracking is the K8s alternative and limits how far you can scale down

Why: Without it, an executor is the only thing that can serve the shuffle files it wrote, so releasing an idle executor means any reducer that later needs those blocks hits FetchFailedException and triggers a map-stage rerun. The external shuffle service runs one long-lived daemon per node, holding the shuffle directories, so scaling down is safe and spark.dynamicAllocation.enabled can actually shrink the cluster between stages. That is why spark.shuffle.service.enabled and dynamic allocation are historically mentioned in the same breath on YARN. On Kubernetes there is typically no node-level daemon, so the equivalent is spark.dynamicAllocation.shuffleTracking.enabled, which simply refuses to release an executor while its shuffle output is still referenced — safer, but it means heavy-shuffle jobs scale down much less than you would hope.

Senior depth: Shuffle tracking is holding them. Those executors still own shuffle blocks that a later stage may fetch, so the allocation manager will not release them. The real fix is structural — fewer or smaller shuffles, or an intermediate table written between phases so the shuffle data stops being live — rather than forcing the timeout down and reintroducing fetch failures.

Common mistake: Enabling dynamic allocation on a shuffle-heavy job with no shuffle service and then blaming the fetch failures on the network.

Follow-up they will ask: Your K8s job never scales below 40 executors despite long idle gaps. Why?

Lesson · Simulation

architect

One job reads a 3 TB fact table and groups it three times at three different grains. How do you cut the shuffle cost?

What a shuffle actually does · tap to open the answer

Say this first: Shuffle once to the finest grain you need, materialize that, and roll the coarser grains up from the small intermediate.

Remember these

  • Nested grains let one fine aggregate serve every coarser roll-up
  • sum, count, min, max compose; average composes only as sum plus count
  • Exact distinct does not roll up; approximate sketches do
  • A materialized fine-grained aggregate becomes a reusable table, not just a tuning hack

Why: Three independent groupBys mean three Exchanges that each move most of the 3 TB. If the three grains are nested — say customer-day, customer-month, and region-month — then the customer-day aggregate is a sufficient statistic for the other two, and the two roll-ups shuffle gigabytes instead of terabytes. This works cleanly for additive measures: sum, count, min, and max all roll up correctly. It breaks for averages unless you carry sum and count separately and divide at the end, and it breaks outright for exact distinct counts, which are not composable — those need either a separate pass or an approximate sketch like approx_count_distinct that does compose. Materializing the fine-grained aggregate as a real table also turns a tuning fix into a reusable asset, since the next team that needs a new grain rolls up from it rather than re-reading the fact.

daily = fact.groupBy("customer_id", "event_date").agg(F.sum("amount").alias("amt"), F.count("*").alias("n"))
daily.write.mode("overwrite").saveAsTable("curated.customer_daily")
monthly = daily.groupBy("customer_id", F.trunc("event_date", "MM")).agg(F.sum("amt"), F.sum("n"))

Senior depth: Then pick the single finest common grain that all three can be derived from, even if no consumer asks for it directly, and accept that it is larger than any individual output. If even that is impractical, cache or checkpoint the shared upstream so the three branches at least stop re-reading the fact table, and let each pay its own Exchange from there.

Common mistake: Rolling up an average of averages and shipping a number that is quietly wrong.

Follow-up they will ask: What do you do when the three grains are not nested at all?

Lesson · Simulation