Job to Stages — map side vs reduce side
Stage 3 is always the SLA miss
Symptoms
- Stages 0–2 healthy
- Stage 3 task max is 18× median
Job to Stages — map side vs reduce side interview questions
Interactive Spark interview questions on Job to Stages — map side vs reduce side. Same topic as /learn/spark/job-to-stages. A shuffle edge splits Job 0 into Stage 0 (map, shuffle write) and a Stage 1 that cannot start until the write finishes.
Question 1 of 3
What splits a job into stages?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat splits a job into stages?
Job to Stages — map side vs reduce side · tap to open the answer
Short: A shuffle — a wide dependency.
Detailed: Narrow transformations pipeline inside a stage. When keys must co-locate (groupBy, sort merge join, repartition), Spark inserts an Exchange and a new stage.
Common mistake: Saying each transformation is its own stage.
Follow-up: Is a filter a new stage?
intermediateStage time is 20 minutes but most tasks finished in 30 seconds. What is the stage time?
Job to Stages — map side vs reduce side · tap to open the answer
Short: The slowest task — the straggler.
Detailed: Stages wait for all tasks. Median is a vanity metric. Read max task duration, spill, and shuffle read on that task.
Common mistake: Optimizing average task time while one key holds the stage.
Follow-up: Which UI chart shows this immediately?
seniorYou see two Exchanges for one join. Is that always wrong?
Job to Stages — map side vs reduce side · tap to open the answer
Short: Not always — both sides may need a shuffle for sort-merge. A broadcast join has zero Exchanges for the big side.
Detailed: SMJ typically shuffles both inputs unless already co-partitioned. BroadcastHashJoin shuffles neither of the fact side. Prove with explain().
Senior: Broadcast the small side, or bucket/co-partition both tables on the join key.
Common mistake: Removing 'an Exchange' without knowing join strategy.
Follow-up: How would you eliminate both shuffles?