What happens when Spark executes a job?
One action, four-hour wall clock
Symptoms
- Single job in Spark UI
- Stage 2 is 3.5 hours
- Stage 0–1 were minutes
What happens when Spark executes a job? interview questions
Interactive Spark interview questions on What happens when Spark executes a job?. Same topic as /learn/spark/execution-flow. Five short labs that trace groupBy + show() from lazy code through the plan, job, stages, tasks, shuffle, and result.
Question 1 of 3
Walk a Spark action from code to rows without skipping a layer.
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWalk a Spark action from code to rows without skipping a layer.
What happens when Spark executes a job? · tap to open the answer
Short: Code → logical plan → job → stages → tasks → (optional shuffle) → result.
Detailed: Transformations are lazy. The action builds a plan, splits it at shuffles into stages, runs one task per partition, then returns a small result or writes files.
Common mistake: Saying 'Spark runs the DataFrame line by line'.
Follow-up: Where does a shuffle sit in that chain?
intermediateWhy does one action create multiple stages?
What happens when Spark executes a job? · tap to open the answer
Short: Each wide transformation (shuffle) is a stage boundary.
Detailed: Narrow steps (filter, map, broadcast join) stay in one stage. groupBy, join-without-broadcast, repartition cut a new stage because data must move.
Common mistake: Counting jobs instead of stages when debugging runtime.
Follow-up: How do you see stage boundaries in the physical plan?
architectA dashboard job has 1 job, 4 stages, 80k tasks. What do you inspect first?
What happens when Spark executes a job? · tap to open the answer
Short: Task count and the slowest stage — not executor count.
Detailed: 80k tasks usually means tiny files or over-partitioning. Stage time is max(task). Find the stage with shuffle or scan skew, then file layout, then AQE/coalesce.
Senior: Hundreds to low thousands of healthy tasks, not tens of thousands of 50 ms tasks.
Common mistake: Doubling the cluster because 'there are many tasks'.
Follow-up: What number of tasks would you aim for on a 1 TB nightly?