Jobs, stages, and tasks
On-call: 'the stage failed'
Symptoms
- They mean a task failed
- Retries then job fail
Jobs, stages, and tasks interview questions
Interactive Spark interview questions on Jobs, stages, and tasks. Same topic as /learn/spark/job-stage-task. An action creates a job. Shuffle boundaries split the job into stages. Each partition in a stage is a task.
Question 1 of 3
beginner
Jobs, stages, and tasksDefine job, stage, and task in one breath.
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerDefine job, stage, and task in one breath.
Jobs, stages, and tasks · tap to open the answer
Short: Job = one action. Stage = slice of the DAG between shuffles. Task = one partition in that stage.
Detailed: This is the vocabulary interviewers use to see if you've opened Spark UI. Mixing them is an instant no-hire for senior roles.
Common mistake: Calling an executor a stage.
Follow-up: How many tasks in a stage with 400 partitions?
intermediateA SQL query shows 3 jobs. Is that a bug?
Jobs, stages, and tasks · tap to open the answer
Short: Not necessarily — multiple actions, AQE, or Databricks SQL extra jobs.
Detailed: count + write is two jobs. AQE can add. Temporary views plus display add more. Map jobs to actions in the notebook.
Common mistake: Assuming one SQL string is always one job.
Follow-up: Where in Spark UI do you map SQL to jobs?
seniorHow do you explain a 1:N:M relationship on a whiteboard?
Jobs, stages, and tasks · tap to open the answer
Short: 1 action → N stages (shuffles + 1) → M tasks per stage (partitions).
Detailed: Draw: show() → Job 1 → Stage 0 scan/filter → Exchange → Stage 1 aggregate → few rows to driver. Tasks = partitions in each stage, possibly different counts after shuffle.
Common mistake: Drawing one box called 'the cluster'.
Follow-up: What changes M without changing N?