DataYatriSpark, seen under the hood
Simulate
spark/internals
intermediate
Connecting…

Jobs, stages, and tasks

An action creates a job. Shuffle boundaries split the job into stages. Each partition in a stage is a task.

Lesson 19 of 29 · Spark path

Explain it at my level

  1. Job
  2. Stage
  3. Task
Watch the canvas:jobstagetaskLive simulation
Job 0 · show()Stage 0map / FileScan + filterStage 1reduce / aggregateTask 0Task 1Task 2Task 3The action is the job.
Job = action. Stage = shuffle slice. Task = one partition in that stage.