spark/fundamentals
intermediate
Connecting…

Stages to Tasks — partitions, tasks, executors

Four input partitions become four tasks. The driver schedules those tasks onto executor JVMs — three different nouns.

  1. ✓ Lazy Evaluation
  2. ✓ Catalyst to Job
  3. ✓ Job to Stages
  4. Stages to Tasks
  5. Shuffle & Result

Explain it at my level

  1. Partitions
  2. Tasks
  3. Executors
Loading execution simulation…