spark/fundamentals
intermediate
Connecting…Stages to Tasks — partitions, tasks, executors
Four input partitions become four tasks. The driver schedules those tasks onto executor JVMs — three different nouns.
- ✓ Lazy Evaluation
- ✓ Catalyst to Job
- ✓ Job to Stages
- Stages to Tasks
- Shuffle & Result
Explain it at my level
- Partitions
- Tasks
- Executors
Loading execution simulation…