spark/fundamentals
intermediate
Connecting…Job to Stages — map side vs reduce side
A shuffle edge splits Job 0 into Stage 0 (map, shuffle write) and a Stage 1 that cannot start until the write finishes.
- ✓ Lazy Evaluation
- ✓ Catalyst to Job
- Job to Stages
- Stages to Tasks
- Shuffle & Result
Explain it at my level
- Job
- Stage 0
- Stage 1 (later)
Loading execution simulation…