spark/fundamentals
intermediate
Connecting…Shuffle and Result — keys move, then we aggregate
Hash partitioning, local shuffle write, network fetch, reduce tasks, and the rows that come back to show().
- ✓ Lazy Evaluation
- ✓ Catalyst to Job
- ✓ Job to Stages
- ✓ Stages to Tasks
- Shuffle & Result
Explain it at my level
- Hash
- Write
- Network
- Reducers
- Aggregate
- Result
Loading execution simulation…