spark/fundamentals
intermediate
Connecting…

Shuffle and Result — keys move, then we aggregate

Hash partitioning, local shuffle write, network fetch, reduce tasks, and the rows that come back to show().

  1. ✓ Lazy Evaluation
  2. ✓ Catalyst to Job
  3. ✓ Job to Stages
  4. ✓ Stages to Tasks
  5. Shuffle & Result

Explain it at my level

  1. Hash
  2. Write
  3. Network
  4. Reducers
  5. Aggregate
  6. Result
Loading execution simulation…