DataYatriSpark, seen under the hood
Simulate
spark/performance
senior
Connecting…

Data skew

When one key owns most of the data, one task becomes the stage. Salt or AQE skew join.

Lesson 26 of 29 · Spark path

Explain it at my level

  1. Even
  2. Hot key
  3. Straggler
  4. Salt
Watch the canvas:healthy taskstragglersalted keyLive simulation
Reduce task durationUS2.1 minEU2 min · 2 GBIN2 min · 2 GBBR2 min · 2 GBUS_0~2.4 minUS_1~2.4 minUS_2~2.4 minUS_3~2.4 minUS_4~2.4 minUS_5~2.4 minUS_6~2.4 minUS_7~2.4 min199 tasks finished 38 minutes ago. The stage is still running.Spark UI → Stages → compare Max vs Median shuffle read40 minutes becomes ~2.5. Same work — better distribution.
Stage duration equals the slowest task, so one fat bar sets the bill.