DataYatriSpark, seen under the hood
Simulate
spark/performance
advanced
Connecting…

Adaptive Query Execution

AQE uses runtime stats to coalesce shuffle partitions, change join strategy, and split skewed partitions.

Lesson 27 of 29 · Spark path

Explain it at my level

  1. Static 200
  2. Coalesce
  3. Join switch
  4. Skew split
Watch the canvas:shuffle partitionskewed partitionLive simulation
200 shuffle partitions (static default)································SortMergeJoinplanned from stale estimatesBroadcastHashJoinbuild side measured at 8 MBpart-7aquarter of the hot keypart-7bquarter of the hot keypart-7cquarter of the hot keypart-7dquarter of the hot key
AQE re-plans between stages using measured sizes instead of estimates.