Executor OOM

One fat task fills Spark memory. The driver stays up. That worker is gone.

Watch the canvas:overheadstorage (cache)execution (shuffle / join)Live simulation

Executor JVM container (spark.executor.memory + overhead)

Off-heap overhead (JVM, NIO, ~10%)60%

Reserved (300MB fixed)100%

Storage — cached RDD / DataFrame partitions0%

Execution — shuffles, joins, aggregations, sorts0%

0 MBContainer cap: heap + overhead · Spark pool ≈ (heap − 300MB) × 0.6

Partition data + shuffle arriving:

Executor JVM started. Base regions allocated.

The driver stays up. This worker is the one a fat task fills.