spark/performance
senior
Connecting…Executor OOM
A skewed partition, MEMORY_ONLY cache, or oversized broadcast kills an executor JVM. The driver is usually still alive.
Lesson 29 of 29 · Spark path
Explain it at my level
- Tasks
- Fat heap
- Lost executor
- Retry
Step 1 of 7
Watch the canvas:overheadstorage (cache)execution (shuffle / join)Live simulation
Executor JVM container (spark.executor.memory + overhead)
Off-heap overhead (JVM, NIO, ~10%)60%
Reserved (300MB fixed)100%
Storage — cached RDD / DataFrame partitions0%
Execution — shuffles, joins, aggregations, sorts0%
0 MBContainer cap: heap + overhead · Spark pool ≈ (heap − 300MB) × 0.6
Partition data + shuffle arriving:
Executor JVM started. Base regions allocated.
The driver stays up. This worker is the one a fat task fills.