DataYatriSpark, seen under the hood
Simulate

Simulation/Spark

Educational simulation
Spark

Executor OOM

One fat task fills Spark memory. The driver stays up. That worker is gone.

Watch the canvas:healthy executorSpark memory fulldriver (fine)Live simulation
Driver JVMstill alivedriver0.6g / 8gExecutor 116g heapused2.2g / 16gExecutor 216g heapused2.4g / 16gExecutor 316g heapused2.0g / 16gSpark unified memory ≈ 9.4g on a 16g executor — execution + storage share itSkew / MEMORY_ONLY cache / fat broadcast fill Spark memory, not the driverExecutorLostFailure. Driver still up. Stage retries the fat task.Salt / MEMORY_AND_DISK / drop the broadcast — then the 9.4g lid holds
Spark memory is (executor.memory − 300MB) × 0.6 — not the whole heap.