Executor OOM
One executor lost on a join, rest healthy
Symptoms
- Task 183 always dies
- Same join key
Executor OOM interview questions
Interactive Spark interview questions on Executor OOM. Same topic as /learn/spark/executor-oom. A skewed partition, MEMORY_ONLY cache, or oversized broadcast kills an executor JVM. The driver is usually still alive.
Question 1 of 3
beginner
Executor OOMHow is executor OOM different from driver OOM?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerHow is executor OOM different from driver OOM?
Executor OOM · tap to open the answer
Short: A worker JVM dies. The driver usually stays up. You see ExecutorLostFailure / container killed.
Detailed: Causes: fat task (skew), huge partition, MEMORY_ONLY cache, big broadcast on the executor, explode() blowup.
Common mistake: Restarting the driver to fix an executor OOM.
Follow-up: What does Spark do after an executor is lost?
intermediateA task explodes a JSON array and dies. What's the fix?
Executor OOM · tap to open the answer
Short: The partition became huge after explode — not the scan.
Detailed: explode multiplies rows in one task. Repartition before explode, filter arrays, or process hot keys separately. Memory fraction / spark.memory.fraction is a last resort.
Common mistake: spark.executor.memory 64g as the first change.
Follow-up: How do you see the task that died?
seniorHow do you choose between salting, more partitions, and more executor memory?
Executor OOM · tap to open the answer
Short: If one key is fat: salt or split. If all tasks are fat: more partitions or a smaller row (drop columns). Memory last.
Detailed: UI: shuffle read per task. One bar huge → skew. All bars huge → partition sizing or explode. GC thrash with even bytes → memory/config or too much cache.
Senior: Copy-pasting spark.memory.fraction from OSS blogs onto Photon/DBR without measuring.
Common mistake: A 2× memory ticket with no histogram attached.
Follow-up: What Spark config is a footgun with off-heap and Databricks?