Interview/Spark

Spark interview questions: Executor OOM

Interactive Spark interview questions on Executor OOM. Practice with the matching lesson. A skewed partition, MEMORY_ONLY cache, or oversized broadcast kills an executor JVM. The driver is usually still alive.

Lesson · Simulation

beginner
Executor OOM

How is executor OOM different from driver OOM?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Executor OOM

One executor lost on a join, rest healthy

Symptoms

  • Task 183 always dies
  • Same join key

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

How is executor OOM different from driver OOM?

Executor OOM · tap to open the answer

Say this first: A worker JVM dies. The driver usually stays up. You see ExecutorLostFailure / container killed.

Why: Causes: fat task (skew), huge partition, MEMORY_ONLY cache, big broadcast on the executor, explode() blowup.

Common mistake: Restarting the driver to fix an executor OOM.

Follow-up they will ask: What does Spark do after an executor is lost?

Lesson · Simulation

intermediate

A task explodes a JSON array and dies. What's the fix?

Executor OOM · tap to open the answer

Say this first: The partition became huge after explode — not the scan.

Why: explode multiplies rows in one task. Repartition before explode, filter arrays, or process hot keys separately. Memory fraction / spark.memory.fraction is a last resort.

Common mistake: spark.executor.memory 64g as the first change.

Follow-up they will ask: How do you see the task that died?

Lesson · Simulation

senior

How do you choose between salting, more partitions, and more executor memory?

Executor OOM · tap to open the answer

Say this first: If one key is fat: salt or split. If all tasks are fat: more partitions or a smaller row (drop columns). Memory last.

Why: UI: shuffle read per task. One bar huge → skew. All bars huge → partition sizing or explode. GC thrash with even bytes → memory/config or too much cache.

Senior depth: Copy-pasting spark.memory.fraction from OSS blogs onto Photon/DBR without measuring.

Common mistake: A 2× memory ticket with no histogram attached.

Follow-up they will ask: What Spark config is a footgun with off-heap and Databricks?

Lesson · Simulation

intermediate

You cache a 200 GB table on a cluster with roughly 120 GB of storage memory. Nothing fails. What breaks later?

Executor OOM · tap to open the answer

Say this first: Only part of it is cached, and with MEMORY_ONLY the missing blocks are recomputed from source on every access.

Remember these

  • A cache that does not fit is partial and silent — check Fraction Cached
  • MEMORY_ONLY blocks that do not fit are recomputed, not spilled
  • Cached blocks squeeze execution memory for every later stage

Why: Spark stores as many blocks as fit and then silently stops, so the Storage tab shows Fraction Cached below 100%. Every later action rescans the source for the missing blocks, so the query that was supposed to be fast is partly a full scan. Worse, the blocks that did fit occupy the unified pool, so the next shuffle has less execution memory, spills, and then evicts those cached blocks down to the storage floor anyway. The visible failure arrives later and somewhere else: an OOM or heavy spill in a join stage that has nothing to do with the cache. Check Fraction Cached before trusting a cache, prefer MEMORY_AND_DISK for anything expensive to rebuild, and unpersist what you are done with.

df.persist(StorageLevel.MEMORY_AND_DISK)
df.count()
# Storage tab: Fraction Cached should read 100%

Common mistake: Assuming cache() either works completely or raises an error.

Follow-up they will ask: Which storage level would you pick for an expensive-to-recompute frame?

Lesson · Simulation

advanced

An executor has 8 GB heap and 5 cores, and each task needs about 2 GB for its hash aggregation. Does it fit?

Executor OOM · tap to open the answer

Say this first: No — five concurrent tasks share one unified pool of roughly 4.5 GB, so each gets under a gigabyte and they spill or die.

Remember these

  • Unified pool ≈ (heap − reserve) × spark.memory.fraction
  • All cores on an executor draw from the same pool, concurrently
  • Peak Execution Memory per task is the demand side of the arithmetic
  • Fewer cores or smaller partitions beats a bigger heap divided by five

Why: Start from the heap: a fixed reserve comes off the top, then spark.memory.fraction — 0.6 by default — of the remainder forms the unified execution and storage pool, so 8 GB yields something like 4.5 GB of managed memory. Every core runs a task concurrently and all of them draw from that one pool, so with five cores the fair share is under a gigabyte per task, not 2 GB. The tasks will borrow free storage space, then spill sorted runs to disk, and only fail outright if a single unspillable record group does not fit. Peak Execution Memory per task on the stage page tells you what the tasks actually demanded. The two honest fixes are fewer cores per executor, so each task gets a larger share, or smaller partitions, so each task needs less — raising heap alone still divides by five.

# stage page: Peak Execution Memory, Spill (Memory), Spill (Disk)
# then try --executor-cores 3 before --executor-memory 16g

Senior depth: Because spill and GC are not free. With fewer concurrent tasks each one gets enough execution memory to finish in memory, so you remove the spill write, the spill read, and the merge from the critical path, and the JVM's live set shrinks so pauses get shorter. You lose parallelism per executor and usually get it back by running more executors. Prove it with Spill (Disk) going to zero and stage time dropping, not with the config diff.

Common mistake: Comparing a task's memory need against the full executor heap instead of against the per-task share.

Follow-up they will ask: Why might lowering executor cores make the job faster as well as safer?

Lesson · Simulation

advanced

The executor OOM happens in the write stage, not in any shuffle. What causes that?

Executor OOM · tap to open the answer

Say this first: A partitioned write holds one open writer per output partition value, each with its own buffer, so high cardinality in one task exhausts the heap.

Remember these

  • One open writer per output partition value, each with a row-group buffer
  • Unsorted input means many writers held open in one task at once
  • Signature: OOM in the final stage with no shuffle skew
  • repartition on the partition columns before the write

Why: Writing with partitionBy means a task may hold rows for many partition values, and each open output file carries a Parquet writer with its own in-memory row group buffer — tens or hundreds of megabytes each when row groups are large. If the data reaching a task is not sorted by the partition columns, it can open a writer for every value it encounters and hold them all simultaneously. Spark mitigates this by sorting on the partition columns before the write, but a plan that skips the sort, or a task that genuinely carries many values, still blows up. The signature is an OOM in the final stage with no Shuffle Read skew, plus an output directory with many small files written by the same task. The fixes are to repartition on the partition columns so each task writes few values, to cap rows per file with maxRecordsPerFile, or to reduce the Parquet row group size so each open writer buffers less.

(df.repartition("event_date", "country")
   .write.partitionBy("event_date", "country")
   .option("maxRecordsPerFile", 1000000).save(path))

Senior depth: Dynamic partition overwrite stages output and then swaps partitions, so the task still holds the same open writers and buffers, plus the commit protocol's bookkeeping. It makes the write heavier, not lighter. If a job both overwrites dynamically and carries many partition values per task, repartition on the partition columns first and keep the value count per task small — that one change addresses both failure modes.

Common mistake: Looking only at shuffle stages when the OOM is in the write, and concluding the data must be skewed.

Follow-up they will ask: How does this interact with dynamic partition overwrite?

Lesson · Simulation

senior

One executor dies and the job takes four times longer, not one twenty-fourth longer. Why does losing one executor cost that much?

Executor OOM · tap to open the answer

Say this first: Its shuffle files die with it, so downstream fetches fail and the DAGScheduler resubmits whole map stages.

Remember these

  • Losing an executor loses its shuffle output, not just its compute
  • FetchFailedException resubmits a parent stage, not just a task
  • A memory-caused death often repeats on the replacement executor
  • External shuffle service or shuffle tracking bounds the blast radius

Why: Shuffle output is written to the dead executor's local disk, and without an external shuffle service those blocks are gone. Reduce tasks fetching them raise FetchFailedException, the DAGScheduler unregisters the map output and resubmits the missing partitions of the parent stage, and everything that depended on them is redone. If the cause was memory pressure, the replacement executor often inherits the same fat partition and dies too, so you get a retry loop that burns the stage retry budget before failing the job. Cached blocks on that executor are lost and recomputed as well, and the retried stage competes with ongoing work. The UI shows a stage with multiple attempts, FetchFailedException in the logs, and a wall clock unrelated to the data volume. Keeping shuffle files alive past executor death — an external shuffle service, or shuffle tracking under dynamic allocation — is what bounds the damage.

spark.conf.get("spark.dynamicAllocation.shuffleTracking.enabled")
# logs: FetchFailedException, then the parent stage resubmitted as attempt 2

Senior depth: When the failure is deterministic. A single fat partition will kill every executor it lands on, so each retry costs a full stage recompute and gets you no closer. Recognise it from the same task index failing across attempts, stop the retry loop, and fix the partition — salt, skew split, or a smaller partition size — before adding any memory. Raising spark.task.maxFailures in that situation just buys more identical failures.

Common mistake: Expecting the cost of a lost executor to be proportional to its share of the cluster.

Follow-up they will ask: When does retrying make things strictly worse?

Lesson · Simulation

architect

Three of your sixty jobs OOM each month and each one gets a memory bump. What do you change structurally?

Executor OOM · tap to open the answer

Say this first: Require evidence before a memory change, and attack the two real causes — partition sizing and skew — as platform defaults.

Remember these

  • No memory change without the task summary attached
  • Max vs median separates skew from partition sizing from GC
  • Fix defaults: AQE skew join, cores per executor, overhead, storage level
  • Track memory per job as a trend, not per incident

Why: Make a memory increase require the stage's task summary: max versus median Shuffle Read, Peak Execution Memory, and Spill (Disk). If max is far above median it is skew and memory is the wrong lever; if every task is large it is partition sizing; if GC dominates with no spill it is cache or user memory. Then fix the defaults rather than the individual jobs: AQE on with skew join and a sensible advisory partition size, four to five cores per executor with memoryOverhead budgeted for PySpark, MEMORY_AND_DISK as the house storage level, and a target partition size everybody can quote. Track memory per job over time as a metric, because a slow upward creep is the signal that one key or one table is outgrowing the design. The structural failure is not that jobs OOM — it is that the cheapest remediation is also the one that hides the cause.

Senior depth: Per job and per stage: max-to-median task duration ratio, total Spill (Disk), GC time as a share of task time, and configured memory over time. Those four separate the failure classes without opening a single log, and the memory trend line is what catches the slow creep. Alert on the ratio crossing a threshold rather than on the OOM, so you get the ticket before the page.

Common mistake: Letting 'bump the memory' be the one-click remediation, so no job is ever actually diagnosed.

Follow-up they will ask: What would you put on a dashboard for this?

Lesson · Simulation