Container killed with no OutOfMemoryError anywhere in the logs
Symptoms
- Executors lost repeatedly, with a kill reason from the cluster manager rather than the JVM
- No OutOfMemoryError in executor stderr
- It always dies during the same pandas UDF stage
Interactive Spark interview questions on Executor memory: execution wins, cache loses. Practice with the matching lesson. Execution and storage share one pool. Execution can evict your cache, then spill to disk, then die.
Question 1 of 9
An executor has 32 GB. How much of that can you actually cache into?
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 4 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
Showing 9 of 9 questions.
An executor has 32 GB. How much of that can you actually cache into?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: Nowhere near all of it — the heap is split into reserved memory, a unified execution/storage region, and user memory.
Why: spark.memory.fraction decides how much of the usable heap the unified region gets, and inside that spark.memory.storageFraction is the share cache is guaranteed to hold onto. Whatever sits outside the fraction holds your own objects and Spark internals. Execution memory is what shuffles, sorts, joins, and aggregations consume.
Common mistake: Treating spark.executor.memory as the cache budget.
Follow-up they will ask: What happens when execution needs memory that cache is currently holding?
Execution and storage share one pool. Who is allowed to evict whom?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: Execution can evict cached blocks; storage can never push out execution memory.
Why: Either side can borrow free space, but execution wins the tug-of-war and will evict cached partitions down to the storageFraction floor. That is why a DataFrame you cached quietly disappears and the next action rescans the source — the Storage tab shows the cached fraction dropping and the plan goes back to a FileScan.
Common mistake: Assuming cache() guarantees the data stays in memory for the rest of the session.
Follow-up they will ask: How would you spot that silent eviction in the UI?
One task spilled 60 GB to disk and finished. A different task's container was killed. Same memory problem?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: No — spill is Spark managing execution memory; a killed container is the cluster manager seeing total process memory exceed its limit.
Why: Spark tracks execution memory and spills sorted runs to local disk when it runs short, so the task survives slowly and you see Spill (Memory) and Spill (Disk) on the stage. YARN or Kubernetes knows nothing about that accounting — it sees the whole process: JVM heap plus off-heap buffers plus Python workers. Cross the container limit and it is killed with no Spark-level OutOfMemoryError anywhere.
Senior depth: spark.executor.memoryOverhead — the non-heap headroom for Python workers, off-heap buffers, and shuffle. On PySpark and pandas UDF workloads that is usually what is actually being exceeded.
Common mistake: Raising spark.executor.memory to fix a container kill, which just asks for a limit it still overshoots.
Follow-up they will ask: Which setting do people forget in that second case?
Why does Tungsten's binary row format belong in a memory conversation, not just a CPU one?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: Binary rows outside the Java object model use far less space per row and stop the GC from walking your data.
Why: Tungsten keeps rows in compact buffers with explicit offsets instead of JVM objects, so the same records occupy much less memory and the collector ignores them. Turning on spark.memory.offHeap.enabled with a size moves execution memory off the heap entirely, which keeps GC pauses down — but that memory still counts against the container's total limit.
Senior depth: Python UDFs — rows leave the binary format and become Python objects, so you pay the conversion and the memory back. That is a memory argument for built-ins, not only a CPU one.
Common mistake: Enabling off-heap memory without raising the container's overall allowance.
Follow-up they will ask: What destroys the Tungsten advantage in a PySpark pipeline?
A stage reports Spill (Memory) 40 GB and Spill (Disk) 6 GB. Why are the two numbers different?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: They measure the same records in two forms: deserialized in memory, then serialized and compressed on disk.
Remember these
Why: When execution memory fills, Spark serializes the in-memory records of a sort or aggregation and writes them to local disk as a sorted run. Spill (Memory) is how much that data occupied in deserialized form before the spill; Spill (Disk) is the bytes actually written after serialization and compression. A large ratio between them is normal and tells you the data compresses well, not that something is broken. What matters is that any disk spill at all means the task needed more execution memory than it was given, and merging those runs is extra I/O on the critical path. Compare the spilling task against the median task in the same stage before reaching for configuration.
Common mistake: Reading the two numbers as two separate spill events, or as evidence of a leak.
Follow-up they will ask: Is spill always worth eliminating?
Besides execution and storage there is user memory. What actually lives there, and when does it matter?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: Everything Spark does not manage — your own objects, UDF state, closures, and class metadata.
Remember these
Why: Spark reserves a fixed slice of the heap, takes spark.memory.fraction of the remainder as the unified execution and storage pool, and leaves the rest as user memory. Nothing in that region is tracked or spilled by Spark, so a large lookup dictionary built inside mapPartitions, results accumulated in a local list, or a fat closure captured by a UDF all consume it silently. That is why lowering spark.memory.fraction is occasionally the right move: a job whose work lives in user-code data structures needs more unmanaged room, and raising the fraction starves it. The symptom is heap pressure and GC time with no spill recorded on the stage, because Spark genuinely does not know where the memory went.
spark.conf.set("spark.memory.fraction", "0.5") # more room for user-code objectsCommon mistake: Assuming every byte on the heap is inside the unified pool and therefore spillable.
Follow-up they will ask: How would you shrink a large per-partition lookup structure?
Walk me from 'execution memory is full' to an actual OutOfMemoryError. What are the steps in between?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: Borrow free space, evict cached blocks down to the storage floor, spill sorted runs to disk, and only then fail.
Remember these
Why: A task asks the memory manager for execution memory. If the unified pool has free space it is granted; if storage is using space above spark.memory.storageFraction, those cached blocks are evicted to make room, because eviction only runs in that direction. When nothing more can be reclaimed, the sorter or the aggregation map serializes its contents to a local disk file as a sorted run and continues with an empty buffer, repeating as needed and merging the runs at the end — that is the Spill (Memory) and Spill (Disk) pair. The JVM only throws OutOfMemoryError when something unspillable does not fit: a single record or key group larger than the available page, an untracked user-code allocation, or a heap exhausted by blocks persisted MEMORY_ONLY. A container kill is a different failure entirely — the process exceeding its total limit, heap plus off-heap plus Python — and Spark's accounting never sees it.
# stage page: Spill (Memory), Spill (Disk), Peak Execution Memory
# executor log: OutOfMemoryError vs a container-killed messageSenior depth: Eviction. There is no counter for blocks evicted by execution on the stage page — you infer it from Fraction Cached on the Storage tab falling between queries, and from a later plan reverting to FileScan where it previously showed InMemoryTableScan. That is the mechanism behind a notebook that gets slower over a session with no error anywhere.
Common mistake: Explaining it as 'memory fills and Spark OOMs', skipping eviction and spill entirely.
Follow-up they will ask: Which of those steps is invisible in the Spark UI?
A stage spends 40% of task time in GC, with no spill and no OOM. What do you do?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: Find what is holding long-lived objects — usually cache or user-code structures — before touching heap size.
Remember these
Why: High GC with no spill means the collector is working hard, not that execution memory ran short, so raising spark.executor.memory often makes it worse by lengthening full-collection pauses. Check the Storage tab first: a large MEMORY_ONLY cache keeps millions of live objects that every collection has to trace. Then look at user memory — a per-partition dictionary, a growing accumulator, or Python worker churn on the same host. Enabling off-heap execution memory with spark.memory.offHeap.enabled moves Tungsten's buffers out of the collector's reach, and smaller executors with fewer cores shrink the live set each JVM manages. In the Executors tab, compare GC Time against Task Time per executor: if one host dominates, suspect cache placement or a noisy neighbour rather than the job itself.
df.unpersist()
spark.conf.set("spark.memory.offHeap.enabled", "true")
spark.conf.set("spark.memory.offHeap.size", "4g")Senior depth: When the live set per JVM is the problem rather than total capacity. Several smaller executors give several smaller heaps, each with shorter pauses and an independent collector, at the cost of more broadcast copies and more shuffle connections. The trade is real: fewer, larger executors reduce broadcast duplication and shuffle fan-out, while more, smaller executors reduce GC pause and the blast radius when one dies.
Common mistake: Treating GC time as a signal to request a larger instance type.
Follow-up they will ask: When is a smaller executor the better answer?
Same budget, two shapes: eight executors with sixteen cores, or thirty-two with four. Which for a shuffle-heavy job?
Executor memory: execution wins, cache loses · tap to open the answer
Say this first: Mid-size executors, because execution memory is divided by concurrent tasks and both extremes hurt in different ways.
Remember these
Why: Every core on an executor runs a task concurrently and all of them draw from the same unified pool, so a sixteen-core executor gives each task roughly a sixteenth of the execution memory — which is how a large executor still spills. Very large executors also mean long GC pauses, a bigger loss when one dies, and heavy object-store contention from many threads in one JVM. Very small executors waste memory on per-executor overhead, hold a broadcast copy each, and multiply shuffle connections, which makes fetch failures more likely. The usual landing zone is four to five cores per executor with spark.executor.memoryOverhead budgeted on top, then sizing the count for total parallelism. Validate with Peak Execution Memory and Spill (Disk) per task rather than with a rule of thumb.
--executor-cores 4 --executor-memory 16g --conf spark.executor.memoryOverhead=3gSenior depth: Python workers live outside the heap, roughly one per task slot, so a high-core executor multiplies Python processes against a single memoryOverhead budget. For pandas UDF workloads keep cores per executor lower and overhead higher, and cap the Arrow batch size — otherwise you converge on container kills with a heap chart that looked healthy the whole way down.
Common mistake: Maximising cores per executor to 'use the machine fully', then raising memory to stop the resulting spill.
Follow-up they will ask: What changes this answer for a PySpark-heavy job?