Driver OOM
collect() 'just for a quick look'
Symptoms
- Filter in the SQL looks tight
- Cardinality was wrong
Driver OOM interview questions
Interactive Spark interview questions on Driver OOM. Same topic as /learn/spark/driver-oom. collect() and toPandas() pull the cluster into one JVM. Executors stay green. The driver is the process that dies.
Question 1 of 3
beginner
Driver OOMWhat is a driver OOM?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat is a driver OOM?
Driver OOM · tap to open the answer
Short: The driver JVM ran out of heap. Executors may still be healthy.
Detailed: Classic causes: collect, toPandas, createDataFrame from a huge list, broadcast of a large table. The notebook kernel dies.
Common mistake: Adding executors to fix a driver OOM.
Follow-up: Which Spark UI tab shows driver memory?
intermediateHow do you rewrite collect() on a 2 TB frame?
Driver OOM · tap to open the answer
Short: Don't. Aggregate, sample, write, or take(n).
Detailed: If you need a local ML sample, sample() then limit, or write a sampled Delta table and read elsewhere. Never collect the grain.
Common mistake: Increasing driver memory from 8 GB to 16 GB as the plan.
Follow-up: What's the Databricks-specific cousin of collect()?
seniorBroadcast join killed the driver, not the executors. Walk the path.
Driver OOM · tap to open the answer
Short: Driver collects the build side to ship the broadcast.
Detailed: If the dimension is 6 GB, the driver must hold it. Executors OOM later if they also can't hold the hashed relation — but the first failure can be the driver. Measure broadcast exchange size.
Senior: Error class → driver vs executor → action/broadcast in the plan → rewrite or size the JVM with a documented cap.
Common mistake: Only looking at executor logs.
Follow-up: What's your evidence-ordered debug?