Driver OOM
collect() 'just for a quick look'
Symptoms
- Filter in the SQL looks tight
- Cardinality was wrong
Spark interview questions: Driver OOM
Interactive Spark interview questions on Driver OOM. Practice with the matching lesson. collect() and toPandas() pull the cluster into one JVM. Executors stay green. The driver is the process that dies.
Question 1 of 3
beginner
Driver OOMWhat is a driver OOM?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat is a driver OOM?
Driver OOM · tap to open the answer
Short: The driver JVM ran out of heap. Executors may still be healthy.
Detailed: Classic causes: collect, toPandas, createDataFrame from a huge list, broadcast of a large table. The notebook kernel dies.
Common mistake: Adding executors to fix a driver OOM.
Follow-up: Which Spark UI tab shows driver memory?
intermediateHow do you rewrite collect() on a 2 TB frame?
Driver OOM · tap to open the answer
Short: Don't. Aggregate, sample, write, or take(n).
Detailed: If you need a local ML sample, sample() then limit, or write a sampled Delta table and read elsewhere. Never collect the grain.
Common mistake: Increasing driver memory from 8 GB to 16 GB as the plan.
Follow-up: What's the Databricks-specific cousin of collect()?
seniorBroadcast join killed the driver, not the executors. Walk the path.
Driver OOM · tap to open the answer
Short: Driver collects the build side to ship the broadcast.
Detailed: If the dimension is 6 GB, the driver must hold it. Executors OOM later if they also can't hold the hashed relation — but the first failure can be the driver. Measure broadcast exchange size.
Senior: Error class → driver vs executor → action/broadcast in the plan → rewrite or size the JVM with a documented cap.
Common mistake: Only looking at executor logs.
Follow-up: What's your evidence-ordered debug?