Driver vs executors
toPandas() on a 'filtered' fact table
Symptoms
- Filter looks selective in the notebook
- Driver dies after the action starts
Driver vs executors interview questions
Interactive Spark interview questions on Driver vs executors. Same topic as /learn/spark/driver-executors. The driver plans. Executors run tasks. collect() and toPandas() pull data to the driver — that is how it dies.
Question 1 of 3
beginner
Driver vs executorsWhat runs on the driver versus an executor?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat runs on the driver versus an executor?
Driver vs executors · tap to open the answer
Short: Driver: SparkSession, Catalyst, DAG, task scheduling. Executor: tasks, cache, shuffle files.
Detailed: Your notebook code until an action is driver-side. After the action, work is split into tasks that executors run on partitions. Results come back only if the action asks (show, collect).
Common mistake: Thinking each executor has its own SparkSession you should create.
Follow-up: Why is creating a SparkSession inside a foreach a bug?
intermediateWhy does write() not send the table through the driver?
Driver vs executors · tap to open the answer
Short: Executors write partitions in parallel to storage. The driver only coordinates the commit.
Detailed: Each task writes its slice (Parquet/Delta files). The driver never materializes the full dataset. That is why write is safe and collect is not.
Common mistake: Using collect() 'to inspect' a 200 GB frame before writing.
Follow-up: What does show() send back?
seniorYou have 200 executors and a 2 GB driver. Which jobs will still fail?
Driver vs executors · tap to open the answer
Short: Any action that pulls the full result to the driver — collect, toPandas, display of an unaggregated frame.
Detailed: Executor count does not grow driver heap. A 2 TB collect still targets one JVM. Broadcast join of a 8 GB dimension also hits the driver, then every executor.
Senior: Aggregate or sample on executors, write the rest, never pull the grain to the notebook.
Common mistake: Scaling workers to fix a driver OOM.
Follow-up: How would you rewrite the notebook so the driver stays small?