DataYatriSpark, seen under the hood
Simulate

Interview/Spark

Spark interview questions: Driver vs executors

Interactive Spark interview questions on Driver vs executors. Practice with the matching lesson. The driver plans. Executors run tasks. collect() and toPandas() pull data to the driver — that is how it dies.

Lesson · Simulation

beginner
Driver vs executors

What runs on the driver versus an executor?

Answer it out loud, then reveal. Play steps through like the simulators.

Open lessonSee it in the simulator

Production scenario

Driver vs executors

toPandas() on a 'filtered' fact table

Symptoms

  • Filter looks selective in the notebook
  • Driver dies after the action starts
Open lessonSee it in the simulator

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

beginner

What runs on the driver versus an executor?

Driver vs executors · tap to open the answer

Short: Driver: SparkSession, Catalyst, DAG, task scheduling. Executor: tasks, cache, shuffle files.

Detailed: Your notebook code until an action is driver-side. After the action, work is split into tasks that executors run on partitions. Results come back only if the action asks (show, collect).

Common mistake: Thinking each executor has its own SparkSession you should create.

Follow-up: Why is creating a SparkSession inside a foreach a bug?

Lesson · Simulation

intermediate

Why does write() not send the table through the driver?

Driver vs executors · tap to open the answer

Short: Executors write partitions in parallel to storage. The driver only coordinates the commit.

Detailed: Each task writes its slice (Parquet/Delta files). The driver never materializes the full dataset. That is why write is safe and collect is not.

Common mistake: Using collect() 'to inspect' a 200 GB frame before writing.

Follow-up: What does show() send back?

Lesson · Simulation

senior

You have 200 executors and a 2 GB driver. Which jobs will still fail?

Driver vs executors · tap to open the answer

Short: Any action that pulls the full result to the driver — collect, toPandas, display of an unaggregated frame.

Detailed: Executor count does not grow driver heap. A 2 TB collect still targets one JVM. Broadcast join of a 8 GB dimension also hits the driver, then every executor.

Senior: Aggregate or sample on executors, write the rest, never pull the grain to the notebook.

Common mistake: Scaling workers to fix a driver OOM.

Follow-up: How would you rewrite the notebook so the driver stays small?

Lesson · Simulation