DataYatriSpark, seen under the hood
Simulate

Interview/Spark

Spark cluster architecture interview questions

Interactive Spark interview questions on Spark cluster architecture. Same topic as /learn/spark/architecture. Driver, cluster manager, and executors. How a SparkSession becomes running JVMs.

Lesson · Simulation

beginner
Spark cluster architecture

Name the three Spark cluster roles and what each one must not do.

Answer it out loud, then reveal. Play steps through like the simulators.

Open lessonSee it in the simulator

Production scenario

Spark cluster architecture

Cluster looks fine, notebook is dead

Symptoms

  • Executors still registered
  • Kernel restart loop
  • No ExecutorLostFailure
Open lessonSee it in the simulator

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

beginner

Name the three Spark cluster roles and what each one must not do.

Spark cluster architecture · tap to open the answer

Short: Driver coordinates. Executors compute. Cluster manager places the JVMs.

Detailed: The driver builds the DAG and tracks tasks. Executors run tasks and hold cache. The cluster manager (YARN, K8s, Databricks) allocates machines. Big data stays on executors and storage — not in the driver heap.

Common mistake: Saying the driver 'processes the data'.

Follow-up: Where does collect() put the result?

Lesson · Simulation

intermediate

If the driver dies, why does the whole job die even if executors are healthy?

Spark cluster architecture · tap to open the answer

Short: The driver owns the SparkSession, DAG, and task scheduler. Executors are workers, not the brain.

Detailed: Lose the driver JVM and there is no one to retry tasks or return the action result. Databricks job clusters fail the run; notebooks disconnect.

Common mistake: Restarting one executor to 'fix a driver OOM'.

Follow-up: What Spark UI page is served from the driver?

Lesson · Simulation

senior

How do you prove a failure is driver-side vs executor-side in five minutes?

Spark cluster architecture · tap to open the answer

Short: Driver OOM: notebook/kernel dies, executors still green. Executor OOM: one worker lost, driver alive, task failed with ExecutorLostFailure.

Detailed: Spark UI Executors tab: look at driver vs executor memory. Driver heap used near max plus a collect/broadcast in the plan → driver. One executor with huge spill or a killed container → executor.

Senior: collect, toPandas, and broadcasting a large dimension. Write the hypothesis, then the UI evidence.

Common mistake: Restarting the cluster without reading the error class.

Follow-up: Which action is the classic driver killer?

Lesson · Simulation