Interview/Spark

Spark interview questions: Spark cluster architecture

Interactive Spark interview questions on Spark cluster architecture. Practice with the matching lesson. Driver, cluster manager, and executors. How a SparkSession becomes running JVMs.

Lesson · Simulation

Name the three Spark cluster roles and what each one must not do.

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Spark cluster architecture

Cluster looks fine, notebook is dead

Symptoms

  • Executors still registered
  • Kernel restart loop
  • No ExecutorLostFailure

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

Name the three Spark cluster roles and what each one must not do.

Spark cluster architecture · tap to open the answer

Say this first: Driver coordinates. Executors compute. Cluster manager places the JVMs.

Why: The driver builds the DAG and tracks tasks. Executors run tasks and hold cache. The cluster manager (YARN, K8s, Databricks) allocates machines. Big data stays on executors and storage — not in the driver heap.

Common mistake: Saying the driver 'processes the data'.

Follow-up they will ask: Where does collect() put the result?

Lesson · Simulation

intermediate

If the driver dies, why does the whole job die even if executors are healthy?

Spark cluster architecture · tap to open the answer

Say this first: The driver owns the SparkSession, DAG, and task scheduler. Executors are workers, not the brain.

Why: Lose the driver JVM and there is no one to retry tasks or return the action result. Databricks job clusters fail the run; notebooks disconnect.

Common mistake: Restarting one executor to 'fix a driver OOM'.

Follow-up they will ask: What Spark UI page is served from the driver?

Lesson · Simulation

senior

How do you prove a failure is driver-side vs executor-side in five minutes?

Spark cluster architecture · tap to open the answer

Say this first: Driver OOM: notebook/kernel dies, executors still green. Executor OOM: one worker lost, driver alive, task failed with ExecutorLostFailure.

Why: Spark UI Executors tab: look at driver vs executor memory. Driver heap used near max plus a collect/broadcast in the plan → driver. One executor with huge spill or a killed container → executor.

Senior depth: collect, toPandas, and broadcasting a large dimension. Write the hypothesis, then the UI evidence.

Common mistake: Restarting the cluster without reading the error class.

Follow-up they will ask: Which action is the classic driver killer?

Lesson · Simulation

beginner

On Databricks you never run spark-submit. So who starts the driver, and which machine is it?

Spark cluster architecture · tap to open the answer

Say this first: The control plane asks the cloud for VMs; the driver node is one of them, and your notebook attaches to a SparkSession already running there.

Remember these

  • Control plane requests VMs; driver and executors run in your data plane
  • Attaching a notebook joins an existing driver, it does not start one
  • Each notebook gets its own SparkSession on the shared driver
  • The Spark UI is served by the driver process

Why: When you create a cluster, the Databricks control plane issues the instance requests into your account's data plane, then starts a driver JVM on the driver node and executor JVMs on the worker nodes. Attaching a notebook does not create a Spark application — it opens a REPL against the driver that is already live, which is why a second notebook on the same cluster shares executors but gets its own SparkSession. The driver node also hosts the Python process behind your PySpark code, so notebook variables and anything you collect live on that machine, next to but outside the JVM heap. A single-node cluster simply runs executors in the driver's process, which is fine for teaching and wrong for anything with real volume. The Spark UI you open is served by the driver, so losing the driver loses the UI too.

Common mistake: Describing a notebook attach as 'submitting a Spark application' and then being unable to say why two notebooks see each other's executors.

Follow-up they will ask: Two notebooks on one cluster both run a wide job. What do they contend for?

Lesson · Simulation

intermediate

Your spark-submit works from the edge node but the identical scheduled run fails. What does deploy mode have to do with it?

Spark cluster architecture · tap to open the answer

Say this first: In client mode the driver runs on the edge node; in cluster mode it runs on a worker, so anything local to the edge node disappears.

Remember these

  • Client mode driver = your submitting host; cluster mode driver = a worker container
  • Local paths and localhost services vanish in cluster mode
  • Ship files with --files and reference them by bare filename
  • Client mode ties application lifetime to the submitting shell

Why: With --deploy-mode client the driver JVM is your edge-node process, so it can read local config files, resolve hostnames on that box, and reach a service on localhost. With --deploy-mode cluster the cluster manager launches the driver inside a container on some worker, and a path like /etc/app/creds.json or a jdbc:...@localhost URL is now a FileNotFoundException or a connection refused. The fix is to ship dependencies explicitly with --files and --jars so they land in the container working directory, and to reference them by filename rather than absolute path. Client mode also keeps the driver tied to your shell: closing the terminal kills the application, and all result traffic crosses the edge node's network link. Cluster mode is the right default for scheduled work precisely because the driver's lifetime and locality stop depending on a human session.

spark-submit --deploy-mode cluster \
  --files /etc/app/creds.json \
  job.py
# inside the driver: open('creds.json')

Common mistake: Assuming a job is environment-independent because it ran interactively on the gateway host.

Follow-up they will ask: Where do you read the driver logs in each mode?

Lesson · Simulation

advanced

An executor container was killed with exit code 137, but the Peak JVM Memory graph never came close to the limit. How is that possible?

Spark cluster architecture · tap to open the answer

Say this first: The container limit covers JVM heap plus overhead plus off-heap plus Python, and only the first of those shows up as heap usage.

Remember these

  • Container limit = spark.executor.memory + spark.executor.memoryOverhead
  • Overhead holds metaspace, thread stacks, netty shuffle buffers, off-heap
  • PySpark spawns one python worker per concurrent task, outside the heap
  • Exit 137 is a kernel kill; java.lang.OutOfMemoryError is a heap kill

Why: The cluster manager enforces a limit on total resident memory for the container, which is roughly spark.executor.memory plus spark.executor.memoryOverhead. Overhead is not slack: it holds JVM metaspace, thread stacks, the netty buffers used for shuffle fetches, and any off-heap Tungsten region you enabled with spark.memory.offHeap.enabled and spark.memory.offHeap.size. With PySpark it also has to cover the python worker processes, one per concurrent task, which is why a job that was healthy in Scala starts getting OOM-killed the week someone adds a Python UDF. Heap graphs look fine because none of that growth is heap — the kernel kills the container while the JVM believes it has room, so you get exit 137 or 'Container killed by YARN for exceeding memory limits' rather than java.lang.OutOfMemoryError. The error class is the diagnostic: heap OOM means tune heap or reduce per-task data; 137 means raise overhead or lower concurrency per executor.

spark.conf.set('spark.executor.memoryOverhead', '4g')
spark.conf.set('spark.executor.pyspark.memory', '2g')

Senior depth: Read the error class before touching a number. Heap OOM and container kill point in opposite directions, and the single most common wasted day on a Spark platform is raising executor memory against a 137. Fewer cores per executor is often the cheaper fix because it directly cuts the number of concurrent python workers and shuffle buffers per container.

Common mistake: Raising spark.executor.memory in response to exit 137, which shrinks the overhead headroom and makes the kill arrive sooner.

Follow-up they will ask: Why can lowering spark.executor.cores fix an exit-137 loop without adding memory?

Lesson · Simulation

senior

Why does Spark usually prefer several mid-sized executors per node over one executor that owns the whole machine, and where does that advice break?

Spark cluster architecture · tap to open the answer

Say this first: Mid-sized executors keep GC pauses and I/O concurrency manageable; too many of them multiply broadcast copies and shuffle connections.

Remember these

  • One giant heap means cluster-wide stalls on full GC
  • High spark.executor.cores shrinks each task's execution memory share
  • Broadcasts are materialised once per executor, not per core
  • More executors means more shuffle locations and fetch connections

Why: A single huge executor means one JVM heap of tens of gigabytes, and full GC on that heap stalls every task in the container at once, which you see as a GC Time column rivalling task duration. Very high spark.executor.cores also means many tasks sharing one execution memory pool, so each task's share shrinks and sorts start spilling earlier. The usual landing zone is a handful of cores per executor, because object-store and HDFS client throughput stops improving past that point anyway. The opposite extreme is just as bad: one core per executor pays full JVM startup and metaspace per task slot, and every broadcast variable is materialised once per executor, so a 500 MB broadcast across 200 tiny executors is 100 GB of duplicated memory cluster-wide. More executors also means more shuffle map-output locations and more concurrent fetch connections per reducer, which is what turns a marginal network into FetchFailed. Size for the job shape: shuffle-heavy wants memory per task, scan-heavy wants cores.

Senior depth: The real variable is memory per task slot, not memory per executor. Divide the usable execution pool by cores and compare it against the largest shuffle partition you expect. If that number is below the partition size you are choosing to spill, and no amount of cluster growth changes it.

Common mistake: Quoting 'five cores per executor' as a law without mentioning broadcast duplication or per-task memory share.

Follow-up they will ask: How does dynamic allocation change how you pick executor size?

Lesson · Simulation

architect

Thirty teams have to share one Spark platform. Where do you put the isolation boundaries?

Spark cluster architecture · tap to open the answer

Say this first: Separate Spark applications per team workload, job clusters for scheduled work, and a shared-but-governed tier only for interactive exploration.

Remember these

  • Within one application there is no memory or driver isolation
  • FAIR scheduler pools share task slots, not memory or fault domains
  • Job clusters make the Spark application the fault boundary
  • Compute sharing must not imply data access; grants live in the catalog

Why: Inside one Spark application, isolation is weak: all notebooks share the same executors, the same block manager, and the same driver, so one person's collect() can kill everyone's session. Scheduler pools with spark.scheduler.mode set to FAIR divide task slots between concurrent queries, which smooths latency but does nothing about memory or a driver crash. That makes the application the only real fault boundary, so scheduled pipelines each get their own job cluster that lives for the duration of the run and dies with it. Interactive work is the hard case, because per-user clusters are expensive and idle most of the day; a shared cluster with autoscaling and enforced timeouts plus a serverless warehouse for SQL covers most of it. On top of compute, the data boundary is the catalog: table-level grants and separate schemas per domain, so compute sharing never implies read access. Document which tier is allowed to run production writes, or the shared tier quietly becomes production.

Senior depth: Pick the boundary by failure mode, not by cost. The question is always 'who else goes down with this', and the answer inside a single Spark application is 'everyone on the driver'. Cost optimisation on shared clusters is worth doing, but only after production writes are off them.

Common mistake: Treating FAIR scheduler pools as isolation and then being surprised when one driver OOM takes down thirty users.

Follow-up they will ask: What would you measure to decide whether the shared interactive tier is too small?

Lesson · Simulation