Spark interview questions
Driver vs executors, lazy evaluation, shuffle, joins, skew, AQE, and OOM — practiced against the same simulators as the lessons.
81 questions
Beginner
Partitions, actions, lazy evaluation, and what the driver actually does.
Senior
Skew, shuffle, join strategy, Catalyst, and Spark UI evidence.
Scenarios
Production failures: symptoms, Spark UI proof, then the change.
By topic
Each topic has the lesson and the simulator alongside the questions.
Spark Overview
What Spark is, why it exists, and the mental model of driver, executors, and lazy plans.
Spark Architecture
Cluster manager, driver JVM, executor JVMs, and how a SparkSession talks to a cluster.
Driver & Executors
The driver plans. Executors run tasks. Never ship a huge collect() back to the driver.
Execution Flow
Five short labs: lazy code → plan → job → stages → tasks → shuffle → result.
Lazy Evaluation (Code → Plan)
Why groupBy does nothing until show(), and how Catalyst writes the plan first.
Catalyst to Job
show() submits Job 0. groupBy inserts the Exchange that will split stages.
Job to Stages
A shuffle edge splits Job 0 into Stage 0 (map write) and a Stage 1 that waits.
Stages to Tasks
Four partitions become four tasks. The driver schedules them onto executor JVMs.
Shuffle & Result
Hash partitioning, local write, network fetch, reduce tasks, and show().
DataFrame
Structured data with a schema. Catalyst can optimize DataFrames in ways RDDs cannot.
Transformations
Narrow vs wide transformations, and why nothing runs until an action arrives.
Actions
show, count, collect, write — the calls that force Spark to produce a result.
Lazy Evaluation
Spark builds a DAG of transformations and waits. That wait is a feature, not a bug.
RDD
Resilient Distributed Datasets: partitions, lineage, and when you still need the RDD API.
Partitions
A partition is a chunk of data. One task processes one partition. Size them for healthy runtime.
Shuffle
Records move across the network so matching keys land on the same reducer.
Joins
Sort-merge, shuffle hash, and broadcast. The strategy Spark picks changes the cost by orders of magnitude.
Caching
persist and cache store computed partitions so later actions do not recompute the whole DAG.
Job / Stage / Task
An action creates a job. Shuffle boundaries split jobs into stages. Stages split into tasks.
DAG
The directed acyclic graph of RDDs / DataFrame operators Spark will execute.
Catalyst
Logical plan → analyzed → optimized → physical plan. Exchange nodes mark shuffles.
Tungsten
Off-heap memory, whole-stage codegen, and cache-aware layout that make Spark SQL fast.
Broadcast Join
Replicate a small table to every executor and skip the shuffle on the large side.
Data Skew
One key owns most of the data. One task runs for 40 minutes while 199 finish in 2.
AQE
Adaptive Query Execution coalesces shuffle partitions, switches join strategy, and handles skew at runtime.
Driver OOM
collect() and toPandas() pull the cluster into one JVM. Executors look fine. The driver is dead.
Executor OOM
Skew, MEMORY_ONLY cache, or a fat broadcast blows an executor heap. The driver is still alive.