Apache Spark interview questions
Apache Spark interview questions on driver vs executors, lazy evaluation, shuffle, joins, skew, AQE, and OOM — practiced against the same simulators as the lessons.
94 questions
Beginner
Partitions, actions, lazy evaluation, and what the driver actually does.
Senior
Skew, shuffle, join strategy, Catalyst, and Spark UI evidence.
Scenarios
Production failures: symptoms, Spark UI proof, then the change.
By topic
Each topic has the lesson and the simulator alongside the questions.
Spark Overview
What Spark is, why it exists, and the mental model of driver, executors, and lazy plans.
Spark Architecture
Cluster manager, driver JVM, executor JVMs, and how a SparkSession talks to a cluster.
Driver & Executors
The driver plans. Executors run tasks. Never ship a huge collect() back to the driver.
Execution Flow
Trace groupBy + show() from lazy code through the plan, job, stages, tasks, shuffle, and result — one picture.
DataFrame
Structured data with a schema. Catalyst can optimize DataFrames in ways RDDs cannot.
Spark SQL
SQL strings and the DataFrame API compile to the same plan. Temp views, the catalog, and when names actually resolve.
Transformations
Narrow vs wide transformations, and why nothing runs until an action arrives.
Actions
show, count, collect, write — the calls that force Spark to produce a result.
Lazy Evaluation
Spark builds a DAG of transformations and waits. That wait is a feature, not a bug.
RDD
Resilient Distributed Datasets: partitions, lineage, and when you still need the RDD API.
Partitions
A partition is a chunk of data. One task processes one partition. Size them for healthy runtime.
Repartition vs Coalesce
repartition shuffles for even partitions, coalesce only merges, and write.partitionBy is a different thing entirely.
Shuffle
Records move across the network so matching keys land on the same reducer.
Joins
Sort-merge, shuffle hash, and broadcast. The strategy Spark picks changes the cost by orders of magnitude.
Caching
persist and cache store computed partitions so later actions do not recompute the whole DAG.
File Formats
Why Parquet reads a fraction of the bytes CSV does: column pruning, row-group stats, and embedded schema.
Window Functions
Rank, lag, and running totals keep every row. partitionBy is a shuffle — and forgetting it funnels everything to one task.
UDFs
A Python UDF is a black box Catalyst cannot optimize. Built-ins, higher-order functions, then pandas_udf.
Job / Stage / Task
An action creates a job. Shuffle boundaries split jobs into stages. Stages split into tasks.
DAG
The directed acyclic graph of RDDs / DataFrame operators Spark will execute.
Catalyst
Logical plan → analyzed → optimized → physical plan. Exchange nodes mark shuffles.
Tungsten
Off-heap memory, whole-stage codegen, and cache-aware layout that make Spark SQL fast.
Memory Management
Execution and storage share one pool. Execution can evict your cache, then spill to disk, then die.
Structured Streaming
A stream is an unbounded table read in micro-batches. Checkpoints give exactly-once; watermarks bound state.
Broadcast Join
Replicate a small table to every executor and skip the shuffle on the large side.
Data Skew
One key owns most of the data. One task runs for 40 minutes while 199 finish in 2.
AQE
Adaptive Query Execution coalesces shuffle partitions, switches join strategy, and handles skew at runtime.
Driver OOM
collect() and toPandas() pull the cluster into one JVM. Executors look fine. The driver is dead.
Executor OOM
Skew, MEMORY_ONLY cache, or a fat broadcast blows an executor heap. The driver is still alive.