toPandas() on a 'filtered' fact table
Symptoms
- Filter looks selective in the notebook
- Driver dies after the action starts
Interactive Spark interview questions on Driver vs executors. Practice with the matching lesson. The driver plans. Executors run tasks. collect() and toPandas() pull data to the driver — that is how it dies.
Question 1 of 8
What runs on the driver versus an executor?
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 3 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
Showing 8 of 8 questions.
What runs on the driver versus an executor?
Driver vs executors · tap to open the answer
Say this first: Driver: SparkSession, Catalyst, DAG, task scheduling. Executor: tasks, cache, shuffle files.
Why: Your notebook code until an action is driver-side. After the action, work is split into tasks that executors run on partitions. Results come back only if the action asks (show, collect).
Common mistake: Thinking each executor has its own SparkSession you should create.
Follow-up they will ask: Why is creating a SparkSession inside a foreach a bug?
Why does write() not send the table through the driver?
Driver vs executors · tap to open the answer
Say this first: Executors write partitions in parallel to storage. The driver only coordinates the commit.
Why: Each task writes its slice (Parquet/Delta files). The driver never materializes the full dataset. That is why write is safe and collect is not.
Common mistake: Using collect() 'to inspect' a 200 GB frame before writing.
Follow-up they will ask: What does show() send back?
You have 200 executors and a 2 GB driver. Which jobs will still fail?
Driver vs executors · tap to open the answer
Say this first: Any action that pulls the full result to the driver — collect, toPandas, display of an unaggregated frame.
Why: Executor count does not grow driver heap. A 2 TB collect still targets one JVM. Broadcast join of a 8 GB dimension also hits the driver, then every executor.
Senior depth: Aggregate or sample on executors, write the rest, never pull the grain to the notebook.
Common mistake: Scaling workers to fix a driver OOM.
Follow-up they will ask: How would you rewrite the notebook so the driver stays small?
Where does a Python UDF actually run, and whose memory does it consume?
Driver vs executors · tap to open the answer
Say this first: In a separate python worker process on each executor node — not in the executor JVM heap and not on the driver.
Remember these
Why: Your UDF body is pickled on the driver, shipped with the task, and executed by a python worker that the executor forks on the same machine. Rows travel from the JVM to that process and back over a local socket, serialised row by row for a plain udf and in Arrow batches for a pandas_udf. The plan node names the mode: BatchEvalPython for the row-at-a-time path, ArrowEvalPython for the vectorised one. Because the worker is its own process, its memory sits outside spark.executor.memory and is bounded by spark.executor.pyspark.memory, which is exactly why Python UDFs show up as container kills rather than heap errors. The driver's only involvement is serialising the closure, so a UDF that captures a large object makes every task payload large.
from pyspark.sql.functions import pandas_udf
@pandas_udf('double')
def to_usd(x):
return x * 1.09 # vectorised: ArrowEvalPython, not BatchEvalPythonCommon mistake: Saying a UDF 'runs on the driver' because it is defined in the notebook.
Follow-up they will ask: What does the plan lose the moment a Python UDF appears in it?
The driver JVM has 8 GB and is barely half used, yet your notebook died with a Python memory error. Are those the same memory?
Driver vs executors · tap to open the answer
Say this first: No — the driver node runs a JVM heap and a separate Python process, and toPandas() fills the Python one.
Remember these
Why: spark.driver.memory sizes the JVM heap where the SparkSession, Catalyst, and the scheduler live. Your PySpark code runs in a Python process on the same machine, and collect() or toPandas() moves rows from executors into the JVM first, then copies them across into Python. With Arrow enabled the transfer is columnar and faster, but it still means two live representations of the result at peak, so the node needs roughly double the result size free. Nothing in the JVM heap graph reflects the Python side, so the Executors tab can look calm while the kernel dies. On top of that, spark.driver.maxResultSize caps the total serialised task results an action may return and will abort the job before either process OOMs. Size the driver node's total RAM, not just the heap, whenever a pipeline pulls results into Python.
spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', 'true')
pdf = df.limit(50000).toPandas() # bounded, and the bound is the pointCommon mistake: Raising spark.driver.memory to fix a Python-side MemoryError on the driver node.
Follow-up they will ask: How would you return a 5 GB result to a Python client without this problem?
A job fails with 'Total size of serialized results is bigger than spark.driver.maxResultSize'. Is raising the limit the fix?
Driver vs executors · tap to open the answer
Say this first: Almost never — the setting is a guardrail against pulling a dataset to one JVM, and the real fix is to stop pulling it.
Remember these
Why: maxResultSize bounds the sum of serialised results across all tasks of a single action, so it fires on the aggregate, not on any one task. Small task results come back inline over the RPC layer, bounded by spark.rpc.message.maxSize; larger ones are stored as blocks and fetched by the driver through the block manager, which is why the driver's memory grows while the error talks about serialised bytes. Raising the limit converts a clean, early abort into a driver heap OOM or a GC death spiral somewhere further along. The cases where raising it is legitimate are narrow: a genuinely needed result in the low gigabytes with a driver node sized for several times that. Otherwise restructure the action — aggregate on executors, write to a table and read it back, or take a bounded sample. Setting it to 0 disables the check entirely and is the fastest way to turn a readable error into an unexplained kernel death.
# instead of collecting every key
keys = spark.sql('SELECT key FROM t GROUP BY key')
keys.write.mode('overwrite').saveAsTable('scratch.keys')Senior depth: Treat maxResultSize as a design assertion: 'no action returns more than this much to one JVM.' When it fires, the right artefact is a changed query, not a changed config. The only time I raise it is for a known, bounded export, and then I raise the driver node size in the same change.
Common mistake: Bumping the value until the error stops, then reporting the driver OOM as a separate unrelated incident.
Follow-up they will ask: Which actions besides collect can trip this limit?
A streaming job's driver heap creeps upward over six days and then dies. Executors are healthy the whole time. What is accumulating?
Driver vs executors · tap to open the answer
Say this first: Driver-side bookkeeping — retained UI state, query progress history, and a backlogged listener queue — not your data.
Remember these
Why: The driver keeps in-memory history for the UI: completed jobs, stages, and tasks bounded by spark.ui.retainedJobs, spark.ui.retainedStages, and spark.ui.retainedTasks, plus SQL executions bounded by spark.sql.ui.retainedExecutions. A micro-batch stream produces a query every trigger interval, so at a ten-second trigger you create tens of thousands of executions a day and the default retention is sized for batch jobs, not for that rate. Events reach those listeners through an async queue sized by spark.scheduler.listenerbus.eventqueue.capacity; when a listener is slow, the queue backs up, memory grows, and you start seeing dropped-event warnings in the driver log. Streaming query progress objects and accumulator state add to the same pile, and the ContextCleaner only reclaims broadcasts and shuffles once references are gone. The signature is unmistakable: a sawtooth-free, monotonic heap climb with flat executor memory and no change in input volume. Lower retention, trim custom listeners, and restart on a schedule until the growth is flat.
spark.conf.set('spark.ui.retainedJobs', '200')
spark.conf.set('spark.ui.retainedStages', '200')
spark.conf.set('spark.sql.ui.retainedExecutions', '200')Senior depth: Long-running drivers need a retention budget the same way executors need a memory budget. Any custom QueryExecutionListener or StreamingQueryListener you add is driver code on the hot path — if it writes to an external system synchronously, it will back the event queue up, and the symptom will look like a memory leak rather than a slow HTTP call.
Common mistake: Hunting for a data leak in the streaming logic when no executor metric has moved.
Follow-up they will ask: How would you tell a retention leak from a genuine state-store growth problem?
Dynamic allocation scaled the job to 300 executors and it got slower. Explain the mechanisms.
Driver vs executors · tap to open the answer
Say this first: More executors multiply shuffle fetch connections, broadcast copies, and decommission-driven recomputation, none of which the scan time pays back.
Remember these
Why: A reducer fetches from every map-output location, so going from 30 to 300 executors turns each reducer's fetch fan-in into ten times as many connections, and Shuffle Read Blocked Time starts dominating task duration. Every broadcast is materialised once per executor, so a 400 MB dimension that cost 12 GB cluster-wide now costs 120 GB and the broadcast step itself takes longer to distribute. Scale-down is the sharper edge: without spark.dynamicAllocation.shuffleTracking.enabled or an external shuffle service, removing an executor destroys the shuffle files it produced, and the next reducer that needs them hits FetchFailed, which retries the whole map stage. That produces the classic pattern of a job that oscillates between scaling up and recomputing stages it had already finished. Bounding spark.dynamicAllocation.maxExecutors to something matched to the stage's real partition count usually restores the previous runtime immediately.
spark.conf.set('spark.dynamicAllocation.maxExecutors', '60')
spark.conf.set('spark.dynamicAllocation.shuffleTracking.enabled', 'true')Senior depth: Parallelism is capped by partitions, not by executors. Once executor cores exceed the stage's partition count, extra executors add shuffle topology and broadcast cost while sitting idle. I cap maxExecutors from the widest stage's partition count and treat anything above that as a cost with no upside.
Common mistake: Treating executor count as monotonically helpful and reading repeated stage retries as flaky infrastructure.
Follow-up they will ask: How many executors can a stage with 200 partitions actually use?