Interview/Spark

Spark interview questions: Jobs, stages, and tasks

Interactive Spark interview questions on Jobs, stages, and tasks. Practice with the matching lesson. An action creates a job. Shuffle boundaries split the job into stages. Each partition in a stage is a task.

Lesson · Simulation

Define job, stage, and task in one breath.

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Jobs, stages, and tasks

On-call: 'the stage failed'

Symptoms

  • They mean a task failed
  • Retries then job fail

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

Define job, stage, and task in one breath.

Jobs, stages, and tasks · tap to open the answer

Say this first: Job = one action. Stage = slice of the DAG between shuffles. Task = one partition in that stage.

Why: This is the vocabulary interviewers use to see if you've opened Spark UI. Mixing them is an instant no-hire for senior roles.

Common mistake: Calling an executor a stage.

Follow-up they will ask: How many tasks in a stage with 400 partitions?

Lesson · Simulation

intermediate

A SQL query shows 3 jobs. Is that a bug?

Jobs, stages, and tasks · tap to open the answer

Say this first: Not necessarily — multiple actions, AQE, or Databricks SQL extra jobs.

Why: count + write is two jobs. AQE can add. Temporary views plus display add more. Map jobs to actions in the notebook.

Common mistake: Assuming one SQL string is always one job.

Follow-up they will ask: Where in Spark UI do you map SQL to jobs?

Lesson · Simulation

senior

How do you explain a 1:N:M relationship on a whiteboard?

Jobs, stages, and tasks · tap to open the answer

Say this first: 1 action → N stages (shuffles + 1) → M tasks per stage (partitions).

Why: Draw: show() → Job 1 → Stage 0 scan/filter → Exchange → Stage 1 aggregate → few rows to driver. Tasks = partitions in each stage, possibly different counts after shuffle.

Common mistake: Drawing one box called 'the cluster'.

Follow-up they will ask: What changes M without changing N?

Lesson · Simulation

beginner

The Spark UI marks a stage as 'skipped'. Did something go wrong?

Jobs, stages, and tasks · tap to open the answer

Say this first: No — skipped means its output was already available, usually from a previous shuffle or a cache, so Spark did not rerun it.

Remember these

  • Skipped means output was reused, not that the stage failed
  • It comes from a prior shuffle's map output or from a cache hit
  • A skipped stage shows zero tasks and no errors
  • Missing skips after caching means your cache is not being matched

Why: When a later job needs a stage whose shuffle output is still on the executors, the DAGScheduler reuses that output instead of recomputing it and shows the stage as skipped with zero tasks. The same happens when a cached RDD or DataFrame covers that part of the plan. It is a good sign: it means you are getting reuse out of work you already paid for, which is exactly what caching and a shared shuffle are supposed to buy. Failures look entirely different — a failed stage carries failed task counts and an error, and a stage that is genuinely not running at all simply never appears. If you expected skipped stages after caching and see full reruns instead, your cache is not matching the plan.

Common mistake: Reporting skipped stages as an incident because the word reads like something was dropped.

Follow-up they will ask: You cached a DataFrame and still see no skipped stages. Where do you look?

Lesson · Simulation

intermediate

A stage has 200 tasks: 199 finish in 3 seconds, one takes 20 minutes. What is the stage duration, and what does that tell you about where to tune?

Jobs, stages, and tasks · tap to open the answer

Say this first: About 20 minutes — a stage ends when its slowest task ends, so the only thing worth tuning is that one task.

Remember these

  • Stage duration is the max task duration, not the mean
  • Summary Metrics max versus median is the skew measurement
  • Idle cores mean more executors cannot help
  • Fix the partition, not the cluster

Why: A stage is complete only when every one of its tasks is complete, and the next stage cannot start until then, so stage wall clock equals the maximum task duration, not the average. That is why averages are actively misleading here: the mean task duration in this stage is around nine seconds and tells you nothing about the twenty minutes you actually waited. The Summary Metrics table on the Stage page is built for this — compare the Max column against the Median column for Duration, Shuffle Read, and Spill, and the ratio names the problem. Adding executors does not help, because 199 cores are already idle waiting; the fix is whatever is making that one partition fat, which is usually key skew, a null or sentinel key, or an unsplittable input.

Common mistake: Quoting average task time in a performance review and concluding the stage is healthy.

Follow-up they will ask: Max and median are close but every task is slow. Now what?

Lesson · Simulation

advanced

Six narrow transformations in a row become one task's work. Explain what happens inside that task and where whole-stage codegen fits.

Jobs, stages, and tasks · tap to open the answer

Say this first: The operators are fused into a single generated loop over the partition's rows, so no intermediate collection is ever materialized.

Remember these

  • Narrow operators in one stage run as one fused loop per partition
  • WholeStageCodegen in the plan marks what got fused, with an id per region
  • Fusion removes per-row virtual calls and intermediate row objects
  • Python UDFs and some operators break the chain into separate codegen regions

Why: Narrow transformations have a one-to-one partition dependency, so the DAGScheduler places them in the same stage and each task runs the entire chain over its own partition. Catalyst then goes further with whole-stage code generation: it compiles the fused operators into a single Java method that pulls a row, applies every projection and filter in sequence, and emits it — which is why the plan shows a WholeStageCodegen node with an id wrapping several operators. The payoff is no virtual function call per operator per row and no intermediate row objects, so the chain costs roughly one pass over the data rather than six. Not everything can be fused: an operator without a codegen implementation, a Python UDF's BatchEvalPython, or a plan too large for the generated-method size limit will break the chain, and you see that as multiple separate WholeStageCodegen nodes or operators sitting outside one entirely. Those break points are worth reading, because an operator that falls out of codegen is often the expensive one.

df.select("id", "amount").where("amount > 0").withColumn("net", F.col("amount") * 0.9).explain("formatted")

Senior depth: Look for operators that are not wrapped in a WholeStageCodegen region, or for several small regions where you expected one. A sudden proliferation of regions after a code change usually points at a newly introduced operator that has no codegen path. You can also disable spark.sql.codegen.wholeStage temporarily to measure what fusion is buying, which is a useful diagnostic and a terrible production setting.

Common mistake: Thinking each transformation writes an intermediate dataset that the next one reads.

Follow-up they will ask: How would you tell from a plan that codegen was disabled or broke?

Lesson · Simulation

advanced

Tasks compute in 200 ms but Scheduler Delay is 8 seconds. What creates scheduler delay and what do you do about it?

Jobs, stages, and tasks · tap to open the answer

Say this first: It is the time spent getting the task to the executor and the result back — serialization, dispatch, and driver contention — and it means your tasks are too small or your closure is too big.

Remember these

  • Scheduler Delay is dispatch and result transport, not compute
  • Too many tiny tasks makes the driver the bottleneck
  • A large captured variable is serialized into every task
  • Broadcast large lookups so they ship once per executor

Why: Scheduler Delay on the Stage page is the task's wall clock minus the time actually spent executing and deserializing, so it covers the driver serializing and dispatching the task, the executor waiting for a free slot, and the result travelling back. Two causes dominate. One is sheer task count: a driver dispatching hundreds of thousands of tiny tasks becomes the bottleneck, and the remedy is fewer, larger partitions. The other is a fat task closure — a Python function or Scala lambda that captures a large lookup structure gets serialized into every single task, so a 50 MB capture across 20,000 tasks is a terabyte of dispatch. The fix there is to broadcast the structure once with sparkContext.broadcast and reference the broadcast variable, so it ships per executor rather than per task. Driver GC pressure produces the same symptom, so check driver memory and GC time before concluding anything.

lookup_bc = spark.sparkContext.broadcast(big_dict)
rate = F.udf(lambda k: lookup_bc.value.get(k, 1.0), "double")

Senior depth: Read Task Deserialization Time alongside Scheduler Delay in Summary Metrics. A large per-task deserialization time with a modest task count points at a fat closure; near-zero deserialization time with an enormous task count points at dispatch volume. The driver log also prints a warning when a task binary is unusually large, which is the cheapest confirmation available.

Common mistake: Adding executors, which increases the number of things the already-saturated driver has to coordinate.

Follow-up they will ask: How do you confirm the closure size is the problem rather than the task count?

Lesson · Simulation

senior

You wrote one groupBy and the job shows seven stages. Where did the other stages come from?

Jobs, stages, and tasks · tap to open the answer

Say this first: Every shuffle boundary, broadcast build, sort, and AQE re-plan adds stages that your code never names.

Remember these

  • Stages come from the physical plan, not from the number of wide calls in code
  • Each Exchange, broadcast build, and global sort contributes a boundary
  • AQE materializes query stages incrementally and can add more
  • Map plan nodes to stages in the SQL tab rather than counting operators

Why: Stage count is a property of the physical plan, not of how many wide operations you typed. Each Exchange cuts a boundary, so a join adds one or two, a window adds its own, and a global sort adds an exchange plus a sort stage. A BroadcastExchange is built in its own job and stage before the probe side runs. AQE re-plans between stages and can submit additional stages as it materializes query stages one at a time. Reading a source with schema inference submits a stage before your query starts, and on Delta or similar formats the commit and statistics work shows up too. The reliable way to account for it is to open the SQL tab, click the query, and walk the plan node by node mapping each Exchange and each broadcast to the stage it produced — that mapping is the skill, and it is also what tells you which of those seven stages is worth your attention.

Senior depth: Sort the stages by duration and start with the one that owns the largest share of wall clock, then read its Summary Metrics to decide whether it is a data-volume problem, a skew problem, or an overhead problem. Optimizing a stage that accounts for four percent of the runtime is the most common way to spend a sprint on nothing.

Common mistake: Counting groupBys and joins in the source code and asserting the stage count from that.

Follow-up they will ask: Of those seven, how do you pick the one to optimize?

Lesson · Simulation