One action, four-hour wall clock
Symptoms
- Single job in Spark UI
- Stage 2 is 3.5 hours
- Stage 0–1 were minutes
Interactive Spark interview questions on What happens when Spark executes a job?. Practice with the matching lesson. Trace groupBy + show() from lazy code through the plan, job, stages, tasks, shuffle, and result — one picture.
Question 1 of 23
Walk a Spark action from code to rows without skipping a layer.
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 8 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
Showing 23 of 23 questions.
Walk a Spark action from code to rows without skipping a layer.
What happens when Spark executes a job? · tap to open the answer
Say this first: Code → logical plan → job → stages → tasks → (optional shuffle) → result.
Why: Transformations are lazy. The action builds a plan, splits it at shuffles into stages, runs one task per partition, then returns a small result or writes files.
Common mistake: Saying 'Spark runs the DataFrame line by line'.
Follow-up they will ask: Where does a shuffle sit in that chain?
Why does one action create multiple stages?
What happens when Spark executes a job? · tap to open the answer
Say this first: Each wide transformation (shuffle) is a stage boundary.
Why: Narrow steps (filter, map, broadcast join) stay in one stage. groupBy, join-without-broadcast, repartition cut a new stage because data must move.
Common mistake: Counting jobs instead of stages when debugging runtime.
Follow-up they will ask: How do you see stage boundaries in the physical plan?
A dashboard job has 1 job, 4 stages, 80k tasks. What do you inspect first?
What happens when Spark executes a job? · tap to open the answer
Say this first: Task count and the slowest stage — not executor count.
Why: 80k tasks usually means tiny files or over-partitioning. Stage time is max(task). Find the stage with shuffle or scan skew, then file layout, then AQE/coalesce.
Senior depth: Hundreds to low thousands of healthy tasks, not tens of thousands of 50 ms tasks.
Common mistake: Doubling the cluster because 'there are many tasks'.
Follow-up they will ask: What number of tasks would you aim for on a 1 TB nightly?
At which point in the chain does Spark decide how many tasks a stage gets?
What happens when Spark executes a job? · tap to open the answer
Say this first: At the leaf for scan stages — from file splits and spark.sql.files.maxPartitionBytes — and at the Exchange for every stage after a shuffle.
Remember these
Why: For the first stage, the number of tasks equals the number of input partitions the file source produced. Spark packs file splits into partitions up to spark.sql.files.maxPartitionBytes and charges each file an opening cost of spark.sql.files.openCostInBytes, which is what makes a thousand tiny files produce a thousand mostly-idle tasks instead of one sensible one. For stages after a shuffle, the count comes from the Exchange's partitioning, which defaults to spark.sql.shuffle.partitions. With AQE enabled, that post-shuffle count is then revised at runtime: small adjacent partitions are coalesced toward spark.sql.adaptive.advisoryPartitionSizeInBytes, so the Stages page can show fewer tasks than the configured number. You can read all of this off the UI without guessing — each stage's task count is on the Stages page, and the Exchange nodes are in the SQL / DataFrame tab's plan.
df.rdd.getNumPartitions() # scan-side partitions
spark.conf.get('spark.sql.shuffle.partitions') # post-shuffle defaultCommon mistake: Believing spark.sql.shuffle.partitions controls how many tasks read the source files.
Follow-up they will ask: Why can the Stages page show 48 tasks when shuffle partitions is set to 200?
The SQL / DataFrame tab shows one query, but the Jobs tab lists three jobs for it. Where did the extra jobs come from?
What happens when Spark executes a job? · tap to open the answer
Say this first: A single query can submit several jobs: broadcast exchanges, AQE shuffle materialisation, and limit scale-up each run as their own job.
Remember these
Why: A query ID in the SQL tab is one logical execution, but the scheduler submits jobs whenever it needs a result before continuing to plan or execute. A BroadcastExchange has to collect the small side and build a hash relation before the join can run, and that collection is its own job. With spark.sql.adaptive.enabled, Spark executes shuffle map stages, reads their real output statistics, and only then finalises the downstream plan — so each materialisation boundary appears as a separate job under the same query, which is why the plan root reads AdaptiveSparkPlan with isFinalPlan=true only at the end. A limit can add more jobs through scale-up retries when the first few partitions do not yield enough rows. Click the query in the SQL tab and it lists the associated job IDs, which is the fastest way to attribute them rather than guessing from the Jobs list.
Common mistake: Counting jobs to count user actions, then concluding the notebook ran a cell twice.
Follow-up they will ask: How would you tell an AQE-added job from a second user action?
What does WholeStageCodegen actually fuse, and name two things that break the fusion.
What happens when Spark executes a job? · tap to open the answer
Say this first: It compiles a chain of supported operators into one generated Java method that loops over rows without virtual calls; unsupported operators split it into separate codegen stages.
Remember these
Why: Without codegen, each operator is an iterator calling next() on its child, so a simple filter-and-project pays a virtual call and an object materialisation per row per operator. Catalyst instead generates a single method whose body inlines the whole chain, so the row stays in registers and the JVM can JIT a tight loop; in explain('formatted') these appear as WholeStageCodegen (n) wrappers and the fused operators are marked with an asterisk. The fusion stops wherever an operator does not support codegen. A Python UDF is the clearest breaker: BatchEvalPython sits between two codegen islands because rows must leave the JVM entirely. Columnar sources break it at the boundary too, which is why you see ColumnarToRow between a vectorised FileScan and the first fused operator. Generated code also has limits — past spark.sql.codegen.maxFields or the bytecode size in spark.sql.codegen.hugeMethodLimit, Spark falls back to the interpreted path for that stage, so a 300-column select can quietly lose codegen while the plan still looks reasonable.
df.filter('amount > 100').select('id', 'amount').explain('formatted')
# look for '* Project' and 'WholeStageCodegen (1)'Senior depth: Codegen is the reason expression-level work in the DataFrame API is nearly free and the reason a single UDF changes the cost structure of a whole stage. When I review a slow plan, I count codegen islands first: every extra island is a materialisation boundary somebody introduced, usually a UDF or an unnecessary conversion.
Common mistake: Reading the asterisk as 'this operator is fast' rather than 'this operator is inside a fused codegen stage'.
Follow-up they will ask: How would you detect codegen fallback on a very wide table?
A stage reports Spill (Disk) in the hundreds of gigabytes. Walk from the memory manager to that number.
What happens when Spark executes a job? · tap to open the answer
Say this first: Each task gets a slice of the executor's execution memory pool; when its sort or aggregation exceeds that slice, it writes sorted runs to local disk and merges them.
Remember these
Why: Spark's unified memory manager carves roughly spark.memory.fraction of the heap into a shared pool, split between execution and storage with spark.memory.storageFraction as the storage floor. Execution memory is then shared dynamically across the tasks running concurrently in that executor, so the effective budget per task is close to the execution pool divided by spark.executor.cores. A sort uses UnsafeExternalSorter and an aggregation uses a hash map that grows until its budget is exhausted; at that point the operator serialises what it has as a sorted run on local disk, resets, and continues, then merges the runs at the end. Every byte therefore gets written and read back at least once, which is why a spilling stage's duration is dominated by disk and not CPU. Two numbers tell you which lever to pull: if a few tasks spill enormously while the median spills nothing, it is skew; if every task spills evenly, the shuffle partitions are too few for the data volume. Raising spark.sql.shuffle.partitions or lowering cores per executor both increase per-task budget, from opposite directions.
spark.conf.set('spark.sql.shuffle.partitions', '2000')
# target roughly 128-256 MB per shuffle partitionSenior depth: Spill is not a failure, it is the engine degrading gracefully — the question is whether you are paying for it on purpose. I compare Spill (Disk) against Shuffle Read per task before touching any config, because that ratio distinguishes 'partition slightly too big' from 'one key holds the stage'.
Common mistake: Raising spark.executor.memory while leaving spark.executor.cores high, so the per-task share barely moves.
Follow-up they will ask: Why does lowering spark.executor.cores sometimes make a spilling stage faster?
A stage was retried after a failure, the job succeeded, and the output row count differs from the previous run. How can a retry change results?
What happens when Spark executes a job? · tap to open the answer
Say this first: If a stage's output is not deterministic, recomputing one partition can produce a different split of rows than the partitions downstream already consumed.
Remember these
Why: Spark's recovery model assumes a lost partition can be recomputed identically from its lineage. That assumption breaks when the upstream output order is not stable — repartition() without a key, rand() without a seed, monotonically_increasing_id(), or a source that lists files in a nondeterministic order. If a reducer fails and Spark recomputes only its input partition, rows can land in a different partition than they did the first time, so some rows are processed twice and others not at all. Spark 3 addresses this by marking such stages indeterminate: when one partition of an indeterminate stage must be recomputed, the scheduler invalidates and reruns the whole stage and everything downstream, which is correct but expensive and shows up as a surprising wave of retried stages in the UI. What it cannot fix is work that already escaped the engine, such as a non-idempotent external write inside foreachPartition. So treat 'succeeded after retries' plus 'row count moved' as a determinism bug, not as flaky infrastructure. Use repartition(col) or a seeded rand(), and make external writes idempotent.
df.repartition(200, 'customer_id') # deterministic placement
df.withColumn('r', rand(seed=42)) # reproducible across retriesSenior depth: Determinism is a correctness property of the pipeline, not a Spark setting. In review I look for unkeyed repartition, unseeded randomness, and generated surrogate keys before I look at performance, because those are the bugs that only appear on the night a node dies and then get closed as 'transient'.
Common mistake: Writing it off as a flaky node because the job eventually reported success.
Follow-up they will ask: Which of those nondeterminism sources is safe to keep if you add a checkpoint?
When does Spark actually read the table?
What happens when Spark executes a job? · tap to open the answer
Say this first: On an action — not when you write filter or groupBy.
Why: The lazy plan is a recipe. show(), count(), write(), collect() cook it. Until then Spark has not scanned storage.
Common mistake: Wrapping every line in count() 'to be safe'.
Follow-up they will ask: What does an unresolved logical plan still not know?
Why can Catalyst rearrange your filters after you wrote them in a certain order?
What happens when Spark executes a job? · tap to open the answer
Say this first: The optimizer is allowed to push predicates and prune partitions.
Why: Your code order is not the physical order. Predicate pushdown and partition pruning happen in analysis/optimization. That is why a late filter can still skip files.
Common mistake: Assuming Python line order is the execution order.
Follow-up they will ask: How do you prove a filter was pushed into the scan?
The plan is huge but the job never starts. What failed?
What happens when Spark executes a job? · tap to open the answer
Say this first: Analysis or optimization on the driver — missing table, bad types, or an exploding logical plan.
Why: If Spark UI has no job, the driver is still in Catalyst. Check analysis errors, huge SQL generation, or a view that unions 400 tables.
Common mistake: Looking at executor logs for a job that never submitted.
Follow-up they will ask: Where do you look if explain() itself hangs?
What is a Spark job, exactly?
What happens when Spark executes a job? · tap to open the answer
Say this first: The work triggered by one action.
Why: Each show/count/write/collect is a job (sometimes more with AQE). A job is not a Databricks Workflow. A job contains stages.
Common mistake: Calling the whole notebook 'one Spark job'.
Follow-up they will ask: How many jobs does df.cache(); df.count(); df.count() typically create?
Why can AQE turn one job into extra stages at runtime?
What happens when Spark executes a job? · tap to open the answer
Say this first: AQE can coalesce, switch join strategy, and split skew after it sees sizes.
Why: The first stages run, statistics update, later stages change. Spark UI shows SQL / AQE details. That is expected, not a second user action.
Common mistake: Thinking extra stages mean the user clicked twice.
Follow-up they will ask: Which AQE feature changes a shuffle join into a broadcast?
Two actions on the same cached DataFrame still scan twice. Why?
What happens when Spark executes a job? · tap to open the answer
Say this first: Nothing was cached — cache is lazy until an action, or the plan changed so the cache key missed.
Why: Need an action after cache to materialize. If you then change the frame (extra filter), it is a new plan. Check Storage tab for the cached RDD/DataFrame.
Common mistake: Calling cache() and assuming the next cell is free.
Follow-up they will ask: When is checkpoint better than cache?
What splits a job into stages?
What happens when Spark executes a job? · tap to open the answer
Say this first: A shuffle — a wide dependency.
Why: Narrow transformations pipeline inside a stage. When keys must co-locate (groupBy, sort merge join, repartition), Spark inserts an Exchange and a new stage.
Common mistake: Saying each transformation is its own stage.
Follow-up they will ask: Is a filter a new stage?
Stage time is 20 minutes but most tasks finished in 30 seconds. What is the stage time?
What happens when Spark executes a job? · tap to open the answer
Say this first: The slowest task — the straggler.
Why: Stages wait for all tasks. Median is a vanity metric. Read max task duration, spill, and shuffle read on that task.
Common mistake: Optimizing average task time while one key holds the stage.
Follow-up they will ask: Which UI chart shows this immediately?
You see two Exchanges for one join. Is that always wrong?
What happens when Spark executes a job? · tap to open the answer
Say this first: Not always — both sides may need a shuffle for sort-merge. A broadcast join has zero Exchanges for the big side.
Why: SMJ typically shuffles both inputs unless already co-partitioned. BroadcastHashJoin shuffles neither of the fact side. Prove with explain().
Senior depth: Broadcast the small side, or bucket/co-partition both tables on the join key.
Common mistake: Removing 'an Exchange' without knowing join strategy.
Follow-up they will ask: How would you eliminate both shuffles?
What is the difference between a partition and a task?
What happens when Spark executes a job? · tap to open the answer
Say this first: A partition is a slice of data. A task is the unit of work that reads that slice in a stage.
Why: One partition → one task in that stage. Executors run many tasks over time. Do not call an executor a task.
Common mistake: Using partition, task, and executor as synonyms.
Follow-up they will ask: If you have 200 partitions and 10 executors, how many tasks in a narrow stage?
Why can 10 000 partitions make a small table slower?
What happens when Spark executes a job? · tap to open the answer
Say this first: Task scheduling overhead dwarfs the work. Each task has launch cost.
Why: Tiny files / too many partitions → thousands of 20 ms tasks. Spark UI shows high scheduler delay. Coalesce or compact files.
Common mistake: repartition(10000) 'for more parallelism' on a 200 MB table.
Follow-up they will ask: How do you pick a partition count for a 200 GB shuffle?
Tasks fail with FetchFailed. What layer actually broke?
What happens when Spark executes a job? · tap to open the answer
Say this first: Shuffle fetch — a reducer could not read a map output, often because an executor died or shuffle service lost files.
Why: Not 'Spark is random'. Check executor loss, disk, and whether speculative execution is hiding a bad node. Retry may succeed; repeated FetchFailed means shuffle storage or OOM on the map side.
Senior depth: On the map-side executor local disk (or shuffle service). If that JVM dies hard, reducers fetch-fail.
Common mistake: Increasing retries without looking at executor logs.
Follow-up they will ask: Where are shuffle files stored?
What comes back to the driver after a shuffle?
What happens when Spark executes a job? · tap to open the answer
Say this first: Only what the action asked for — a few show() rows, a count, or nothing for write.
Why: Shuffle moves data between executors. It does not dump the table on the driver. write() commits files; the driver stays small. collect() is the action that pulls.
Common mistake: Believing every shuffle returns the full dataset to the notebook.
Follow-up they will ask: Does saveAsTable send rows through the driver?
Why is shuffle both a network problem and a disk problem?
What happens when Spark executes a job? · tap to open the answer
Say this first: Map tasks write shuffle files locally; reducers fetch over the network; spill hits disk again if a reducer partition is fat.
Why: Spark UI: shuffle write, shuffle read, fetch wait, spill memory/disk. A healthy shuffle is balanced. A sick one has one reducer reading 10× bytes.
Common mistake: Only looking at CPU when a stage is an Exchange.
Follow-up they will ask: What is shuffle partition size aiming at?
spark.sql.shuffle.partitions = 200 on a 4 TB agg. What goes wrong?
What happens when Spark executes a job? · tap to open the answer
Say this first: Each reducer partition is huge — spill, OOM, or hour-long tasks.
Why: 200 is a default, not a law. Target ~128–256 MB shuffle partitions. AQE coalesce helps small data; large data needs more partitions or a better plan (pre-agg, broadcast).
Senior depth: It can coalesce empty/small partitions after seeing map-side sizes — it will not save you from one skewed key.
Common mistake: Leaving 200 forever because 'that's Spark'.
Follow-up they will ask: How does AQE change this number at runtime?