Failure recomputes 14 shuffles
Symptoms
- One executor lost near the end
- Runtime doubles on retry
Interactive Spark interview questions on The Spark DAG. Practice with the matching lesson. The DAG is the graph of operators. Wide edges are shuffles. Reading the DAG is reading the cost.
Question 1 of 8
What is the Spark DAG?
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 3 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
Showing 8 of 8 questions.
What is the Spark DAG?
The Spark DAG · tap to open the answer
Say this first: The graph of RDD/DataFrame dependencies the scheduler uses to run stages.
Why: Narrow edges pipeline. Wide edges (shuffles) cut stages. Lineage lets Spark recompute lost partitions.
Common mistake: DAG as a Databricks workflow graph.
Follow-up they will ask: What happens to the DAG when an executor loses a cached partition?
Why can a long lineage make a job fragile?
The Spark DAG · tap to open the answer
Say this first: Recompute after failure replays the whole chain; the DAG is huge.
Why: Checkpoint or write Delta to cut lineage. Spark UI DAG visualization gets unreadable after dozens of wide steps — that's a smell.
Common mistake: More cache() to 'shorten' a DAG without an action to materialize.
Follow-up they will ask: What's the difference between DAG Scheduler and Task Scheduler?
How do you use the DAG to pick a broadcast vs a shuffle without guessing?
The Spark DAG · tap to open the answer
Say this first: Find the join node and whether an Exchange sits above each child.
Why: No Exchange on the fact side means the small side was broadcast and the big side stays where it is. An Exchange above both children means a shuffle-based SortMergeJoin, so every row of both tables crosses the network. The SQL text tells you nothing here — the same JOIN compiles to either plan depending on the estimated size of the build side against spark.sql.autoBroadcastJoinThreshold. Read the plan, not the query. Under AQE the physical plan can also switch after the first shuffle reports real sizes, so the Spark UI's SQL tab is the version that actually ran.
Common mistake: Reading only the SQL string.
Follow-up they will ask: Where is that visible besides explain()?
Your notebook cell chains five transformations and returns in 40 milliseconds. What did Spark build, and what did it not do?
The Spark DAG · tap to open the answer
Say this first: It extended a graph of operators in the driver and read nothing.
Remember these
Why: Each transformation returns a new DataFrame that records its parent and the operation, so the cell only grows a graph in driver memory — no executor is contacted and no file is opened. The graph is directed because data flows one way, and acyclic because a node never depends on its own output, which is what makes replay after a failure safe. An action such as count() or write() is what hands the final logical plan to Catalyst, which produces a physical plan, after which the DAGScheduler cuts it into stages and submits tasks. That is why the Jobs tab stays empty until the action runs and then suddenly shows a job with several stages. The 40 ms was plan construction and analysis, including resolving your column names against the table schema.
df = spark.table("silver.orders").filter("amount > 0").select("id", "amount")
# nothing in the Jobs tab yet
df.count() # now a job appearsCommon mistake: Saying Spark 'reads the data lazily, row by row' — nothing is read at all until the action.
Follow-up they will ask: Which part of that 40 ms would still fail without a cluster?
The DAG visualization shows a greyed-out stage labelled 'skipped'. Did Spark lose work?
The Spark DAG · tap to open the answer
Say this first: No — it reused shuffle files an earlier job already wrote, so that stage had nothing to recompute.
Remember these
Why: Shuffle map output survives on the executors' local disks after the stage that wrote it finishes, and the driver keeps the map output locations. When a later job needs the same shuffle, the DAGScheduler sees the parent stage's output is already registered and marks it skipped instead of rerunning it. You get this whenever two actions share a prefix of the plan: the second job's early stages are grey and its Shuffle Read comes from existing files. It is the one form of reuse you get without calling cache(). The reuse disappears if an executor dies and no external shuffle service is holding those files, at which point the skipped stage is resubmitted.
agg = df.groupBy("region").count()
agg.count() # job 1 runs the shuffle
agg.show() # job 2: the map stage shows as skippedCommon mistake: Reading 'skipped' as data being dropped, or as a sign something failed.
Follow-up they will ask: How is that reuse different from cache()?
For read → filter → withColumn → groupBy().agg() → write, how many stages do you expect and where exactly is the boundary?
The Spark DAG · tap to open the answer
Say this first: Two stages, cut at the Exchange the aggregation's shuffle introduces.
Remember these
Why: filter and withColumn are narrow: each output partition depends on exactly one input partition, so the scheduler pipelines them into the same task as the scan and whole-stage codegen fuses them into one generated loop. groupBy().agg() needs every row for a key in one place, so the physical plan inserts an Exchange hashpartitioning, which is a wide dependency and therefore a stage boundary. Stage one runs scan → filter → project → partial HashAggregate and writes shuffle files; stage two reads them, runs the final HashAggregate, and writes. In explain("formatted") you can count stages by counting Exchange nodes and adding one. The write itself adds no stage unless it repartitions — a partitionBy that forces a sort on the partition columns does.
df.groupBy("region").sum("amount").explain("formatted")
# count the Exchange nodes to count the stage boundariesSenior depth: State it as a rule and then qualify it. Exchange count plus one holds for a linear plan, but a join has two children, so a branching plan gives you parallel stages the scheduler can run concurrently once their parents finish. AQE can also add stages after the fact, so the count in the UI can exceed what a static explain predicted.
Common mistake: Counting one stage per transformation, as if every operator cost its own pass over the data.
Follow-up they will ask: What would adding a join to the same plan do to the stage count?
Is lineage the same thing as the DAG the DAGScheduler schedules?
The Spark DAG · tap to open the answer
Say this first: No — lineage is the dependency record used for recovery; the scheduled DAG is the stage graph built from the physical plan.
Remember these
Why: Lineage is the chain of parent dependencies every RDD or DataFrame carries, and its purpose is recovery: lose a partition and Spark replays its ancestors to rebuild just that partition. The DAG the scheduler works with is downstream of Catalyst — it is the physical plan grouped into stages at wide dependencies, with one task per partition inside each stage. Catalyst can collapse, reorder, or delete operators from your lineage before the scheduler ever sees it, so the two graphs are not the same shape. The practical consequence is that a long lineage is a recovery and planning cost, while a deep stage graph is a shuffle cost. checkpoint() is the operation that truncates lineage, writing to reliable storage and cutting the parent chain, which is not what cache() does.
spark.sparkContext.setCheckpointDir("/tmp/chk")
df2 = df.checkpoint() # materialized, parent lineage droppedSenior depth: localCheckpoint() writes to executor local disk, so it cuts lineage but is not fault tolerant — lose the executor and the data is gone with no parents left to replay from. Use it inside an iterative loop on a stable cluster, never as the durability story for a long pipeline that has to survive a node failure.
Common mistake: Using cache() to 'cut lineage' — cache keeps every parent so it can recompute evicted blocks.
Follow-up they will ask: When is localCheckpoint() the wrong choice?
A task fails with FetchFailedException. Does the DAGScheduler or the TaskScheduler handle that, and what happens next?
The Spark DAG · tap to open the answer
Say this first: The DAGScheduler, because a missing shuffle block means a whole parent stage has to be resubmitted, not just a task retried.
Remember these
Why: The TaskScheduler owns individual task attempts: it launches them on executors, retries ordinary failures up to spark.task.maxFailures, and handles speculation. A FetchFailedException is different — it says the shuffle output a reduce task needs is gone, usually because the executor that wrote it died. The DAGScheduler unregisters that map output, marks the parent shuffle map stage as failed, and resubmits only its missing partitions before retrying the reduce stage. In the UI this looks like a stage number repeating with an attempt counter and a 'Resubmitted' note, and the job's wall clock jumps far more than one lost task would explain. Running an external shuffle service keeps shuffle files alive past executor death, which is why it matters so much under dynamic allocation.
Senior depth: Dynamic allocation removes idle executors, and if those executors still hold shuffle files the next reduce stage fetch-fails and the map stage is recomputed. Enable the external shuffle service, or on newer Spark set spark.dynamicAllocation.shuffleTracking.enabled, so the scheduler will not reclaim an executor whose shuffle output is still needed.
Common mistake: Raising spark.task.maxFailures to fix fetch failures — the task was never the problem.
Follow-up they will ask: Why does this get worse with dynamic allocation?