The Spark DAG
Failure recomputes 14 shuffles
Symptoms
- One executor lost near the end
- Runtime doubles on retry
The Spark DAG interview questions
Interactive Spark interview questions on The Spark DAG. Same topic as /learn/spark/dag. The DAG is the graph of operators. Wide edges are shuffles. Reading the DAG is reading the cost.
Question 1 of 3
beginner
The Spark DAGWhat is the Spark DAG?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat is the Spark DAG?
The Spark DAG · tap to open the answer
Short: The graph of RDD/DataFrame dependencies the scheduler uses to run stages.
Detailed: Narrow edges pipeline. Wide edges (shuffles) cut stages. Lineage lets Spark recompute lost partitions.
Common mistake: DAG as a Databricks workflow graph.
Follow-up: What happens to the DAG when an executor loses a cached partition?
intermediateWhy can a long lineage make a job fragile?
The Spark DAG · tap to open the answer
Short: Recompute after failure replays the whole chain; the DAG is huge.
Detailed: Checkpoint or write Delta to cut lineage. Spark UI DAG visualization gets unreadable after dozens of wide steps — that's a smell.
Common mistake: More cache() to 'shorten' a DAG without an action to materialize.
Follow-up: What's the difference between DAG Scheduler and Task Scheduler?
seniorHow do you use the DAG to pick a broadcast vs a shuffle without guessing?
The Spark DAG · tap to open the answer
Short: Find the join node and whether an Exchange sits above each child.
Detailed: No Exchange on the fact side → broadcast. Exchange on both → SMJ. That's the DAG, not the SQL text.
Common mistake: Reading only the SQL string.
Follow-up: Where is that visible besides explain()?