The Spark DAG
Failure recomputes 14 shuffles
Symptoms
- One executor lost near the end
- Runtime doubles on retry
Spark interview questions: The Spark DAG
Interactive Spark interview questions on The Spark DAG. Practice with the matching lesson. The DAG is the graph of operators. Wide edges are shuffles. Reading the DAG is reading the cost.
Question 1 of 3
beginner
The Spark DAGWhat is the Spark DAG?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat is the Spark DAG?
The Spark DAG · tap to open the answer
Short: The graph of RDD/DataFrame dependencies the scheduler uses to run stages.
Detailed: Narrow edges pipeline. Wide edges (shuffles) cut stages. Lineage lets Spark recompute lost partitions.
Common mistake: DAG as a Databricks workflow graph.
Follow-up: What happens to the DAG when an executor loses a cached partition?
intermediateWhy can a long lineage make a job fragile?
The Spark DAG · tap to open the answer
Short: Recompute after failure replays the whole chain; the DAG is huge.
Detailed: Checkpoint or write Delta to cut lineage. Spark UI DAG visualization gets unreadable after dozens of wide steps — that's a smell.
Common mistake: More cache() to 'shorten' a DAG without an action to materialize.
Follow-up: What's the difference between DAG Scheduler and Task Scheduler?
seniorHow do you use the DAG to pick a broadcast vs a shuffle without guessing?
The Spark DAG · tap to open the answer
Short: Find the join node and whether an Exchange sits above each child.
Detailed: No Exchange on the fact side → broadcast. Exchange on both → SMJ. That's the DAG, not the SQL text.
Common mistake: Reading only the SQL string.
Follow-up: Where is that visible besides explain()?