Why Spark is lazy
Pipeline 'green' in interactive, red in jobs
Symptoms
- Notebook had extra cells that never ran in the job
- Job starts at the action
Why Spark is lazy interview questions
Interactive Spark interview questions on Why Spark is lazy. Same topic as /learn/spark/lazy-evaluation. Spark waits so Catalyst can push filters into scans and skip unused columns. Eager execution would waste I/O.
Question 1 of 3
beginner
Why Spark is lazyWhat does lazy evaluation mean for a DataFrame?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat does lazy evaluation mean for a DataFrame?
Why Spark is lazy · tap to open the answer
Short: Spark records the recipe and waits for an action.
Detailed: That lets Catalyst optimize the whole chain (predicate pushdown, join reorder) instead of running each line. It also means errors can appear late.
Common mistake: Evaluating each transformation when the line runs, like pandas.
Follow-up: How do you force execution without collect()?
intermediateYou persist a DataFrame, then add a filter, then count. Why is cache unused?
Why Spark is lazy · tap to open the answer
Short: The cached plan is the unfiltered one; the new plan does not match, or you never materialized the cache.
Detailed: cache() is lazy. Need an action to populate Storage. A new filter is a different lineage unless you cache after the filter.
Common mistake: Assuming persist survives arbitrary extra operators.
Follow-up: How do you confirm a scan hit cache in Spark UI?
seniorWhen is laziness a production footgun?
Why Spark is lazy · tap to open the answer
Short: When invalid tables, bad schemas, or huge plans only fail at the action — in a job cluster at 2am.
Detailed: Validate schema and table existence early. Unit-test explain() on representative data. Don't leave 15 debug actions that each rerun a shuffle.
Common mistake: Adding count() everywhere as 'validation'.
Follow-up: What is a cheap eager check that isn't a full shuffle?