Why Spark is lazy
Pipeline 'green' in interactive, red in jobs
Symptoms
- Notebook had extra cells that never ran in the job
- Job starts at the action
Spark interview questions: Why Spark is lazy
Interactive Spark interview questions on Why Spark is lazy. Practice with the matching lesson. Spark waits so Catalyst can push filters into scans and skip unused columns. Eager execution would waste I/O.
Question 1 of 3
beginner
Why Spark is lazyWhat does lazy evaluation mean for a DataFrame?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat does lazy evaluation mean for a DataFrame?
Why Spark is lazy · tap to open the answer
Short: Spark records the recipe and waits for an action.
Detailed: That lets Catalyst optimize the whole chain (predicate pushdown, join reorder) instead of running each line. It also means errors can appear late.
Common mistake: Evaluating each transformation when the line runs, like pandas.
Follow-up: How do you force execution without collect()?
intermediateYou persist a DataFrame, then add a filter, then count. Why is cache unused?
Why Spark is lazy · tap to open the answer
Short: The cached plan is the unfiltered one; the new plan does not match, or you never materialized the cache.
Detailed: cache() is lazy. Need an action to populate Storage. A new filter is a different lineage unless you cache after the filter.
Common mistake: Assuming persist survives arbitrary extra operators.
Follow-up: How do you confirm a scan hit cache in Spark UI?
seniorWhen is laziness a production footgun?
Why Spark is lazy · tap to open the answer
Short: When invalid tables, bad schemas, or huge plans only fail at the action — in a job cluster at 2am.
Detailed: Validate schema and table existence early. Unit-test explain() on representative data. Don't leave 15 debug actions that each rerun a shuffle.
Common mistake: Adding count() everywhere as 'validation'.
Follow-up: What is a cheap eager check that isn't a full shuffle?