Lazy Evaluation — Code to Plan
show() is the first time the table-not-found appears
Symptoms
- Transformations 'worked'
- Action throws AnalysisException
Lazy Evaluation — Code to Plan interview questions
Interactive Spark interview questions on Lazy Evaluation — Code to Plan. Same topic as /learn/spark/lazy-to-plan. Why groupBy does nothing until show(), and how Catalyst turns that code into a plan before any byte moves.
Question 1 of 3
beginner
Lazy Evaluation — Code to PlanWhen does Spark actually read the table?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhen does Spark actually read the table?
Lazy Evaluation — Code to Plan · tap to open the answer
Short: On an action — not when you write filter or groupBy.
Detailed: The lazy plan is a recipe. show(), count(), write(), collect() cook it. Until then Spark has not scanned storage.
Common mistake: Wrapping every line in count() 'to be safe'.
Follow-up: What does an unresolved logical plan still not know?
intermediateWhy can Catalyst rearrange your filters after you wrote them in a certain order?
Lazy Evaluation — Code to Plan · tap to open the answer
Short: The optimizer is allowed to push predicates and prune partitions.
Detailed: Your code order is not the physical order. Predicate pushdown and partition pruning happen in analysis/optimization. That is why a late filter can still skip files.
Common mistake: Assuming Python line order is the execution order.
Follow-up: How do you prove a filter was pushed into the scan?
seniorThe plan is huge but the job never starts. What failed?
Lazy Evaluation — Code to Plan · tap to open the answer
Short: Analysis or optimization on the driver — missing table, bad types, or an exploding logical plan.
Detailed: If Spark UI has no job, the driver is still in Catalyst. Check analysis errors, huge SQL generation, or a view that unions 400 tables.
Common mistake: Looking at executor logs for a job that never submitted.
Follow-up: Where do you look if explain() itself hangs?