Catalyst to Job — the action and the boundary
Nightly notebook submits 40 jobs
Symptoms
- Many display/count debug cells left in
- Each action rebuilds similar shuffles
Catalyst to Job — the action and the boundary interview questions
Interactive Spark interview questions on Catalyst to Job — the action and the boundary. Same topic as /learn/spark/plan-to-job. show() submits Job 0. groupBy inserts an Exchange — the shuffle boundary that will split stages.
Question 1 of 3
What is a Spark job, exactly?
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerWhat is a Spark job, exactly?
Catalyst to Job — the action and the boundary · tap to open the answer
Short: The work triggered by one action.
Detailed: Each show/count/write/collect is a job (sometimes more with AQE). A job is not a Databricks Workflow. A job contains stages.
Common mistake: Calling the whole notebook 'one Spark job'.
Follow-up: How many jobs does df.cache(); df.count(); df.count() typically create?
intermediateWhy can AQE turn one job into extra stages at runtime?
Catalyst to Job — the action and the boundary · tap to open the answer
Short: AQE can coalesce, switch join strategy, and split skew after it sees sizes.
Detailed: The first stages run, statistics update, later stages change. Spark UI shows SQL / AQE details. That is expected, not a second user action.
Common mistake: Thinking extra stages mean the user clicked twice.
Follow-up: Which AQE feature changes a shuffle join into a broadcast?
seniorTwo actions on the same cached DataFrame still scan twice. Why?
Catalyst to Job — the action and the boundary · tap to open the answer
Short: Nothing was cached — cache is lazy until an action, or the plan changed so the cache key missed.
Detailed: Need an action after cache to materialize. If you then change the frame (extra filter), it is a new plan. Check Storage tab for the cached RDD/DataFrame.
Common mistake: Calling cache() and assuming the next cell is free.
Follow-up: When is checkpoint better than cache?