Actions that trigger jobs
Scheduled job OOMs on the last cell
Symptoms
- Writes succeeded
- Last cell display(df)
Actions that trigger jobs interview questions
Interactive Spark interview questions on Actions that trigger jobs. Same topic as /learn/spark/actions. Transformations build a DAG. Actions (count, show, collect, write) submit a job.
Question 1 of 3
beginner
Actions that trigger jobsName four Spark actions and what each returns to the driver.
Answer it out loud, then reveal. Play steps through like the simulators.
Production scenario
Case 1 of 1 · symptoms
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
beginnerName four Spark actions and what each returns to the driver.
Actions that trigger jobs · tap to open the answer
Short: count → a number. show → a few printed rows. collect → all rows. write → nothing (files on storage).
Detailed: take/limit also pull a small result. foreach runs on executors. The dangerous one is collect on a large frame.
Common mistake: Using collect() as the default way to 'see data'.
Follow-up: Which action is safest to check a pipeline ran?
intermediateWhy can count() be expensive even though it returns one integer?
Actions that trigger jobs · tap to open the answer
Short: It still executes the full plan, including shuffles.
Detailed: count after a wide transformation pays the shuffle. It is a cheap result, not a cheap job. Sometimes Spark can optimize count on a scan; not after a join.
Common mistake: Sprinkling count() after every transform in production.
Follow-up: When is approx_count_distinct the better interview answer?
seniordisplay() in Databricks feels harmless. When is it collect in disguise?
Actions that trigger jobs · tap to open the answer
Short: When the notebook pulls a large unaggregated result to render.
Detailed: UI limits help, but a wide table with huge strings still hammers the driver. Prefer aggregations or LIMIT. Job clusters should not display fact tables.
Common mistake: Leaving display(df) in a scheduled notebook.
Follow-up: How do you QA a 2 TB write without pulling it?