Interview/Spark

Spark interview questions: Actions that trigger jobs

Interactive Spark interview questions on Actions that trigger jobs. Practice with the matching lesson. Transformations build a DAG. Actions (count, show, collect, write) submit a job.

Lesson · Simulation

Name four Spark actions and what each returns to the driver.

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Actions that trigger jobs

Scheduled job OOMs on the last cell

Symptoms

  • Writes succeeded
  • Last cell display(df)

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 8 of 8 questions.

beginner

Name four Spark actions and what each returns to the driver.

Actions that trigger jobs · tap to open the answer

Say this first: count → a number. show → a few printed rows. collect → all rows. write → nothing (files on storage).

Why: take/limit also pull a small result. foreach runs on executors. The dangerous one is collect on a large frame.

Common mistake: Using collect() as the default way to 'see data'.

Follow-up they will ask: Which action is safest to check a pipeline ran?

Lesson · Simulation

intermediate

Why can count() be expensive even though it returns one integer?

Actions that trigger jobs · tap to open the answer

Say this first: It still executes the full plan, including shuffles.

Why: count after a wide transformation pays the shuffle. It is a cheap result, not a cheap job. Sometimes Spark can optimize count on a scan; not after a join.

Common mistake: Sprinkling count() after every transform in production.

Follow-up they will ask: When is approx_count_distinct the better interview answer?

Lesson · Simulation

senior

display() in Databricks feels harmless. When is it collect in disguise?

Actions that trigger jobs · tap to open the answer

Say this first: When the notebook pulls a large unaggregated result to render.

Why: UI limits help, but a wide table with huge strings still hammers the driver. Prefer aggregations or LIMIT. Job clusters should not display fact tables.

Common mistake: Leaving display(df) in a scheduled notebook.

Follow-up they will ask: How do you QA a 2 TB write without pulling it?

Lesson · Simulation

beginner

df.first() returns instantly on a 2 TB table. Why is that not evidence that the plan is fast?

Actions that trigger jobs · tap to open the answer

Say this first: A limit pushes into the plan, so Spark reads only enough partitions to produce one row and stops.

Remember these

  • first() is take(1), planned as CollectLimit over a few partitions
  • Input Size / Records shows how little was actually read
  • A limit above an aggregate still pays for the aggregate
  • Under a selective filter, the scan expands partition-set by partition-set

Why: first() is take(1), and take is planned as a CollectLimit: Spark runs the job against a small number of partitions, checks whether it has enough rows, and only scans more if it does not. On a wide, unfiltered table the very first partition supplies the row, so the action touches a fraction of a percent of the data and Input Size / Records in the UI confirms it. That tells you nothing about a full scan, and it tells you nothing about the shuffle a groupBy in the same plan would cost, because a limit above an aggregate still has to finish the aggregate. The behaviour also inverts under a selective filter: if almost no partition contains a matching row, Spark keeps expanding how many partitions it scans — governed by spark.sql.limit.scaleUpFactor — and the cheap-looking action becomes a near-full scan across several jobs. Use count() or an explicit aggregate when you want to know what the plan actually costs.

df.first()                 # cheap: one partition is usually enough
df.filter('rare_flag').first()  # may expand to most of the table

Common mistake: Using a fast first() or show() as a smoke test for whether the pipeline's real scan will meet its SLA.

Follow-up they will ask: Why can the same first() be slow on a heavily filtered table?

Lesson · Simulation

intermediate

You need to know how many rows a job wrote. Why not just call count() on the DataFrame before writing?

Actions that trigger jobs · tap to open the answer

Say this first: count() re-executes the whole plan, and the number it returns is not necessarily what the write produced; read the write's own metrics instead.

Remember these

  • count() before write() executes the plan a second time in full
  • The two executions can disagree if the source or plan is non-deterministic
  • Delta DESCRIBE HISTORY exposes numOutputRows for the committed version
  • The write stage's Output Size / Records is the UI equivalent

Why: write() is an action, so a count() before it is a second, independent execution of the same plan — you pay every scan and shuffle twice. Worse, the two executions can disagree whenever the source is still changing or the plan contains anything non-deterministic, so the count becomes a number you log rather than a number you can trust. The writer already records what it did: on Delta, DESCRIBE HISTORY exposes operationMetrics with numOutputRows and numFiles for the version your job committed, and the Spark UI reports Output Size / Records for the write stage. For a programmatic hook that works across formats, register a QueryExecutionListener and read the executed plan's output-rows metric after each query. If you need a validation count for a quality gate, run it against the written table afterwards — then you are measuring the artefact rather than a second guess at it.

df.write.mode('append').saveAsTable('silver.orders')
spark.sql('DESCRIBE HISTORY silver.orders').select('version', 'operationMetrics').show(1, False)

Common mistake: Logging a pre-write count as the row count of the write, and doubling the pipeline's cost to get it.

Follow-up they will ask: How would you make that count a hard quality gate without a third execution?

Lesson · Simulation

advanced

take(1000) on a filtered table submitted eleven jobs. Explain the mechanism.

Actions that trigger jobs · tap to open the answer

Say this first: Limit scale-up: Spark scans a small partition set, finds too few matching rows, multiplies the partition count by spark.sql.limit.scaleUpFactor, and runs again.

Remember these

  • CollectLimit grows the scanned partition set between attempts
  • Each attempt is a separate job under one SQL query ID
  • Rare matching rows make the total work approach a full scan
  • Prunable predicates fix it; a bigger scale-up factor does not

Why: CollectLimit is deliberately incremental. It starts with one partition, and if the rows collected so far are fewer than the requested limit, it retries with the partition count multiplied by spark.sql.limit.scaleUpFactor, repeating until it has enough rows or has exhausted the table. Each of those attempts is a separate job under the same SQL query, which is exactly the pattern you saw: eleven jobs with growing task counts and growing Input Size / Records. When the filter is unselective this terminates on the first or second attempt and the action feels free. When the matching rows are rare or clustered at the end of the file listing, every attempt fails to fill the limit and the total work approaches a full scan plus the overhead of re-planning and re-launching repeatedly. The fix is to make the predicate prunable rather than to tune the factor — a partition filter, Delta data skipping, or clustering on the filter column turns the first attempt into a hit. Raising the scale-up factor only makes each attempt bigger; it does not reduce the number of rows that must be found.

spark.conf.get('spark.sql.limit.scaleUpFactor')
df.filter("event_date = '2024-05-01' AND rare_flag").take(1000)

Senior depth: Scale-up is the engine making a reasonable bet that a limit is cheap. The lesson for pipeline design is that take and show are not constant-cost probes — their cost is a function of predicate selectivity, which is why a sampling helper that works on bronze can quietly become a full scan on a filtered gold table.

Common mistake: Reading the repeated jobs as retries from a failure rather than as limit scale-up attempts.

Follow-up they will ask: Why does adding a partition filter change the number of attempts?

Lesson · Simulation

senior

A team wants to push rows to an external API with foreachPartition. What goes wrong, and what would you do instead?

Actions that trigger jobs · tap to open the answer

Say this first: Task retries and speculation re-run partitions, so the API receives duplicates; the write needs to be idempotent or go through a staged table.

Remember these

  • Task retries and speculation can execute a partition more than once
  • Spark has no transaction boundary around external side effects
  • Idempotency keys or a staged table are the only safe patterns
  • Hundreds of tasks mean hundreds of concurrent connections

Why: foreachPartition runs your function on executors with one call per partition, which correctly amortises connection setup compared with foreach's per-row cost. The problem is the execution model: a task can fail and be retried up to spark.task.maxFailures, and with spark.speculation enabled a slow task can be run twice concurrently and the loser's work is discarded by Spark but not by your API. Spark has no transaction boundary around external side effects, so there is no at-most-once guarantee and partial partitions are visible to the outside world. The design that survives this is idempotency: send a deterministic idempotency key per row so the server can deduplicate, or write to a staging Delta table and have a separate, single-threaded process drain it with committed offsets. Add bounded concurrency and retries with backoff inside the function too, because a few hundred tasks opening connections at once is a load spike the API owner did not agree to. In streaming the equivalent is foreachBatch, which at least gives you a batch id you can use as the idempotency key.

def send(rows):
    with client() as api:
        for r in rows:
            api.put(key=f'{r.id}:{r.version}', body=r.asDict())

df.foreachPartition(send)

Senior depth: I treat any external write from executors as at-least-once by definition and design the receiver accordingly. The alternative that actually scales is to stop writing from executors: land the rows in a table, then let one owner with checkpointed offsets push them out, so retry semantics live in one place you can reason about.

Common mistake: Assuming a successful job means every row was delivered exactly once.

Follow-up they will ask: What would you use as the idempotency key for a streaming source?

Lesson · Simulation

senior

Two actions on the same DataFrame returned two different results in one run. How?

Actions that trigger jobs · tap to open the answer

Say this first: Each action re-executes the plan, so a changing source or a non-deterministic expression yields a different answer the second time.

Remember these

  • Every action re-executes the plan from the source
  • Concurrent appends mean two actions read two table versions
  • current_timestamp() and unseeded rand() differ per execution
  • Pin the input with a Delta version; cache is not a snapshot guarantee

Why: Without a cache or a checkpoint, a DataFrame is a recipe, and every action cooks it again from the source. If a concurrent job appended to the input table between your count() and your write(), the two executions read different table versions and both numbers are correct for their moment. The same thing happens with no concurrency at all when the plan contains current_timestamp(), rand() without a seed, or a filter relative to the wall clock, because those are evaluated per execution. The honest fix is to pin the input rather than to cache the output: read a specific Delta version or timestamp so every action in the run sees one snapshot. Caching makes the symptom less likely but is not a guarantee, since a cached partition can be evicted and recomputed from the source. Logging the Delta version the run consumed alongside the row count is what makes the result reproducible after the fact.

v = spark.sql('DESCRIBE HISTORY bronze.orders').first()['version']
src = spark.read.option('versionAsOf', v).table('bronze.orders')

Senior depth: Reproducibility comes from pinning inputs, not from caching intermediates. Any pipeline whose output I might have to defend reads an explicit table version and records it in the run metadata, so 'why did last Tuesday's number change' has an answer that does not depend on what was in memory.

Common mistake: Adding cache() to make two actions agree, and treating that as a correctness guarantee.

Follow-up they will ask: Why is cache() not a snapshot?

Lesson · Simulation