Interview/Spark

Spark interview questions: Structured Streaming: an unbounded table

Interactive Spark interview questions on Structured Streaming: an unbounded table. Practice with the matching lesson. A stream is an unbounded table read in micro-batches. Checkpoints give exactly-once; watermarks bound state.

Lesson · Simulation

Your streaming query looks like ordinary DataFrame code. What is the engine actually doing under it?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Structured Streaming: an unbounded table

Streaming aggregation OOMs after four days of clean running

Symptoms

  • Runs fine for days, then executors die and restart in a loop
  • Batch duration climbing steadily in the days before the failure
  • State row count grows every batch and never comes down

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 9 of 9 questions.

beginner

Your streaming query looks like ordinary DataFrame code. What is the engine actually doing under it?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: Running a series of micro-batches: each batch reads a new offset range and executes the same incremental plan.

Why: The engine records which offsets it has processed in the checkpoint's offset log, plans a batch over the new range, and writes a commit when the batch finishes. That is why Spark UI shows a steady stream of small jobs rather than one long-running job.

Common mistake: Describing it as row-by-row streaming, like a message-queue consumer loop.

Follow-up they will ask: What guarantees the same row is not processed twice after a restart?

Lesson · Simulation

intermediate

On-call deleted the checkpoint directory to 'clear a stuck stream'. What did they just do?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: Threw away the offset log, the commit log, and all state — the stream now restarts with no memory of what it processed.

Why: The checkpoint location holds offsets, commits, and the state store, and exactly-once depends on those lining up with the sink. Deleting it means reprocessing from the configured starting position — duplicates into an append sink, or silently skipped data if it starts at latest — and every aggregation's state is gone.

Common mistake: Treating the checkpoint directory as a cache you can safely clear when something looks stuck.

Follow-up they will ask: When is deleting a checkpoint actually the correct move?

Lesson · Simulation

senior

A stateful aggregation ran clean for a week, then started OOMing every few hours. First hypothesis?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: No watermark, so state retained every key forever and finally outgrew the executors.

Why: Without withWatermark the engine cannot know a group will never receive late data, so its state is kept indefinitely. The streaming progress metrics show the state operator's numRowsTotal climbing monotonically and state store commit time growing with it. A watermark lets keys older than the event-time threshold be dropped.

Senior depth: Events later than the watermark are dropped and cannot be retrofitted — you are signing up to a bounded lateness contract, so route stragglers to a repair batch job if the business needs them.

Common mistake: Restarting the stream on a bigger cluster, which just moves the failure a few days out.

Follow-up they will ask: What do you give up when you add the watermark?

Lesson · Simulation

architect

Your foreachBatch writes to a transactional store and a restart happens mid-batch. How do you keep the sink correct?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: Make the write idempotent and keyed, because the same batchId can be re-executed.

Why: foreachBatch hands you the DataFrame and the batchId, and on failure the engine may replay a batch that already wrote part of its rows. Use business keys (or the batchId) in a MERGE / upsert, or record the last committed batchId in the sink transactionally so a replay becomes a no-op. Output mode matters too: append emits only new rows, update emits changed keys, complete rewrites the whole result and only works for small aggregates.

Senior depth: When latency tolerance is hours: AvailableNow consumes everything currently available across several micro-batches and stops, so you keep streaming semantics and checkpointed offsets without paying for a cluster that never sleeps.

Common mistake: Assuming the checkpoint's exactly-once guarantee extends to an arbitrary external sink you wrote yourself.

Follow-up they will ask: When would you pick Trigger.AvailableNow over an always-on stream?

Lesson · Simulation

intermediate

Your windowed aggregation in append mode has written nothing for fifteen minutes, but input rows keep arriving. Is it broken?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: Probably not — append emits a window only after the watermark passes its end, so output lags by the window plus the lateness threshold.

Remember these

  • Append emits a window only after the watermark clears its end
  • The watermark advances from event time in the data, not wall clock
  • Update mode emits earlier but needs an upsert-capable sink

Why: In append mode a row must be final before it can be written, and a window is final only once the watermark has advanced past its end time. With a five-minute window and a ten-minute watermark, nothing can be emitted until event time has moved roughly fifteen minutes past a window's start, so a slow source looks like a stuck query. The watermark also advances from the maximum event time seen in the data, not from wall clock, so a source with no new events never advances it and never releases pending windows. Read the watermark and max event time from the query's lastProgress before assuming a failure. If you need visibility sooner, update mode emits changed windows every batch — at the cost of a sink that can handle upserts.

q = agg.writeStream.outputMode("append").start()
print(q.lastProgress["eventTime"])  # watermark and max event time

Common mistake: Restarting the query because append mode 'produced no output', when it was waiting on the watermark by design.

Follow-up they will ask: What does complete mode do to state size?

Lesson · Simulation

advanced

A batch failed halfway and the stream restarted. Some rows had already reached the Delta sink. Why are there no duplicates?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: The offset log is written before a batch runs and the commit log after, and the sink records which batch it last wrote.

Remember these

  • Offsets are written before the batch, commits after — that gap drives replay
  • Replay reuses the identical offset range, so the batch is deterministic
  • The sink must dedupe on batch id or business key for this to hold

Why: Before executing batch N the engine writes its offset range to the write-ahead offset log in the checkpoint directory. Only when the batch completes does it append to the commit log. On restart the engine finds an offset entry with no matching commit, so it replays batch N over exactly the same offset range — determinism of that range is half the guarantee. The other half is the sink: the Delta sink records the stream's identifier and the last batch id it committed in the table's transaction log, so a replay of an already-written batch is recognised and skipped. That combination is what people mean by end-to-end exactly-once, and it is why the guarantee does not extend automatically to a sink you wrote yourself inside foreachBatch.

(df.writeStream
   .option("checkpointLocation", "/chk/orders")
   .toTable("gold.orders"))

Senior depth: Then you have at-most-once at best. A source that cannot be re-read deterministically — a queue that acknowledges on read, or a file source whose files a retention job deleted before the replay — leaves the engine unable to reconstruct the batch. That is why Kafka retention has to exceed your worst-case outage window, and why file sources need a cleanup policy that respects the stream's committed position.

Common mistake: Attributing exactly-once to the checkpoint alone, with no requirement placed on the sink.

Follow-up they will ask: What breaks this if the source cannot replay the same offsets?

Lesson · Simulation

advanced

Both sides of a stream-stream join have watermarks, and state still grows without bound. What is missing?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: A time-range condition in the join predicate — watermarks alone do not tell Spark when a buffered row can be dropped.

Remember these

  • Watermarks on both sides are necessary but not sufficient
  • The join needs a time-range predicate or an event-time window equality
  • numRowsTotal that never plateaus is the signature

Why: For a stream-stream join the engine buffers rows from both sides because a match may arrive later. To evict a buffered row it must prove no future row can match it, and that needs two things: a watermark on each side's event-time column, and a constraint relating the two event times, either a range condition in the predicate or a join on an event-time window. With watermarks alone the predicate is unbounded in time, so nothing can be proven stale and state keeps every row forever. Add a bounded condition between the two timestamps and the engine can compute an eviction threshold. Watch stateOperators numRowsTotal in the progress report — if it never plateaus after the watermark has advanced, the missing time condition is why.

orders.join(clicks,
  (orders.id == clicks.order_id) &
  (clicks.ts.between(orders.ts, orders.ts + expr("INTERVAL 30 MINUTES"))))

Senior depth: An outer join cannot emit an unmatched row until it is certain no match will arrive, so it holds the row until the watermark passes the range bound and only then emits it with nulls. Outer results are therefore delayed by the full range plus the watermark, and state is strictly larger than an inner join's. Make sure downstream consumers expect that delay before choosing an outer join in a streaming pipeline.

Common mistake: Assuming withWatermark on both inputs is enough for state eviction in a stream-stream join.

Follow-up they will ask: What does an outer stream-stream join add on top of that?

Lesson · Simulation

senior

Batch duration looks fine but the stream is steadily falling behind the source. Where do you look, and what do you change?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: Compare processed rows per second to input rows per second, then bound batch size and attack the per-batch overhead.

Remember these

  • inputRowsPerSecond above processedRowsPerSecond is the backlog signal
  • durationMs separates real work from per-batch overhead
  • Kafka partition count caps source-read parallelism
  • 200 shuffle partitions is usually wrong for a small stateful batch

Why: In the query progress, inputRowsPerSecond above processedRowsPerSecond means the backlog is growing however healthy each batch looks. Break the batch down with durationMs: time in addBatch is real work, while large latestOffset or walCommit values mean per-batch overhead dominates and you are running too many small batches. For Kafka, maxOffsetsPerTrigger bounds rows per batch so a backlog drains predictably instead of in one enormous batch; for files, maxFilesPerTrigger or maxBytesPerTrigger does the same. Then check parallelism: a topic with eight partitions caps source read parallelism at eight tasks no matter the cluster size, and spark.sql.shuffle.partitions left at 200 for a small stateful batch spends the batch on task overhead. Fix the source partition count and the shuffle partition count together.

(spark.readStream.format("kafka")
   .option("maxOffsetsPerTrigger", 500000)
   .option("subscribe", "orders").load())

Senior depth: Because state is partitioned by that number and stored under the checkpoint, so a stateful query fixes it when the checkpoint is created and refuses to start if you change it. Choose it deliberately before the first run — low enough that batches are not dominated by task overhead, high enough to carry future state volume — because changing it later means a state migration or a fresh checkpoint.

Common mistake: Adding executors when the source has fewer partitions than the cluster already has task slots.

Follow-up they will ask: Why is spark.sql.shuffle.partitions awkward to change for a stateful query?

Lesson · Simulation

architect

You need to add a column to a streaming aggregation's grouping key, and the query will not start against the existing checkpoint. What is your migration plan?

Structured Streaming: an unbounded table · tap to open the answer

Say this first: Treat it as a new query with a new checkpoint, resumed at the old query's committed offsets, with the history backfilled in batch.

Remember these

  • Changing the grouping key changes state schema; resume is refused by design
  • Start the new query at the old one's committed offsets, not at latest
  • Backfill history in batch and reconcile before cutting consumers over
  • Version checkpoint paths so a schema change is a deploy, not an outage

Why: The state store is keyed by the grouping key, so changing the key changes the state schema and the engine refuses to resume rather than produce wrong results. There is no in-place migration for that state, so the plan is: stop the old query cleanly, record its last committed offsets from the commit log, start the new query against a new checkpoint directory at exactly those offsets so there is no gap and no overlap, and reprocess the historical window in a batch job that writes the new grain. Run both outputs into separate tables and reconcile counts before cutting consumers over. Build this into the design: one checkpoint per query, checkpoint paths carrying a version suffix, and sinks written through an idempotent MERGE so a replay or an overlap is recoverable rather than a data incident.

# new checkpoint path, source resumed at the old query's committed offsets
(spark.readStream.format("kafka")
   .option("startingOffsets", last_committed_json).load())
# writeStream.option("checkpointLocation", "/chk/orders_v2")

Senior depth: Whenever there is any gap between stopping the old query and starting the new one, because latest skips everything produced in between and nothing reports it — counts are simply lower. Resume from the recorded committed offsets instead, and if you must use latest, pair it with a batch backfill bounded by the two timestamps and reconcile row counts before anyone trusts the table.

Common mistake: Deleting the old checkpoint so the new code starts, then finding the gap or the duplicates in the gold table a day later.

Follow-up they will ask: When does startingOffsets 'latest' quietly lose data?

Lesson · Simulation