DataYatriSpark, seen under the hood
Simulate
spark/internals
advanced
Connecting…

Structured Streaming: an unbounded table

A stream is an unbounded table read in micro-batches. Checkpoints give exactly-once; watermarks bound state.

Lesson 24 of 29 · Spark path

Explain it at my level

  1. Micro-batch
  2. Checkpoint
  3. State
  4. Watermark
  5. Triggers
Watch the canvas:offset rangesmicro-batch jobcheckpointstate storewatermarktrigger modeLive simulation
Unbounded input table · source offsetsBatch N−2offsets 0–120 · committedBatch N−1offsets 121–240 · committedBatch Noffsets 241–300 · new onlyTrigger fires → one ordinary batch jobJob for batch Nstages, tasks, shuffles — as usualCheckpoint diroffset log · commit log · state snapshotsState store · one entry per window key10:00–10:05count 1210:05–10:10count 3110:10–10:15count 810:15–10:20count 3Trigger modesdefaultnext batch as soon as one endsprocessingTime='1 minute'predictable batches, idle gapsavailableNowdrain the source, then stopBatch N is a normal Spark job over 60 new offsets.
Each trigger turns new offsets into an ordinary batch job; the checkpoint makes restart safe and the watermark is what stops state from growing forever.