Simulation/Spark

Educational simulation
Spark

Structured Streaming

Micro-batches, a checkpoint, and a watermark dropping state.

Watch the canvas:offset rangesmicro-batch jobcheckpointstate storewatermarktrigger modeLive simulation
Unbounded input table · source offsetsBatch N−2offsets 0–120 · committedBatch N−1offsets 121–240 · committedBatch Noffsets 241–300 · new onlyTrigger fires → one ordinary batch jobJob for batch Nstages, tasks, shuffles — as usualCheckpoint diroffset log · commit log · state snapshotsState store · one entry per window key10:00–10:05count 1210:05–10:10count 3110:10–10:15count 810:15–10:20count 3Trigger modesdefaultnext batch as soon as one endsprocessingTime='1 minute'predictable batches, idle gapsavailableNowdrain the source, then stopBatch N is a normal Spark job over 60 new offsets.
Each trigger turns new offsets into an ordinary batch job; the checkpoint makes restart safe and the watermark is what stops state from growing forever.