DataYatriSpark, seen under the hood
Simulate

Simulation/Spark

Educational simulation
Spark

Window functions

Keep every row, add a column. partitionBy is the shuffle.

Watch the canvas:input rowscollapsed groupswindow columnpartition laneexchange / frameLive simulation
Input rowsUS · t1amount 100US · t2amount 140EU · t1amount 90US · t3amount 160EU · t2amount 120groupBy(region).sum(amount)USsum 400EUsum 2105 rows in → 2 rows outts and amount are gonesum(amount).over(w)US · t1no column yetUS · t2no column yetEU · t1no column yetUS · t3no column yetEU · t2no column yetpartitionBy(region) · Exchange hashpartitioningPartition: USt2 140 · t1 100 · t3 160 (no order)🔒Partition: EUt2 120 · t1 90🔒Aggregation is lossy by design. The detail columns have nowhere to go.
groupBy collapses rows; a window keeps them. partitionBy is the shuffle, orderBy gives position, the frame gives the number.