Say this first: Hash partitioning satisfies a downstream operator's distribution requirement, so the planner can drop its Exchange; round-robin cannot.
Remember these
- EnsureRequirements removes an Exchange when the child partitioning already satisfies it
- Hash partitioning on the key co-locates; round-robin only balances
- Partition counts must match for the downstream Exchange to be dropped
- Count Exchange nodes before and after to prove the optimization landed
Why: Every operator that needs co-location declares a required child distribution, and the EnsureRequirements rule inserts an Exchange only when the child's existing partitioning does not satisfy it. hashpartitioning(customer_id, 200) satisfies a following groupBy, SortMergeJoin, or Window on customer_id with the same partition count, so no second Exchange appears and you have paid for one shuffle instead of two. RoundRobinPartitioning satisfies nothing except 'some distribution', so the next wide operator shuffles again. The practical way to check is to count Exchange nodes in explain('formatted') before and after the change; if the count did not drop, the keyed repartition bought you nothing and you are now paying an extra shuffle for even-sized partitions you could have gotten for free. Round-robin also used to be a correctness hazard on retry because the row-to-partition assignment is not a function of the data, which is why Spark sorts within round-robin partitions before writing shuffle output.
df.repartition(200, "customer_id").groupBy("customer_id").sum("amount").explain("formatted")
Senior depth: The counts no longer match, so the groupBy's required distribution is not satisfied and the planner reinserts an Exchange at 400. You have paid for two shuffles and saved none. Either align the repartition count with the shuffle partition count or let the engine plan the single Exchange itself.
Common mistake: Assuming any repartition before a groupBy saves a shuffle, regardless of columns or count.
Follow-up they will ask: What happens if you repartition to 200 but spark.sql.shuffle.partitions is 400?