DataYatriSpark, seen under the hood
Simulate
databricks
intermediate
Connecting…

Delta MERGE

MERGE INTO upserts a change set into a Delta table by rewriting only the files that the join key touches.

Explain it at my level

  1. Target
  2. Match
  3. Rewrite
  4. Commit
Watch the canvas:targetsource changesrewritten fileLive simulation
Target · ordersid 1id 2id 3Source · changesid 2 updid 4 newWHEN MATCHED UPDATE · WHEN NOT MATCHED INSERT · WHEN MATCHED DELETEFiles after MERGEpart-000untouchedpart-001rewritten (id 2)part-002added (id 4)Commit 12 · atomic snapshotadds + removes in one log entry
MERGE rewrites files, then the log swaps the snapshot.