DataYatriSpark, seen under the hood
Simulate
databricks
intermediate
Connecting…

Delta Lake

Delta is ACID on object storage: Parquet files plus a transaction log. Time travel, OPTIMIZE, and MERGE all come from that log.

Lesson 5 of 14 · Databricks path

Explain it at my level

  1. Parquet
  2. _delta_log
  3. ACID
  4. Time travel
Watch the canvas:data filelog commitsnapshotLive simulation
Object storagepart-000.parquetpart-001.parquetpart-002.parquet_delta_log/000.jsonadd part-000001.jsonadd part-001002.checkpointstate so farCommit 3 succeededevery reader now sees snapshot 3
Delta = Parquet files + an ordered log that says which files count.