BI reads stale or duplicate Parquet
Symptoms
- Someone listed the folder and read *.parquet
- Concurrent MERGE in progress
Interactive Databricks interview questions on Delta Lake. Same topic as /learn/databricks/delta. Delta is ACID on object storage: Parquet files plus a transaction log. Time travel, OPTIMIZE, and MERGE all come from that log.
Question 1 of 3
What is Delta Lake?
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 1 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
What is Delta Lake?
Delta Lake · tap to open the answer
Short: Parquet files plus a transaction log (_delta_log) that gives ACID on object storage.
Detailed: Every write commits JSON (and checkpoints). Time travel is 'read version N'. OPTIMIZE and MERGE use the log to know which files are current.
Common mistake: Delta as a file extension you rename to .delta.
Follow-up: What happens if you read the Parquet files and ignore the log?
Why can two jobs writing the same Delta table not corrupt it the way raw Parquet can?
Delta Lake · tap to open the answer
Short: Optimistic concurrency: each commit names the files added/removed. Conflicts abort.
Detailed: Concurrent overwrites of the same partition can still fail — you retry or isolate. The log is the source of truth, not 'whoever finished last silently'.
Common mistake: Writing Parquet next to Delta and expecting ACID.
Follow-up: How do you time-travel to yesterday 8am?
Readers see partial data during a write. What's broken?
Delta Lake · tap to open the answer
Short: They are not reading through the Delta log — or the writer isn't Delta.
Detailed: Spark parquet. format on a Delta path, a broken symlink, or copying files in S3. DESCRIBE HISTORY should exist. If not, it's not a Delta table.
Senior: VACUUM deletes unreferenced files. Time travel older than retention dies. Never VACUUM 0 hours in prod.
Common mistake: Restarting the cluster to 'clear the partial write'.
Follow-up: When do you VACUUM and what's the footgun?