BI reads stale or duplicate Parquet
Symptoms
- Someone listed the folder and read *.parquet
- Concurrent MERGE in progress
Interactive Databricks interview questions on Delta Lake. Practice with the matching lesson. Delta is ACID on object storage: Parquet files plus a transaction log. Time travel, OPTIMIZE, and MERGE all come from that log.
Question 1 of 9
What is Delta Lake?
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 3 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
Showing 9 of 9 questions.
What is Delta Lake?
Delta Lake · tap to open the answer
Say this first: Parquet files plus a transaction log (_delta_log) that gives ACID on object storage.
Why: Every write commits JSON (and checkpoints). Time travel is 'read version N'. OPTIMIZE and MERGE use the log to know which files are current.
Common mistake: Delta as a file extension you rename to .delta.
Follow-up they will ask: What happens if you read the Parquet files and ignore the log?
Why can two jobs writing the same Delta table not corrupt it the way raw Parquet can?
Delta Lake · tap to open the answer
Say this first: Optimistic concurrency: each commit names the files added/removed. Conflicts abort.
Why: Concurrent overwrites of the same partition can still fail — you retry or isolate. The log is the source of truth, not 'whoever finished last silently'.
Common mistake: Writing Parquet next to Delta and expecting ACID.
Follow-up they will ask: How do you time-travel to yesterday 8am?
Readers see partial data during a write. What's broken?
Delta Lake · tap to open the answer
Say this first: They are not reading through the Delta log — or the writer isn't Delta.
Why: Spark parquet. format on a Delta path, a broken symlink, or copying files in S3. DESCRIBE HISTORY should exist. If not, it's not a Delta table.
Senior depth: VACUUM deletes unreferenced files. Time travel older than retention dies. Never VACUUM 0 hours in prod.
Common mistake: Restarting the cluster to 'clear the partial write'.
Follow-up they will ask: When do you VACUUM and what's the footgun?
A nightly append failed with a schema mismatch because the source added a column. What are your options, and what does each cost?
Delta Lake · tap to open the answer
Say this first: Reject the file and fix upstream, or allow the column with mergeSchema; overwriteSchema is the destructive option that drops columns.
Remember these
Why: Delta validates the incoming DataFrame schema against the table schema on write, and that enforcement is the reason an unexpected column fails the job instead of silently landing nulls or writing files the table cannot describe. Adding .option("mergeSchema", "true") commits the new column into the table schema and reads existing rows as null for it, which is cheap and auditable in DESCRIBE HISTORY. overwriteSchema on an overwrite is different: it replaces the schema, so a column missing from today's source is removed for everyone, which is how a single upstream typo deletes a field. The layer decides the policy — bronze usually accepts evolution because losing data is worse, while gold wants the hard failure because dashboards depend on its contract.
df.write.format("delta").mode("append")
.option("mergeSchema", "true")
.saveAsTable("ops.bronze.orders_raw")Common mistake: Enabling mergeSchema globally so every upstream typo becomes a permanent table column.
Follow-up they will ask: Where in a medallion design do you want enforcement and where do you want evolution?
A small table got steadily slower to read and _delta_log now holds hundreds of thousands of JSON files. Explain the mechanism.
Delta Lake · tap to open the answer
Say this first: Reconstructing the current version means replaying commits since the last checkpoint, so a log with a huge commit tail makes every query pay metadata time before it reads data.
Remember these
Why: Each write appends one numbered JSON commit, and periodically Delta writes a Parquet checkpoint that collapses the accumulated state so readers can start from it and replay only the commits after it. A job committing every few seconds inflates the log far faster than checkpointing and cleanup can compress it, so the planning phase has to list and read a large number of small objects before a single scan task launches. In the query profile that shows up as time before the first scan, and on the driver as listing work and memory pressure, which is why it looks like the cluster is idle at the start of every query. delta.logRetentionDuration governs how long expired commits survive cleanup, so the folder is bounded in principle but only if commits are not arriving faster than they expire. The real fix is fewer, larger commits: raise the streaming trigger interval, batch the micro-writes, and compact.
DESCRIBE DETAIL ops.silver.orders;
DESCRIBE HISTORY ops.silver.orders LIMIT 20;Common mistake: Deleting old JSON files out of _delta_log by hand to tidy the folder.
Follow-up they will ask: What makes a streaming write produce thousands of commits per hour?
What does Delta's optimistic concurrency actually compare, and why do two writers working on different dates still collide?
Delta Lake · tap to open the answer
Say this first: It compares the files each commit read and wrote; without a partition or predicate boundary the writers' file sets overlap, so one fails with ConcurrentAppendException.
Remember these
Why: A writer snapshots the table, plans its work against that snapshot, and at commit time checks whether another commit has since added or removed files that would have changed what it read. If so, it aborts rather than producing a result based on a stale view — that abort is the correctness guarantee, not a bug. ConcurrentAppendException specifically means a concurrent commit added files inside the region your predicate covered, and Delta cannot prove your two writers are disjoint unless the table layout or the predicate tells it so. That is why the fix is structural: partition by the write boundary, or include the partition predicate in the MERGE condition so each writer's file set is provably separate. Retrying blindly succeeds often enough to hide the design problem until the retry count grows.
MERGE INTO ops.silver.orders t
USING staged s
ON t.event_date = '2026-10-23' AND t.order_id = s.order_idCommon mistake: Assuming Delta locks rows, so writers touching different rows can never conflict.
Follow-up they will ask: Which exception do you get instead when the other writer deleted files you were reading?
A DELETE on a billion-row table finished in seconds and barely rewrote any files. What happened, and what does it mean for readers?
Delta Lake · tap to open the answer
Say this first: Deletion vectors recorded the removed rows as per-file bitmaps instead of a copy-on-write rewrite, and readers apply the vectors during the scan.
Remember these
Why: With deletion vectors enabled, DELETE, UPDATE, and MERGE can mark rows as removed in a small side file rather than rewriting every Parquet file that contained a matched row, turning a large copy-on-write rewrite into a metadata-sized write. The cost moves to the read path: scans must merge the vectors in, so a table that accumulates many vectors reads more slowly until OPTIMIZE materialises the changes into clean files. Enabling the feature raises the table's reader and writer protocol requirements, so an external engine that does not understand deletion vectors can no longer read the table correctly — that is the compatibility decision you are making. DESCRIBE DETAIL shows the enabled table features, and OPTIMIZE is the maintenance that purges accumulated vectors.
ALTER TABLE ops.silver.orders
SET TBLPROPERTIES ('delta.enableDeletionVectors' = 'true');Senior depth: Deletion vectors move cost from write time to read time, which is the right trade for update-heavy tables and the wrong one for a table that is written rarely and scanned constantly. So I enable them where MERGE rewrite amplification is the bottleneck, keep OPTIMIZE on a schedule so the read-side cost does not accumulate, and check the consumer list before flipping any protocol-raising feature. The question I always ask first is who reads this table outside Databricks, because protocol upgrades are not reversible in practice.
Common mistake: Enabling table features on a table that an older external reader still consumes, then being surprised by a protocol error.
Follow-up they will ask: Why do you still schedule OPTIMIZE on a table that uses deletion vectors?
A downstream team wants row-level changes from a silver table and proposes diffing two snapshots. What do you offer instead?
Delta Lake · tap to open the answer
Say this first: Change Data Feed: enable delta.enableChangeDataFeed and read table_changes(), which returns inserted, updated, and deleted rows per version.
Remember these
Why: With CDF enabled, Delta records the change type alongside each commit, so table_changes('ops.silver.orders', 120) returns rows carrying _change_type, _commit_version, and _commit_timestamp. That is exactly the input a downstream MERGE wants: only the deltas, with deletes represented explicitly and updates visible as before and after images. A snapshot diff cannot match that — it costs a full scan of both versions, it cannot distinguish an update from a delete-plus-insert, and it silently collapses a row that changed twice inside the window. The constraints to state: CDF only covers commits from the version where you enabled it, and the change data is stored in the table's storage, so VACUUM and retention apply to it the same way they apply to time travel. Downstream consumers therefore need to keep up, or you need retention long enough to cover their lag.
ALTER TABLE ops.silver.orders
SET TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true');
SELECT * FROM table_changes('ops.silver.orders', 120, 125);Senior depth: CDF turns a pull-the-whole-table integration into an incremental contract, which is the architectural win: the downstream job's cost becomes proportional to change volume instead of table size. The discipline it demands is that consumers cannot fall behind retention, so I treat the consumer lag as a monitored SLA and size retention above it with margin. If a consumer genuinely needs an arbitrary historical window, that is a deep clone or a replay from bronze, not CDF.
Common mistake: Enabling CDF and expecting changes from before the enable commit to be readable.
Follow-up they will ask: What happens to change data after VACUUM runs?
A team wants to keep writing plain Parquet and add Delta later. Make the argument with mechanisms, not slogans.
Delta Lake · tap to open the answer
Say this first: Without the log there is no atomic commit, no schema enforcement, no isolated reader snapshot, and no file statistics — so every consumer re-implements correctness.
Remember these
Why: A Parquet directory's state is whatever the object store happens to list, so a reader can see a half-finished job, a failed job leaves orphan files nobody will remove, and two writers to the same prefix have no conflict detection at all. The _delta_log makes the file list itself the transaction: readers resolve a version and see a consistent snapshot, writers commit atomically after a conflict check, and each commit carries per-file min and max statistics that drive data skipping. That same metadata is the foundation for MERGE, time travel, Change Data Feed, and OPTIMIZE, so 'add it later' means not having any of them while the dataset and the number of consumers grow. The migration cost argument is also weaker than people think: CONVERT TO DELTA writes a log over existing Parquet in place, without rewriting data. The honest framing is that Delta is not a competing file format — Delta data files are Parquet, plus a log that makes the directory a table.
CONVERT TO DELTA parquet.`s3://lake/orders`;
DESCRIBE HISTORY delta.`s3://lake/orders`;Senior depth: I make it a cost-of-correctness argument rather than a feature list. Every guarantee the log provides is one somebody has to reimplement in every reader and writer: partial-write visibility, retry semantics, schema drift, orphan cleanup, and statistics for pruning. That work is unpaid, inconsistent across teams, and only discovered during an incident. Since CONVERT TO DELTA is an in-place metadata operation, the real decision is not whether to pay the migration cost but how many ungoverned consumers you want to untangle before you do.
Common mistake: Framing Delta and Parquet as rival file formats when Delta's data files are Parquet.
Follow-up they will ask: What breaks if someone drops files into that directory after the conversion without going through Delta?