Databricks interview questions: Time travel: reading the commit log backwards

Short: Read the older version: SELECT * FROM gold.revenue VERSION AS OF 812, or TIMESTAMP AS OF '2026-09-12 09:00:00'.

Detailed: Every Delta write appends an ordered JSON commit to _delta_log, and the superseded data files are still referenced by earlier versions, so reading an old version is a metadata read against a different file list — nothing is restored and nothing is written. DESCRIBE HISTORY gold.revenue lists versions, timestamps, and the operation behind each commit so you can pick the right one.

Common mistake: Assuming the data is gone and asking the platform team for a bucket-level restore.

Follow-up: Does that query change the table in any way?

Lesson · Simulation

Short: Use RESTORE TABLE t TO VERSION AS OF 812 — it writes a new forward commit, so the rollback itself is auditable.

Detailed: RESTORE does not rewind the log; it adds a new version whose file list matches the target version, so DESCRIBE HISTORY shows a RESTORE operation and downstream readers just see another new version. Hand-rolling it with INSERT OVERWRITE loses that audit trail and splits the read and write into steps that can interleave with other writers.

Common mistake: Trying to delete JSON files out of _delta_log to 'undo' a commit.

Follow-up: What happens to a structured streaming reader on that table after a RESTORE?

Lesson · Simulation

Short: VACUUM physically deleted the data files those older versions point at, so the commits still exist but the Parquet does not.

Detailed: VACUUM removes files that the current version no longer references once they are older than delta.deletedFileRetentionDuration, and those unreferenced files are exactly what time travel reads. Commit history can outlive the data because delta.logRetentionDuration is a separate knob, so DESCRIBE HISTORY may still list a version you can no longer query.

Senior: Two independent dials. delta.deletedFileRetentionDuration decides how long VACUUM leaves unreferenced data files alone — restorability actually depends on that one. delta.logRetentionDuration decides how long the commit metadata survives. Set the file dial from the recovery requirement, accept that you pay storage for every superseded file for that long, and stop calling it a backup: a dropped table or a deleted storage prefix takes the log with it.

Common mistake: Saying 'time travel goes back 30 days' as if it were a fixed guarantee instead of a function of retention and VACUUM.

Follow-up: Which property do you raise if legal wants 60 days of restorability, and what does it cost?

Lesson · Simulation

Short: In-flight readers and the concurrent writer can fail on missing files, and every earlier version becomes unreadable — that retention floor exists to protect concurrent operations.

Detailed: Delta refuses a retention below the configured threshold unless someone sets spark.databricks.delta.retentionDurationCheck.enabled to false, precisely because a query that resolved its file list minutes ago may still be reading files VACUUM just deleted. Once it lands, only the current version is guaranteed readable, so RESTORE and VERSION AS OF for anything earlier are gone, along with change-feed and streaming replays that needed those files.

Senior: The retention check is a guardrail, not a nuisance — anyone disabling it should be able to name the longest-running reader on that table. I would VACUUM on a schedule with retention at or above the longest read plus the recovery window, keep it out of the write window on hot tables, and get real durability from something VACUUM cannot touch: cloud storage versioning and periodic DEEP CLONE snapshots to a separate location.

Common mistake: Disabling the retention check as a habit because the error message was in the way.

Follow-up: How do you get the storage savings safely instead?

Lesson · Simulation