Interview/Databricks

Databricks interview questions: Delta Live Tables (Lakeflow)

Interactive Databricks interview questions on Delta Live Tables (Lakeflow). Practice with the matching lesson. Declare bronze/silver/gold tables and expectations. DLT infers the graph, checkpoints each table, and stops downstream on failure.

Lesson · Simulation

What is Delta Live Tables (Lakeflow Declarative Pipelines)?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Delta Live Tables (Lakeflow)

Pipeline green, 18% of ids missing in gold

Symptoms

  • expect_or_drop on invalid ids
  • Nobody watched the metric

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 9 of 9 questions.

beginner

What is Delta Live Tables (Lakeflow Declarative Pipelines)?

Delta Live Tables (Lakeflow) · tap to open the answer

Say this first: You declare tables and expectations; DLT infers the graph and runs in order.

Why: dlt.read / @dlt.table instead of a 12-task Workflow of notebooks. Expectations drop or fail bad rows. Still Delta under the hood.

Common mistake: DLT as a replacement for Spark.

Follow-up they will ask: How does DLT know bronze runs before gold?

Lesson · Simulation

intermediate

Expectations vs a filter in a notebook job — why bother?

Delta Live Tables (Lakeflow) · tap to open the answer

Say this first: Expectations are cataloged, metrics are visible, and bad rows can fail the pipeline.

Why: A silent filter hides data quality. DLT event log shows expectation pass rate. That's what on-call wants at 3am.

Common mistake: expect_or_drop everything so the pipeline never fails.

Follow-up they will ask: When is a Workflow of notebooks still better than DLT?

Lesson · Simulation

senior

DLT pipeline is green but gold is wrong. Where do you look?

Delta Live Tables (Lakeflow) · tap to open the answer

Say this first: Expectations that drop rows, a wrong grain in silver, or a live table reading a stale source snapshot.

Why: Event log + UC lineage. Don't start in Spark UI Stages — DLT may have many datasets. Compare bronze counts vs gold. Check if development vs production mode used different data.

Senior depth: Full refresh on a sampled bronze, expect metrics, then prod. Never full-refresh prod as a test.

Common mistake: Scaling the DLT cluster first.

Follow-up they will ask: How do you test a DLT graph locally-ish?

Lesson · Simulation

beginner

What actually changes when you flip a pipeline from development to production mode?

Delta Live Tables (Lakeflow) · tap to open the answer

Say this first: Compute lifecycle and retry behaviour: development keeps the cluster around and does not retry failures, production terminates the cluster and retries.

Remember these

  • Development reuses compute between updates for fast iteration
  • Development disables retries so the first error is the one you see
  • Production terminates compute per update and retries failures
  • The mode does not change the target catalog or schema

Why: In development mode the pipeline's compute is reused between updates for a period rather than being torn down, so the next iteration starts in seconds instead of waiting for a cluster, and failures are not retried so you see the error immediately instead of watching three attempts. In production mode the compute is provisioned for the update and terminated when it ends, and failures are retried according to the service's policy, which is what you want for a scheduled pipeline that hits a transient storage error. That is the whole difference. It is not a different code path, and it is not a different dataset — the catalog and schema a pipeline writes to come from its configuration, so a development-mode pipeline pointed at production targets will happily write to them. If you want a separate sandbox, that is a configuration or Asset Bundle target, not the mode toggle.

Common mistake: Assuming development mode writes somewhere safe, then discovering a half-built table in the production schema.

Follow-up they will ask: How would you give each developer their own isolated target without editing the pipeline by hand?

Lesson · Simulation

intermediate

In a declarative pipeline, when do you declare a streaming table and when a materialized view?

Delta Live Tables (Lakeflow) · tap to open the answer

Say this first: Streaming tables process only new rows from an append-only source and keep state; materialized views keep a query's result correct, recomputing when they cannot do it incrementally.

Remember these

  • Streaming tables need an append-only source and carry checkpoints
  • Materialized views keep a query result correct, recomputing if needed
  • Streaming aggregates cannot revise rows they already emitted
  • Wrong choice means either full recompute or permanently stale results

Why: A streaming table reads its source as a stream and maintains a checkpoint, so each update processes only what arrived since the last one. That is the right shape for bronze ingest from a landing zone and for silver tables that append or apply changes, because cost tracks new data rather than table size. A materialized view is defined by a query over its sources, and the service keeps its result consistent on every update — incrementally when the query shape allows it, by recomputing when it does not. That is the right shape for gold aggregates and for anything joining a dimension that can change retroactively, because a streaming aggregate cannot revise a row it has already emitted. Picking wrongly is expensive in both directions: a materialized view over a 10 TB bronze table may recompute from scratch on every run, while a streaming table over a source that gets updated in place will serve stale answers forever. The decision question is whether the source is append-only and whether late restatements have to be reflected.

CREATE OR REFRESH STREAMING TABLE bronze_orders AS
  SELECT * FROM STREAM read_files('/Volumes/landing/orders', format => 'json');

CREATE OR REFRESH MATERIALIZED VIEW gold_daily AS
  SELECT order_date, sum(amount) FROM silver_orders GROUP BY order_date;

Common mistake: Declaring everything as a streaming table because the word streaming sounds cheaper, then being unable to restate a corrected aggregate.

Follow-up they will ask: Your source occasionally restates yesterday's rows. Which do you pick for silver?

Lesson · Simulation

intermediate

Map expect, expect_or_drop and expect_or_fail onto an SLA. Which one goes on a required business key?

Delta Live Tables (Lakeflow) · tap to open the answer

Say this first: expect records and passes the row, expect_or_drop removes it, expect_or_fail aborts the update — a required key belongs on fail or on a quarantine pattern, never a silent drop.

Remember these

  • All three write pass/fail counts to the event log
  • expect_or_drop is a silent data-loss channel if nobody alerts on the rate
  • expect_or_fail is for violations that make the output wrong, not thin
  • Quarantine tables keep the SLA and keep the rejected rows

Why: All three record pass and fail counts into the pipeline event log, which is the part that matters operationally: an expectation is a metric first and an action second. expect lets the bad row through and only counts it, which suits a soft rule you want to watch before you enforce. expect_or_drop removes the row from the target and counts the drop, which is appropriate for genuinely unusable records as long as somebody is watching the rate. expect_or_fail aborts the table update so nothing downstream publishes, which is the only correct choice when a violation means the output would be wrong rather than incomplete. A required business key sits in that last category, because dropping it means revenue rows quietly disappear and the pipeline stays green — that is how you get an 18% gap nobody notices. The middle path many teams land on is a quarantine pattern: a negated expectation writing failures into their own table so the main pipeline keeps its SLA and the bad rows are still recoverable.

@dlt.expect('amount_present', 'amount IS NOT NULL')
@dlt.expect_or_drop('valid_ts', 'event_ts IS NOT NULL')
@dlt.expect_or_fail('valid_id', 'id IS NOT NULL')

Common mistake: Using expect_or_drop everywhere so the pipeline never goes red, which converts every data problem into a silent row loss.

Follow-up they will ask: How do you alert on a drop rate that creeps from 0.1% to 4% over a month?

Lesson · Simulation

advanced

It is 3am, the pipeline is running, and one table has not updated in two hours. Which surface do you query first, and what do you look for?

Delta Live Tables (Lakeflow) · tap to open the answer

Say this first: The pipeline event log — it records per-flow progress, expectation results, and update state as queryable rows, which driver logs do not.

Remember these

  • event_log() exposes pipeline events as queryable rows
  • flow_progress carries per-flow row counts and backlog metrics
  • data_quality rows are the only record of expectation results
  • Cluster events reveal time spent waiting rather than processing

Why: Every pipeline writes an event log you can query with the event_log table-valued function, or read as a published table if the pipeline is configured that way. Each row has a timestamp, an event_type and a details struct you extract JSON fields from. For a stalled table, filter on flow_progress and read the metrics for that flow: whether batches are completing, how many rows went in, and whether a backlog is growing, which separates 'stuck' from 'slow' immediately. Then check data_quality events for that flow, because an expectation failing on every row looks like no progress downstream. update_progress tells you the update's state transitions, and cluster-related events tell you whether the pipeline spent those two hours waiting for capacity rather than processing. The reason to start here rather than in Spark stages is that a pipeline has many datasets and the event log is the only place that attributes work and quality metrics to a specific flow.

SELECT timestamp, event_type, details:flow_progress.metrics
FROM event_log(TABLE(ops.pipelines.silver_orders))
WHERE event_type = 'flow_progress'
ORDER BY timestamp DESC LIMIT 50

Senior depth: The event log is the pipeline's observability contract, so I would not wait for an incident to discover it. I persist it, build the three queries on-call needs — per-flow freshness, expectation drop rates, and update outcomes — and alert off those rather than off job success, because a declarative pipeline is perfectly capable of succeeding while dropping a fifth of the rows. Resizing compute is the last move, after the log has shown that the flow is making progress and still falling behind.

Common mistake: Going straight to the driver log or resizing the pipeline cluster before establishing whether any flow made progress at all.

Follow-up they will ask: The flow is progressing but the backlog grows every batch. What is your next decision?

Lesson · Simulation

advanced

A teammate clicks 'Full refresh all' on a production pipeline to clear a bad row. What does that do, and when is it unrecoverable?

Delta Live Tables (Lakeflow) · tap to open the answer

Say this first: It truncates the targets and reprocesses from the sources, discarding streaming state — unrecoverable when the source no longer holds the history.

Remember these

  • Full refresh truncates targets and discards streaming state
  • Safe for materialized views whose sources still exist
  • Fatal for streaming tables whose source aged out of retention
  • pipelines.reset.allowed = false protects non-replayable tables

Why: A normal update processes new data; a full refresh throws away the target's contents and the streaming state and rebuilds from the sources. That is fine for a materialized view over an intact silver table, because its sources are still there. It is catastrophic for a streaming table whose source is a landing zone under a lifecycle policy or a message topic past its retention window: the data the table was built from no longer exists, so the rebuild produces whatever is left and the rest is simply gone. The control is the pipelines.reset.allowed table property — set it to false on tables whose sources are not replayable, and a full refresh skips them instead of truncating them. Operationally, refresh a single table rather than all of them, and do it in a development copy first. The deeper point is that a full refresh is only safe when there is a replayable copy of the raw data, which is the actual job of a bronze layer.

@dlt.table(table_properties={'pipelines.reset.allowed': 'false'})
def bronze_orders():
    return spark.readStream.format('cloudFiles').load(landing)

Senior depth: Full refresh is a destructive operation with a reassuring name, so I would make it structurally hard to get wrong rather than relying on care. Bronze tables fed from a landing zone or a topic get pipelines.reset.allowed set to false, which also documents the intent in the code. Silver and gold stay resettable precisely because they can be rebuilt from bronze, and that replay path is worth exercising on a schedule — if you have never rebuilt gold from bronze, you do not actually know that a full refresh is recoverable.

Common mistake: Treating full refresh as a harmless 'rerun' button because it fixed a development pipeline once.

Follow-up they will ask: Which tables in a medallion pipeline would you mark as not resettable, and why only those?

Lesson · Simulation

senior

You have a CDC feed with inserts, updates and deletes that arrive out of order. How do you land it in a declarative pipeline without hand-writing MERGE?

Delta Live Tables (Lakeflow) · tap to open the answer

Say this first: APPLY CHANGES INTO with KEYS, SEQUENCE BY and APPLY AS DELETE WHEN, stored as SCD Type 1 or Type 2 depending on whether you need history.

Remember these

  • SEQUENCE BY is what makes out-of-order changes safe
  • APPLY AS DELETE WHEN turns tombstone rows into real deletes
  • SCD TYPE 2 maintains start and end columns for you
  • You trade merge-predicate tuning for correctness and idempotency

Why: APPLY CHANGES INTO takes a streaming source of change rows and maintains the target for you: KEYS names the identity, SEQUENCE BY names the column that orders changes so an older row arriving late cannot overwrite a newer one, and APPLY AS DELETE WHEN identifies tombstones so a delete flag becomes an actual removal. STORED AS SCD TYPE 1 keeps only current state; TYPE 2 keeps history with start and end columns the service maintains. The target has to be a streaming table, and the service owns the merge logic, idempotency across retries, and out-of-order handling — which is precisely the set of things hand-written MERGE jobs get wrong. What you give up is control: you cannot tune the merge predicate to prune files the way you would in a hand-written statement, so layout work on the target matters more. The honest comparison in an interview is correctness guarantees against tuning freedom, and for most CDC feeds the guarantees are worth more.

APPLY CHANGES INTO LIVE.silver_orders
FROM STREAM(LIVE.bronze_orders)
KEYS (order_id)
APPLY AS DELETE WHEN op = 'D'
SEQUENCE BY seq
STORED AS SCD TYPE 2

Senior depth: The whole correctness of CDC rests on the sequence column, so that is the first thing I interrogate — it has to come from the source's commit order, like a log sequence number or a transaction timestamp, not from when your pipeline happened to read the file. If the source cannot provide one, I would not pretend: either get the upstream to emit it, or accept that the target is eventually-consistent-at-batch-granularity and document that. Where it exists, APPLY CHANGES is the better default than hand-written MERGE, because out-of-order handling and retry idempotency are exactly the parts a human implementation gets wrong six months after the author moves team.

Common mistake: Omitting SEQUENCE BY or pointing it at the ingestion timestamp instead of the source's commit order, so a replayed batch overwrites newer state.

Follow-up they will ask: Your feed has no reliable sequence column. What do you do?

Lesson · Simulation