Pipeline green, 18% of ids missing in gold
Symptoms
- expect_or_drop on invalid ids
- Nobody watched the metric
Interactive Databricks interview questions on Delta Live Tables (Lakeflow). Same topic as /learn/databricks/dlt. Declare bronze/silver/gold tables and expectations. DLT infers the graph, checkpoints each table, and stops downstream on failure.
Question 1 of 3
What is Delta Live Tables (Lakeflow Declarative Pipelines)?
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 1 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
What is Delta Live Tables (Lakeflow Declarative Pipelines)?
Delta Live Tables (Lakeflow) · tap to open the answer
Short: You declare tables and expectations; DLT infers the graph and runs in order.
Detailed: dlt.read / @dlt.table instead of a 12-task Workflow of notebooks. Expectations drop or fail bad rows. Still Delta under the hood.
Common mistake: DLT as a replacement for Spark.
Follow-up: How does DLT know bronze runs before gold?
Expectations vs a filter in a notebook job — why bother?
Delta Live Tables (Lakeflow) · tap to open the answer
Short: Expectations are cataloged, metrics are visible, and bad rows can fail the pipeline.
Detailed: A silent filter hides data quality. DLT event log shows expectation pass rate. That's what on-call wants at 3am.
Common mistake: expect_or_drop everything so the pipeline never fails.
Follow-up: When is a Workflow of notebooks still better than DLT?
DLT pipeline is green but gold is wrong. Where do you look?
Delta Live Tables (Lakeflow) · tap to open the answer
Short: Expectations that drop rows, a wrong grain in silver, or a live table reading a stale source snapshot.
Detailed: Event log + UC lineage. Don't start in Spark UI Stages — DLT may have many datasets. Compare bronze counts vs gold. Check if development vs production mode used different data.
Senior: Full refresh on a sampled bronze, expect metrics, then prod. Never full-refresh prod as a test.
Common mistake: Scaling the DLT cluster first.
Follow-up: How do you test a DLT graph locally-ish?