Databricks interview questions: Lakeflow: ingest, transform, orchestrate

Say this first: Lakeflow is the name over Connect, Declarative Pipelines, and Jobs — mostly a unification of Delta Live Tables and Workflows, plus managed ingestion.

Why: Three products that used to be separate sit under one name. Lakeflow Connect is managed connectors that pull from SaaS systems and databases. Lakeflow Declarative Pipelines is what was Delta Live Tables: you declare tables and the engine runs the graph. Lakeflow Jobs is what was Databricks Workflows: a DAG of tasks with retries and repair. Existing DLT pipelines and jobs keep working under the new names. The new thing to adopt, if you need it, is Connect, which replaces the extractor you would otherwise write for Salesforce or SQL Server. There is no flag day where pipelines must be rewritten.

Common mistake: Scheduling a migration project to move working DLT pipelines onto Lakeflow.

Follow-up they will ask: So what would you actually adopt Lakeflow Connect for?

Say this first: Files landing in object storage are Auto Loader. A SaaS API or a database is Lakeflow Connect, because someone has to speak that system's incremental protocol.

Why: Auto Loader watches cloud storage, tracks which files it has seen in its checkpoint, and incrementally reads new ones. It knows nothing about Salesforce's API, cursors, or rate limits. Connect is a managed connector for a specific source: it handles authentication, incremental extraction, and change data capture, and lands the result as tables. Using Connect for files adds a service you do not need. Using a hand-rolled notebook for Salesforce means you own pagination, schema drift, and the retry policy forever. The split is by source shape, not by which product is newer.

Common mistake: Replacing a working Auto Loader path with Connect because Lakeflow is the new standard.

Follow-up they will ask: Where does the connector's incremental state live, and what happens if you reset it?

Say this first: The engine reads the queries, sees which tables each one reads, and infers the dependency graph.

Why: Each declaration is a table name plus the query that produces it. If silver reads bronze and gold reads silver, that is an edge, and the engine topological-sorts the graph before it runs anything. You do not write depends_on between tables inside the pipeline. The engine also owns the checkpoint and the incremental state for streaming tables, and it retries a failed table rather than you coding that. Expectations declared on a table — CONSTRAINT ... EXPECT — are evaluated as the data flows, and a failed expectation can drop the row, fail the row, or just record it. The graph you see in the pipeline UI is that inferred graph, not a diagram someone drew.

CREATE OR REFRESH STREAMING TABLE bronze.orders AS
SELECT * FROM STREAM read_files('/mnt/raw/orders', format => 'json');

CREATE OR REFRESH MATERIALIZED VIEW silver.orders AS
SELECT * FROM bronze.orders
WHERE order_id IS NOT NULL

Common mistake: Adding a job task per table to force an order the pipeline already infers.

Follow-up they will ask: What happens to gold if silver's expectation is set to fail the pipeline?

Say this first: In a Lakeflow Job. The pipeline is one task in the job, with the export task before it and the report task after it.

Why: A declarative pipeline is the right tool for a graph of tables. It is the wrong tool for 'wait for an external system, then run this, then send that'. A job is a DAG of tasks with depends_on, and a pipeline can be one of those tasks. The export task runs first; the pipeline task depends on it and runs the whole table graph as a unit; the report task depends on the pipeline. If the pipeline fails, downstream tasks are skipped, and a repair run reruns the failed task and anything that was skipped — the export does not have to be fetched again. Putting the export and the email inside the pipeline couples operational chores to the table graph.

Senior depth: Repair marks the export task as already succeeded and starts from the failed pipeline task, then runs the report if the pipeline succeeds. Re-running the export would duplicate data unless that task is idempotent, which is why repair exists.

Common mistake: Encoding the whole company workflow as tables inside one pipeline so there is only one thing to schedule.

Follow-up they will ask: The export task succeeded and the pipeline failed. What exactly does repair run execute?

Say this first: Streaming tables and materialized views are implemented on the declarative pipeline engine, so creating one from SQL creates a pipeline to run it.

Why: CREATE MATERIALIZED VIEW in DBSQL does not store a passive object. It declares a query whose result must be refreshed, and the runtime for that declaration is a Lakeflow Declarative Pipeline, created for you and visible in the pipelines list. That is why the object has an event log, a refresh history, and a schedule, and why serverless shows up on the bill. The same engine was open-sourced into Apache Spark, which is why the syntax is not a proprietary side path. If you delete the pipeline, you delete the thing that refreshes the view. Look at the pipeline's event log when a refresh was slow — it records whether the refresh was incremental or a full recompute.

Senior depth: The event log and the refresh history name the refresh type. A COMPLETE refresh on a view you expected to be incremental is the signal to look at the SQL — usually a non-deterministic function — rather than at the warehouse size.

Common mistake: Deleting the unfamiliar pipeline as clutter and then wondering why the view went stale.

Follow-up they will ask: Where in that pipeline do you see whether yesterday's refresh recomputed the whole table?

Say this first: Connect for a SaaS or database source, Auto Loader for files in storage, a declarative pipeline for the table graph, and a job for anything that is not itself a table.

Why: Connect owns the protocol of an external system, so use it when the source is Salesforce, SQL Server, or something with its own incremental API. Auto Loader owns files appearing in a bucket, which Connect is not for. Once data is in the lakehouse, a declarative pipeline is where bronze, silver, and gold should live, because the graph, the expectations, and the incremental state are the product. A job is the outer frame: cross-system steps, notifications, conditional branches, and tasks that update something other than a table. The failure mode is using a job of notebooks for the table graph, where order is hand-maintained and a new table means editing the DAG, or using a pipeline for the email step, where a table declaration is the wrong shape.

Senior depth: A job task. The moment the work is 'call this system, handle its errors, write a result', it is orchestration, and burying it in a pipeline hides it from the depends_on graph everyone else reasons about.

Common mistake: Picking whichever of the four the team already knows and stretching it over the other three jobs.

Follow-up they will ask: A notebook that calls an external API and writes a table — pipeline task or job task?

Say this first: The pipeline event log records how many rows each expectation dropped or failed, which is the number the aggregate will not show you.

Why: ON VIOLATION DROP ROW keeps the pipeline green while removing data, so the job status is the wrong signal. The event log for the update records each expectation's pass and fail counts. A sudden jump in dropped rows on the silver table is the 12%, and it will not appear as a failure anywhere in the job run. Alert on the drop count, not only on pipeline failure. If the rule should have stopped the line, the expectation belongs on fail, not drop — drop is for rows you have decided are expendable, and that decision needs an owner who looks at the count.

CONSTRAINT valid_id EXPECT (order_id IS NOT NULL)
ON VIOLATION DROP ROW

Senior depth: The update fails, downstream tables in the pipeline do not refresh, and the bad rows are listed as the cause. You trade a quiet 12% loss for a loud stop, which is correct when the rows are not expendable.

Common mistake: Treating a green pipeline run as evidence that every source row landed.

Follow-up they will ask: What changes in the event log if you switch that expectation from drop to fail?

Say this first: One lineage and one event trail from the source system to the gold table. You still need an owner for the connector credentials, the expectations, and the job.

Why: Separate products meant three UIs, three permission models, and a lineage story that stopped at each boundary. With Connect landing into a pipeline that a job triggers, a column can be traced from the source object to the gold table in one place, and a failure can be read in one event log. That does not remove ownership. Connect credentials are a production secret and belong to a platform owner, not to whoever first clicked through the wizard. Expectations encode data-quality policy and belong to the domain team that knows which rows are expendable. The job's run-as identity should be a service principal so a person leaving does not take the pipeline with them. Unification removes the seams. It does not decide who is on call.

Senior depth: The connector's auth is tied to their identity, the job runs as them, and both stop. Recreate the connection under a service principal and set the job's run-as before that week, not during it.

Common mistake: Treating the unification as an org design and leaving every pipeline owned by whoever created it.

Follow-up they will ask: What breaks the week after the person who clicked through the Connect wizard leaves?