Question 1

Leadership announces a migration to Lakeflow and asks for a project plan. What do you tell them the project actually is?

Accepted Answer

Lakeflow is the name over Connect, Declarative Pipelines, and Jobs — mostly a unification of Delta Live Tables and Workflows, plus managed ingestion. Three products that used to be separate sit under one name. Lakeflow Connect is managed connectors that pull from SaaS systems and databases. Lakeflow Declarative Pipelines is what was Delta Live Tables: you declare tables and the engine runs the graph. Lakeflow Jobs is what was Databricks Workflows: a DAG of tasks with retries and repair. Existing DLT pipelines and jobs keep working under the new names. The new thing to adopt, if you need it, is Connect, which replaces the extractor you would otherwise write for Salesforce or SQL Server. There is no flag day where pipelines must be rewritten. Connect, Declarative Pipelines, and Jobs are the three parts. Declarative Pipelines is the new name for Delta Live Tables. Jobs is the new name for Workflows. Existing pipelines and jobs do not have to be rewritten.

Question 2

You have CSVs landing in a bucket and a Salesforce object to ingest. Which Lakeflow piece handles which, and why are they different?

Accepted Answer

Files landing in object storage are Auto Loader. A SaaS API or a database is Lakeflow Connect, because someone has to speak that system's incremental protocol. Auto Loader watches cloud storage, tracks which files it has seen in its checkpoint, and incrementally reads new ones. It knows nothing about Salesforce's API, cursors, or rate limits. Connect is a managed connector for a specific source: it handles authentication, incremental extraction, and change data capture, and lands the result as tables. Using Connect for files adds a service you do not need. Using a hand-rolled notebook for Salesforce means you own pagination, schema drift, and the retry policy forever. The split is by source shape, not by which product is newer. Auto Loader is for files that appear in object storage. Connect is for APIs and databases, including CDC. Connect owns auth, incremental state, and rate limits for that source. Pick by source shape, not by which product is newer.

Question 3

A declarative pipeline has three tables and nobody specified an order. How does it know bronze runs before gold?

Accepted Answer

The engine reads the queries, sees which tables each one reads, and infers the dependency graph. Each declaration is a table name plus the query that produces it. If silver reads bronze and gold reads silver, that is an edge, and the engine topological-sorts the graph before it runs anything. You do not write depends_on between tables inside the pipeline. The engine also owns the checkpoint and the incremental state for streaming tables, and it retries a failed table rather than you coding that. Expectations declared on a table — CONSTRAINT ... EXPECT — are evaluated as the data flows, and a failed expectation can drop the row, fail the row, or just record it. The graph you see in the pipeline UI is that inferred graph, not a diagram someone drew. Dependencies come from which tables each query reads. There is no depends_on between tables inside one pipeline. The engine owns checkpoints, incremental state, and retries. Expectations are constraints declared on the table, not a downstream check.

Question 4

The pipeline updates tables, but finance wants it to run only after their warehouse export lands, and then to kick off a report. Where does that orchestration live?

Accepted Answer

In a Lakeflow Job. The pipeline is one task in the job, with the export task before it and the report task after it. A declarative pipeline is the right tool for a graph of tables. It is the wrong tool for 'wait for an external system, then run this, then send that'. A job is a DAG of tasks with depends_on, and a pipeline can be one of those tasks. The export task runs first; the pipeline task depends on it and runs the whole table graph as a unit; the report task depends on the pipeline. If the pipeline fails, downstream tasks are skipped, and a repair run reruns the failed task and anything that was skipped — the export does not have to be fetched again. Putting the export and the email inside the pipeline couples operational chores to the table graph. A pipeline is one task inside a job, not a replacement for the job. depends_on orders the export, the pipeline, and the report. A failed pipeline task skips everything downstream of it. Repair run reruns the failed branch, not the tasks that succeeded.

Question 5

You created a materialized view from a SQL warehouse and now there is a pipeline in the workspace you never made. Where did it come from?

Accepted Answer

Streaming tables and materialized views are implemented on the declarative pipeline engine, so creating one from SQL creates a pipeline to run it. CREATE MATERIALIZED VIEW in DBSQL does not store a passive object. It declares a query whose result must be refreshed, and the runtime for that declaration is a Lakeflow Declarative Pipeline, created for you and visible in the pipelines list. That is why the object has an event log, a refresh history, and a schedule, and why serverless shows up on the bill. The same engine was open-sourced into Apache Spark, which is why the syntax is not a proprietary side path. If you delete the pipeline, you delete the thing that refreshes the view. Look at the pipeline's event log when a refresh was slow — it records whether the refresh was incremental or a full recompute. A DBSQL materialized view is backed by a declarative pipeline. The pipeline is created for you and shows up in the workspace. Its event log records incremental versus full refresh. Deleting the pipeline deletes the refresh, not just a UI entry.

Question 6

Give the rule you actually use for Connect versus Auto Loader versus a pipeline versus a job.

Accepted Answer

Connect for a SaaS or database source, Auto Loader for files in storage, a declarative pipeline for the table graph, and a job for anything that is not itself a table. Connect owns the protocol of an external system, so use it when the source is Salesforce, SQL Server, or something with its own incremental API. Auto Loader owns files appearing in a bucket, which Connect is not for. Once data is in the lakehouse, a declarative pipeline is where bronze, silver, and gold should live, because the graph, the expectations, and the incremental state are the product. A job is the outer frame: cross-system steps, notifications, conditional branches, and tasks that update something other than a table. The failure mode is using a job of notebooks for the table graph, where order is hand-maintained and a new table means editing the DAG, or using a pipeline for the email step, where a table declaration is the wrong shape. Connect speaks the source system's API and CDC protocol. Auto Loader speaks files landing in object storage. A pipeline owns the table-to-table graph and its expectations. A job owns steps that are not table declarations.

Question 7

An expectation is set to drop rows that fail, and a downstream aggregate quietly shrunk by 12%. How do you find that?

Accepted Answer

The pipeline event log records how many rows each expectation dropped or failed, which is the number the aggregate will not show you. ON VIOLATION DROP ROW keeps the pipeline green while removing data, so the job status is the wrong signal. The event log for the update records each expectation's pass and fail counts. A sudden jump in dropped rows on the silver table is the 12%, and it will not appear as a failure anywhere in the job run. Alert on the drop count, not only on pipeline failure. If the rule should have stopped the line, the expectation belongs on fail, not drop — drop is for rows you have decided are expendable, and that decision needs an owner who looks at the count. DROP ROW keeps the pipeline green while deleting data. The event log has per-expectation pass and fail counts. Alert on the drop count, not only on pipeline status. Drop is a business decision and needs an owner.

Question 8

What do you actually gain from these three being one product family, and what do you still have to assign to a team?

Accepted Answer

One lineage and one event trail from the source system to the gold table. You still need an owner for the connector credentials, the expectations, and the job. Separate products meant three UIs, three permission models, and a lineage story that stopped at each boundary. With Connect landing into a pipeline that a job triggers, a column can be traced from the source object to the gold table in one place, and a failure can be read in one event log. That does not remove ownership. Connect credentials are a production secret and belong to a platform owner, not to whoever first clicked through the wizard. Expectations encode data-quality policy and belong to the domain team that knows which rows are expendable. The job's run-as identity should be a service principal so a person leaving does not take the pipeline with them. Unification removes the seams. It does not decide who is on call. Lineage can run from the source system to the gold table in one view. Connector credentials are a platform concern, not a wizard default. Expectations encode domain policy and need a domain owner. The job should run as a service principal, not an employee.