Databricks beginner interview questions
Control plane vs data plane, Delta basics, and bronze / silver / gold.
Question 1 of 93
What is the Databricks control plane versus the data plane?
Answer it out loud, then reveal. Play steps through like the simulators.
All questions on this page
Indexed as FAQ. Open any item if you prefer a list to Play.
Showing 93 of 93 questions.
beginnerWhat is the Databricks control plane versus the data plane?
Databricks architecture · tap to open the answer
Say this first: Control plane is Databricks SaaS (workspace, jobs, identity). Data plane is compute and storage in your cloud account.
Why: You click notebooks in the control plane. Clusters, executors, and table files run in your VPC next to S3/ADLS/GCS. Databricks does not keep your Parquet in the SaaS plane.
Common mistake: Thinking the workspace is where Spark executors run.
Follow-up they will ask: Where does a Jobs cluster actually start?
intermediateDoes Databricks store your fact tables?
Databricks architecture · tap to open the answer
Say this first: No. Tables stay in your bucket. The workspace orchestrates.
Why: Reads and writes hit s3://, abfss://, or gs://. The control plane sends API calls and gets job status / small action results. A collect() still OOMs the driver inside your VPC.
Common mistake: Uploading extracts into the workspace 'because it's Databricks'.
Follow-up they will ask: What belongs in Unity Catalog versus the storage account?
beginnerA new engineer asks why a cluster takes four minutes to start if Databricks is a SaaS product. What do you tell them?
Databricks architecture · tap to open the answer
Say this first: The control plane only sends an API call; the VMs are provisioned in your own cloud account, and that provisioning is the wait.
Remember these
- Control plane sends the request; your cloud account provisions the VMs
- Cluster event log is the timeline for every start and failure
- Capacity errors in the event log come from the cloud provider, not Databricks
- Pools and serverless remove provisioning from the critical path
Why: Clicking Start calls the Databricks control plane, which asks your cloud provider for instances in your VPC or VNet subnets. Those VMs boot, pull the Databricks Runtime image, and the driver and executors register with each other before the cluster is usable. The cluster event log narrates exactly that: STARTING, instance acquisition, driver healthy, running. If the node type is scarce in that region the event log shows a capacity error instead, which is a cloud-provider problem, not a Databricks one. Instance pools and serverless cut the wait because the machines are already running before you ask.
Common mistake: Assuming Databricks keeps machines warm for your workspace by default.
Follow-up they will ask: What would you read first if a cluster never reached the running state?
intermediateDraw the line for me: name things the control plane stores and things that never leave your cloud account.
Databricks architecture · tap to open the answer
Say this first: Control plane holds notebook source, job and cluster definitions, query history and Unity Catalog metadata; table files, executor heap, and the disk cache stay in your account.
Remember these
- Notebook source, job config, query history: control plane
- Parquet, _delta_log, executor memory, disk cache: your account
- Notebook and SQL result previews are persisted control-plane side
- Unity Catalog stores metadata and grants, not data files
Why: Databricks-managed storage holds notebook source and revision history, job and cluster configuration, query history, cluster event logs, and the Unity Catalog metadata and grants. Your cloud account holds the Parquet data files, the _delta_log, the executor JVM heap, and the local SSD disk cache on the workers. The detail that catches people out is command output: notebook results and SQL result previews are persisted in the control plane up to a size limit, so a display() of PII rows does put a sample outside your account. That is why the honest answer is 'table files never leave the data plane' rather than 'nothing leaves'. Unity Catalog is names and permissions in the control plane; it is not a second copy of the bytes.
Common mistake: Saying 'nothing leaves our account' and forgetting notebook outputs and query result previews.
Follow-up they will ask: Where does a display() of 10,000 PII rows end up?
beginnerWhat is Delta Lake?
Delta Lake · tap to open the answer
Say this first: Parquet files plus a transaction log (_delta_log) that gives ACID on object storage.
Why: Every write commits JSON (and checkpoints). Time travel is 'read version N'. OPTIMIZE and MERGE use the log to know which files are current.
Common mistake: Delta as a file extension you rename to .delta.
Follow-up they will ask: What happens if you read the Parquet files and ignore the log?
intermediateWhy can two jobs writing the same Delta table not corrupt it the way raw Parquet can?
Delta Lake · tap to open the answer
Say this first: Optimistic concurrency: each commit names the files added/removed. Conflicts abort.
Why: Concurrent overwrites of the same partition can still fail — you retry or isolate. The log is the source of truth, not 'whoever finished last silently'.
Common mistake: Writing Parquet next to Delta and expecting ACID.
Follow-up they will ask: How do you time-travel to yesterday 8am?
beginnerA nightly append failed with a schema mismatch because the source added a column. What are your options, and what does each cost?
Delta Lake · tap to open the answer
Say this first: Reject the file and fix upstream, or allow the column with mergeSchema; overwriteSchema is the destructive option that drops columns.
Remember these
- Schema enforcement is why the job fails instead of writing nulls
- mergeSchema adds the column and backfills existing rows as null
- overwriteSchema removes columns absent from the new write
- Evolve in bronze, enforce in gold
Why: Delta validates the incoming DataFrame schema against the table schema on write, and that enforcement is the reason an unexpected column fails the job instead of silently landing nulls or writing files the table cannot describe. Adding .option("mergeSchema", "true") commits the new column into the table schema and reads existing rows as null for it, which is cheap and auditable in DESCRIBE HISTORY. overwriteSchema on an overwrite is different: it replaces the schema, so a column missing from today's source is removed for everyone, which is how a single upstream typo deletes a field. The layer decides the policy — bronze usually accepts evolution because losing data is worse, while gold wants the hard failure because dashboards depend on its contract.
df.write.format("delta").mode("append")
.option("mergeSchema", "true")
.saveAsTable("ops.bronze.orders_raw")Common mistake: Enabling mergeSchema globally so every upstream typo becomes a permanent table column.
Follow-up they will ask: Where in a medallion design do you want enforcement and where do you want evolution?
intermediateA small table got steadily slower to read and _delta_log now holds hundreds of thousands of JSON files. Explain the mechanism.
Delta Lake · tap to open the answer
Say this first: Reconstructing the current version means replaying commits since the last checkpoint, so a log with a huge commit tail makes every query pay metadata time before it reads data.
Remember these
- Current version = last checkpoint plus the commits after it
- Thousands of tiny commits make planning dominate short queries
- Log growth shows up as time before the first scan task
- delta.logRetentionDuration expires commits; it does not slow their arrival
Why: Each write appends one numbered JSON commit, and periodically Delta writes a Parquet checkpoint that collapses the accumulated state so readers can start from it and replay only the commits after it. A job committing every few seconds inflates the log far faster than checkpointing and cleanup can compress it, so the planning phase has to list and read a large number of small objects before a single scan task launches. In the query profile that shows up as time before the first scan, and on the driver as listing work and memory pressure, which is why it looks like the cluster is idle at the start of every query. delta.logRetentionDuration governs how long expired commits survive cleanup, so the folder is bounded in principle but only if commits are not arriving faster than they expire. The real fix is fewer, larger commits: raise the streaming trigger interval, batch the micro-writes, and compact.
DESCRIBE DETAIL ops.silver.orders;
DESCRIBE HISTORY ops.silver.orders LIMIT 20;Common mistake: Deleting old JSON files out of _delta_log by hand to tidy the folder.
Follow-up they will ask: What makes a streaming write produce thousands of commits per hour?
intermediateWhat does Delta's optimistic concurrency actually compare, and why do two writers working on different dates still collide?
Delta Lake · tap to open the answer
Say this first: It compares the files each commit read and wrote; without a partition or predicate boundary the writers' file sets overlap, so one fails with ConcurrentAppendException.
Remember these
- Conflict detection compares read and written file sets, not rows
- ConcurrentAppendException means files landed inside your predicate's range
- A partition predicate in the MERGE condition proves writers are disjoint
- Unbounded retries hide a layout problem instead of solving it
Why: A writer snapshots the table, plans its work against that snapshot, and at commit time checks whether another commit has since added or removed files that would have changed what it read. If so, it aborts rather than producing a result based on a stale view — that abort is the correctness guarantee, not a bug. ConcurrentAppendException specifically means a concurrent commit added files inside the region your predicate covered, and Delta cannot prove your two writers are disjoint unless the table layout or the predicate tells it so. That is why the fix is structural: partition by the write boundary, or include the partition predicate in the MERGE condition so each writer's file set is provably separate. Retrying blindly succeeds often enough to hide the design problem until the retry count grows.
MERGE INTO ops.silver.orders t
USING staged s
ON t.event_date = '2026-10-23' AND t.order_id = s.order_idCommon mistake: Assuming Delta locks rows, so writers touching different rows can never conflict.
Follow-up they will ask: Which exception do you get instead when the other writer deleted files you were reading?
beginnerThe Snowflake team asks you to run a nightly job that exports your Delta gold tables to Iceberg. Why do you push back?
UniForm and Iceberg: one copy, two formats · tap to open the answer
Say this first: Delta and Iceberg are both Parquet data files plus a metadata layer, so UniForm can publish Iceberg metadata over the files you already wrote instead of copying anything.
Remember these
- Delta and Iceberg differ in metadata, not in the Parquet data files
- UniForm writes Iceberg metadata alongside the Delta log over the same files
- One physical copy of the data, readable as either format
- A nightly export doubles storage and guarantees a stale second copy
Why: The Parquet in a Delta table and the Parquet in an Iceberg table are the same kind of file; what differs is the metadata that lists which files belong to the current snapshot. UniForm uses that fact: set delta.enableIcebergCompatV2 on the table and Databricks writes Iceberg metadata alongside the _delta_log, pointing at the exact same data files. One physical copy, two readable formats, no conversion step. A nightly export instead doubles storage, doubles the write bill, and guarantees the Iceberg copy is hours behind. It also creates a second table that someone will eventually edit independently.
ALTER TABLE sales.orders SET TBLPROPERTIES(
'delta.enableIcebergCompatV2' = 'true',
'delta.universalFormat.enabledFormats' = 'iceberg'
)Common mistake: Describing UniForm as a conversion that rewrites the table into Iceberg files.
Follow-up they will ask: If there is only one copy of the data, what exactly got added to the storage account?
intermediateYour Snowflake dashboard shows a count that is one load behind what the Databricks notebook shows, on the same UniForm table. Is the table corrupt?
UniForm and Iceberg: one copy, two formats · tap to open the answer
Say this first: No — Iceberg metadata is generated asynchronously after the Delta commit, so Iceberg readers can legitimately lag the Delta log by one or more commits.
Remember these
- Iceberg metadata is produced after the Delta commit, not inside it
- Iceberg readers can legitimately trail the Delta log by a commit
- Compare the converted Delta version property against DESCRIBE HISTORY
- Design consumers for eventual consistency, not for commit-for-commit parity
Why: A write to a UniForm table commits to the Delta log first and returns. Generating the matching Iceberg snapshot happens after that commit, as a separate step, so there is a window where Delta readers see version N and Iceberg readers still see version N minus one. On a table written every few minutes, that window is almost always open when someone compares the two side by side. You can see where Iceberg is by reading the table properties Databricks maintains for the conversion — the converted Delta version and the converted timestamp — and comparing them with the latest version in DESCRIBE HISTORY. The correct reaction is to treat Iceberg reads as eventually consistent with the Delta table, not to declare a bug.
DESCRIBE EXTENDED sales.orders;
-- compare delta.universalFormat.icebergCompat.convertedDeltaVersion
-- with the latest version from DESCRIBE HISTORY sales.ordersCommon mistake: Assuming the Iceberg view is transactionally identical to the Delta log at every instant.
Follow-up they will ask: How would you word the freshness SLA you give the Snowflake team?
intermediateFinance looks at the storage line after you enabled UniForm on forty tables and asks whether you are now paying for the data twice. What do you show them?
UniForm and Iceberg: one copy, two formats · tap to open the answer
Say this first: Storage grew by the Iceberg metadata files only; the Parquet is shared, so the delta is small and does not scale with row count the way a copy would.
Remember these
- Only Iceberg metadata is added; Parquet data files are shared
- Metadata size tracks file count and commit count, not row count
- High-frequency writers accumulate Iceberg snapshots quickly
- Compare storage by prefix to prove the data files were not duplicated
Why: UniForm adds Iceberg manifest and snapshot metadata next to the existing _delta_log. Those files describe which Parquet files are in the snapshot, so their size tracks the number of files and the number of commits, not the number of rows. A copy, by contrast, would have doubled the Parquet bytes, which is where the real money is. The number worth watching instead is metadata churn on a high-frequency writer: every commit produces another Iceberg snapshot, so a table written every minute accumulates metadata quickly and benefits from compaction and from expiring old snapshots. Show the storage breakdown by prefix rather than arguing from theory.
Common mistake: Promising 'zero extra storage' when metadata on a frequently committed table is measurable.
Follow-up they will ask: What would you change if the table is written once a minute by a streaming job?
beginnerAn analyst overwrote a gold table at 2pm and finance needs this morning's numbers. What do you type?
Time travel: reading the commit log backwards · tap to open the answer
Say this first: Read the older version: SELECT * FROM gold.revenue VERSION AS OF 812, or TIMESTAMP AS OF '2026-09-12 09:00:00'.
Why: Every Delta write appends an ordered JSON commit to _delta_log, and the superseded data files are still referenced by earlier versions, so reading an old version is a metadata read against a different file list — nothing is restored and nothing is written. DESCRIBE HISTORY gold.revenue lists versions, timestamps, and the operation behind each commit so you can pick the right one.
Common mistake: Assuming the data is gone and asking the platform team for a bucket-level restore.
Follow-up they will ask: Does that query change the table in any way?
intermediateA MERGE with a bad join condition mangled the table and you want version 812 back. Do you INSERT OVERWRITE from VERSION AS OF 812?
Time travel: reading the commit log backwards · tap to open the answer
Say this first: Use RESTORE TABLE t TO VERSION AS OF 812 — it writes a new forward commit, so the rollback itself is auditable.
Why: RESTORE does not rewind the log; it adds a new version whose file list matches the target version, so DESCRIBE HISTORY shows a RESTORE operation and downstream readers just see another new version. Hand-rolling it with INSERT OVERWRITE loses that audit trail and splits the read and write into steps that can interleave with other writers.
Common mistake: Trying to delete JSON files out of _delta_log to 'undo' a commit.
Follow-up they will ask: What happens to a structured streaming reader on that table after a RESTORE?
beginnerTIMESTAMP AS OF '2026-09-01' fails saying the timestamp is before the earliest version, but the table is two years old. Why?
Time travel: reading the commit log backwards · tap to open the answer
Say this first: Timestamp travel resolves against the commits the log still retains, and those older commits were already expired by log cleanup.
Remember these
- Timestamps resolve to the latest commit at or before the requested time
- Expired commits give a resolution error, not a missing-file error
- VACUUMed files give a missing-file error on a commit that still exists
- DESCRIBE HISTORY tells you the real oldest version available
Why: A timestamp query resolves to the latest commit at or before the time you asked for, using the commit timestamps recorded in _delta_log. delta.logRetentionDuration governs how long those commits survive cleanup, so a table can hold two years of data while its earliest known version is recent. The opposite failure also exists and looks different: the commit is still listed but the Parquet it references was removed by VACUUM, which gives a missing-file error rather than a resolution error. DESCRIBE HISTORY is the first command either way, because it tells you the oldest version and timestamp you can actually ask for. The data's age is never the time-travel window; retention and VACUUM are.
DESCRIBE HISTORY ops.silver.orders;
SELECT * FROM ops.silver.orders TIMESTAMP AS OF '2026-09-01 00:00:00';Common mistake: Treating the table's age as the time-travel window.
Follow-up they will ask: Which retention property would you change, and what would it not fix?
intermediateRow counts dropped four percent overnight and nine commits landed. Find the bad one without guessing.
Time travel: reading the commit log backwards · tap to open the answer
Say this first: Read operationMetrics per commit in DESCRIBE HISTORY to find where rows disappeared, then diff that version against its parent with two AS OF reads.
Remember these
- operationMetrics identifies the destructive commit from metadata only
- EXCEPT between two AS OF versions shows the exact rows involved
- Both steps are reads, safe to run against live production
- RESTORE reverts everything after the bad commit; a MERGE does not
Why: Every commit records its operation and operationMetrics — numOutputRows, numTargetRowsDeleted, numFilesAdded and removed — so a DELETE or a MERGE that removed rows is identifiable from metadata alone, without scanning any data. Narrow to the commit where the count changes, then run an EXCEPT between VERSION AS OF n-1 and VERSION AS OF n to see the actual rows involved. Both of those are reads, so this is safe to do on a production table that is still being written. Only after you know which commit and which rows do you choose a remedy, because the choice matters: RESTORE reverts the whole table and throws away the eight good commits that landed after the bad one, whereas a targeted MERGE from the AS OF read repairs just the affected rows. Writing the hypothesis, the evidence, and then the change is the sequence interviewers are listening for.
SELECT * FROM ops.silver.orders VERSION AS OF 940
EXCEPT
SELECT * FROM ops.silver.orders VERSION AS OF 941;Common mistake: Running RESTORE to the last known-good version and silently discarding the good commits that followed.
Follow-up they will ask: When is a targeted MERGE the better fix than RESTORE?
beginnerWhat is Unity Catalog's three-level namespace?
Unity Catalog · tap to open the answer
Say this first: catalog.schema.table (plus volumes for files).
Why: It is the governance plane: grants, lineage, audit. Table files still live in cloud storage. hive_metastore is the old two-level world.
Common mistake: Unity Catalog as a replacement for Delta files.
Follow-up they will ask: Where do you grant SELECT — catalog, schema, or table?
intermediateA user can see a table in the UI but Spark says permission denied. Why?
Unity Catalog · tap to open the answer
Say this first: Workspace access ≠ UC privilege on that catalog/schema/table.
Why: They need USE CATALOG, USE SCHEMA, and SELECT (or a share). Cluster must be UC-enabled. A personal compute with the wrong catalog still fails.
Common mistake: Making the table public because 'they're in the workspace'.
Follow-up they will ask: What's the difference between a storage credential and an external location?
beginnerYou dropped a table and the files were still in the bucket. Another time you dropped one and the data was gone. What is the difference?
Unity Catalog · tap to open the answer
Say this first: Managed tables are owned by Unity Catalog, so DROP removes the data too; external tables only lose their registration and the files stay.
Remember these
- Managed table: Unity Catalog owns the files, DROP deletes data
- External table: DROP removes registration only, files remain
- DESCRIBE EXTENDED shows the table type and location
- UNDROP recovers a managed table only within a retention window
Why: A managed table lives in the managed storage location of its catalog or schema, and Unity Catalog owns its lifecycle, so DROP removes the metadata and the underlying files — recoverable only through UNDROP inside a retention window. An external table is created with an explicit LOCATION inside an external location you registered, so DROP removes the registration and leaves every Parquet file and the _delta_log where they were. DESCRIBE EXTENDED or Catalog Explorer tells you which kind you have and where it points, and that is the command to run before any cleanup script. The trap cuts both ways: people drop a managed table expecting a harmless metadata operation, and people drop external tables expecting storage to be reclaimed and then wonder why the bill is unchanged. Choose deliberately — managed for anything whose lifecycle you want governed, external when another system owns the files.
DESCRIBE EXTENDED finance.gold.orders;
UNDROP TABLE finance.gold.orders;Common mistake: Assuming DROP TABLE is always metadata-only, the way it was for an external path in hive_metastore.
Follow-up they will ask: Which would you choose for a bronze landing table, and why?
intermediateA job needs to read abfss://.../raw/ as a governed table. Walk me from the storage account to a SELECT.
Unity Catalog · tap to open the answer
Say this first: Create a storage credential for the cloud identity, an external location over the path prefix, grant on that location, then define the external table or volume.
Remember these
- Storage credential holds the cloud identity Unity Catalog uses
- External location binds a URL prefix and is the grantable object
- READ FILES and WRITE FILES are granted on the external location
- Direct path access from compute bypasses all of it, including audit
Why: The storage credential wraps the cloud identity — an IAM role or an Azure managed identity — that Unity Catalog uses to reach storage, so no cluster needs its own credentials for that path. An external location binds a URL prefix to that credential and becomes the grantable object, which is what makes GRANT READ FILES ON EXTERNAL LOCATION possible. On top of it you define external tables or external volumes, which inherit governed access and show up in lineage and audit. Two layers exist so a path can be revoked once centrally instead of on every cluster that happened to have an instance profile. The failure mode to name out loud: if compute can still read that path directly, you have a second ungoverned route and the audit log will not show those reads at all.
CREATE EXTERNAL LOCATION raw_zone
URL 'abfss://lake@acct.dfs.core.windows.net/raw'
WITH (STORAGE CREDENTIAL lake_cred);
GRANT READ FILES ON EXTERNAL LOCATION raw_zone TO `data-eng`;Common mistake: Attaching an instance profile to the cluster and calling that Unity Catalog governance.
Follow-up they will ask: How would you stop a cluster from reading that path directly, outside Unity Catalog?
beginnerThe current design gives each region a filtered view over the same fact table. Why is a row filter on the table a stronger control?
Row filters and column masks · tap to open the answer
Say this first: A view only protects people who cannot reach the base table; a row filter is enforced at the table, so every access path gets filtered.
Remember these
- A filtered view protects only those denied base-table access
- A row filter is enforced at the table, on every access path
- The filter is a BOOLEAN UDF referenced by ALTER TABLE
- One function replaces a view per audience and the drift between them
Why: With the view approach, security depends on withholding SELECT on the base table from everyone who should see only part of it. The moment someone gets base access for a legitimate reason — a debugging session, a pipeline that needs the raw grain, a migration script — the restriction is gone and nothing records that. A row filter attached with ALTER TABLE ... SET ROW FILTER applies inside Unity Catalog when the table is read, so the same rules hold through a SQL warehouse, a notebook, a JDBC client, or a downstream view. You also stop maintaining one view per audience, which is where the drift lives. The filter is a BOOLEAN user-defined function, so the logic is defined once and referenced by every table that needs it.
CREATE FUNCTION sec.region_filter(region STRING)
RETURN is_account_group_member('emea_analysts') AND region = 'EMEA'
OR is_account_group_member('global_analysts');
ALTER TABLE sales.orders SET ROW FILTER sec.region_filter ON (region)Common mistake: Calling a filtered view equivalent security, when it depends entirely on nobody having base-table access.
Follow-up they will ask: What happens when an engineer is granted SELECT on the base table for a debugging task?
intermediateWrite me a column mask that shows the full national ID to the compliance group and the last four digits to everyone else. What are the rules the function has to obey?
Row filters and column masks · tap to open the answer
Say this first: A column mask is a UDF that returns the same type as the column, takes the column as an argument, and is attached with ALTER COLUMN ... SET MASK.
Remember these
- The mask UDF must return the column's own data type
- Branch on is_account_group_member() or current_user() inside the function
- ALTER COLUMN ... SET MASK attaches it at the table
- Extra row columns can be passed in when the decision depends on them
Why: The mask function receives the column value and returns a value of the same data type, so a STRING column needs a STRING back — you cannot return NULL of a different type or a struct. Inside the function you branch on identity using is_account_group_member() for group membership or current_user() for the individual. Attaching it with ALTER TABLE ... ALTER COLUMN ... SET MASK means the rewrite happens in Unity Catalog on every read of that column, regardless of client. You can pass other columns of the same row into the function as extra arguments if the masking decision depends on them, for example masking only rows flagged as restricted. The function lives in a schema like any other object, and callers need EXECUTE on it.
CREATE FUNCTION sec.mask_nid(nid STRING) RETURN
CASE WHEN is_account_group_member('compliance') THEN nid
ELSE concat('*****', right(nid, 4)) END;
ALTER TABLE hr.people ALTER COLUMN national_id SET MASK sec.mask_nidCommon mistake: Writing a mask that returns a different type than the column, or returning NULL where the column is non-nullable in a consumer's contract.
Follow-up they will ask: How would you mask only the rows flagged as restricted rather than the whole column?
intermediateA user with SELECT on the table now gets an error mentioning the masking function instead of rows. What is missing?
Row filters and column masks · tap to open the answer
Say this first: The querying principal needs EXECUTE on the filter or mask function, and USE on the schema holding it, not just SELECT on the table.
Remember these
- Readers need EXECUTE on the policy function, plus USE on its schema
- The function owner never hits this, so it surfaces only after rollout
- Granting EXECUTE broadly is safe; the function enforces the restriction
- Always test as a non-owner principal before shipping
Why: A row filter or column mask is a function invocation injected into the query plan, and Unity Catalog checks privileges on that function like any other. If the policy functions live in a dedicated security schema, readers need USE CATALOG and USE SCHEMA on that schema plus EXECUTE on the function. This bites during rollout because the person who tested it was the function's owner and never hit the check. The practical pattern is to grant EXECUTE on the security schema's functions to a broad group — account users or a wide analyst group — since the function itself is what enforces the restriction and seeing its definition does not give anyone data. Test as a non-owner principal before you declare the rollout done.
GRANT USE SCHEMA ON SCHEMA prod.sec TO `account users`;
GRANT EXECUTE ON FUNCTION prod.sec.region_filter TO `account users`Common mistake: Granting SELECT on the table and assuming the policy function comes along with it.
Follow-up they will ask: Why is granting EXECUTE broadly on these functions not a security problem?
beginnerA partner needs your daily aggregates and proposes an SFTP drop of CSVs. Make the case for Delta Sharing instead.
Delta Sharing and Clean Rooms · tap to open the answer
Say this first: The recipient reads your existing Delta table directly from your storage through the sharing protocol, so there is no copy, no export job, and revocation is immediate.
Remember these
- No export job and no second copy of the data
- The sharing server returns short-lived pre-signed URLs after a permission check
- The recipient reads directly from your object storage, seeing live data
- Revocation is a grant change, effective on the next request
Why: With a file drop you build an export job, pay to write a second copy, and the partner's data is only as fresh as the last run. Delta Sharing removes the copy: you create a share, add the table, create a recipient, and grant SELECT on the share. When the recipient queries, the sharing server checks their permission and returns short-lived pre-signed URLs, and their client reads the Parquet directly from your object storage. The data they see is the live table, so there is no export schedule to go wrong. Revoking is a single grant change rather than chasing files that were already downloaded. It is an open protocol, so the recipient does not have to be a Databricks customer.
CREATE SHARE partner_metrics;
ALTER SHARE partner_metrics ADD TABLE prod.gold.daily_revenue;
CREATE RECIPIENT acme;
GRANT SELECT ON SHARE partner_metrics TO RECIPIENT acmeCommon mistake: Describing Delta Sharing as a managed replication service that pushes data to the recipient.
Follow-up they will ask: What does the recipient's client actually download, and from where?
intermediateOne recipient is another Databricks account and another is a Python team with no Databricks at all. Does the setup differ?
Delta Sharing and Clean Rooms · tap to open the answer
Say this first: Yes — Databricks-to-Databricks sharing uses the recipient's Unity Catalog sharing identifier and mounts as a catalog; open sharing issues a credential file for any client.
Remember these
- D2D uses the recipient metastore's sharing identifier and mounts as a catalog
- Open sharing issues a credential file usable by any Delta Sharing client
- D2D preserves recipient-side identity for audit and onward grants
- A credential file is a bearer secret and needs expiry and rotation
Why: For a Databricks recipient you create the recipient with their metastore's sharing identifier, which establishes a trusted relationship between the two Unity Catalogs. On their side the share appears as a catalog they can query with normal three-level names, and their reads carry their own identities, which makes auditing and onward governance much better. For a non-Databricks recipient you use open sharing: Databricks generates an activation link that yields a credential file containing a bearer token and endpoint, and any Delta Sharing client — the Python connector, pandas, Spark, a BI tool — can use it. The tradeoff is that the credential file is a secret you have handed out: anyone who holds it is the recipient, so it needs rotation and an expiry, and you lose per-user attribution on their side.
-- Databricks-to-Databricks
CREATE RECIPIENT acme USING ID 'aws:us-west-2:<metastore-uuid>';
-- open sharing
CREATE RECIPIENT ext_team;
ALTER RECIPIENT ext_team SET PROPERTIES ('expiration_time' = '2026-12-31')Common mistake: Using open sharing for a Databricks recipient out of habit and giving away a long-lived bearer token.
Follow-up they will ask: How would you rotate an open-sharing credential without breaking their jobs?
intermediateLegal asks how fast you can cut a recipient off at the end of a contract. What is the honest answer?
Delta Sharing and Clean Rooms · tap to open the answer
Say this first: Revoking the grant stops the next metadata or URL request immediately, but pre-signed URLs already issued remain valid until they expire.
Remember these
- Revoking stops the next request; it does not recall issued URLs
- Pre-signed URLs stay valid for a short, bounded lifetime
- Deleting the recipient invalidates their credentials entirely
- Data already read and stored by the recipient is a contractual matter
Why: Access is checked when the recipient asks the sharing server for a table's files, and the server responds with pre-signed URLs that are valid for a limited time. Revoke SELECT on the share and the next request is refused, so nothing new can be fetched. A query that was already in flight holds URLs that will keep working until they expire, which is a short window but not zero. So the honest answer is 'effective immediately for new access, with a bounded tail measured in the URL lifetime'. You can also delete the recipient entirely, which invalidates their credentials rather than just one share. What you cannot undo is anything they already read and stored on their side, which is a contractual control, not a technical one.
REVOKE SELECT ON SHARE partner_metrics FROM RECIPIENT acme;
DROP RECIPIENT acmeCommon mistake: Promising instant, total cutoff without mentioning in-flight pre-signed URLs or data the recipient already pulled.
Follow-up they will ask: What would you write into the contract to cover the part technology cannot enforce?
beginnerFinance says the workspace cost $94,000 last month and asks which team spent it. The usage dashboard does not break it down that way. Where do you actually look?
System tables: who spent that? · tap to open the answer
Say this first: system.billing.usage is a Delta table you query with ordinary SQL, and custom tags on the workloads are what turn a DBU number into a team name.
Remember these
- System tables are ordinary Delta tables in the system catalog
- system.billing.usage records SKU, DBUs, workspace, and tags per workload
- Join list_prices on SKU and the usage window to estimate dollars
- custom_tags is the column that attributes spend to a team
Why: The usage dashboards are built on system tables, so they only slice the bill along the dimensions Databricks chose. system.billing.usage records every workload: which SKU ran, for how many DBUs, which workspace, and the tags attached to the cluster, warehouse, or job. Joining it to system.billing.list_prices on the SKU and the time window turns DBUs into an estimated dollar figure. Grouping by usage.custom_tags is the line that answers 'which team'. Without tags the table still tells you what ran, but not who owns it, and that is the gap the dashboard cannot close either.
SELECT u.custom_tags.team AS team, SUM(u.usage_quantity * p.pricing.default) AS usd
FROM system.billing.usage u
JOIN system.billing.list_prices p
ON u.sku_name = p.sku_name
AND u.usage_start_time BETWEEN p.price_start_time AND COALESCE(p.price_end_time, NOW())
GROUP BY team
ORDER BY usd DESCCommon mistake: Treating the usage dashboard as the source of truth and concluding the bill cannot be attributed.
Follow-up they will ask: What do you do about the rows where the team tag is null?
intermediateYou enabled system.billing yesterday and the CFO wants the last six months attributed. Why is that query going to disappoint them?
System tables: who spent that? · tap to open the answer
Say this first: System schemas collect from the moment they are enabled. There is no backfill of history from before that.
Remember these
- System schemas are enabled per metastore and several start off
- Collection begins at enablement, with no backfill
- Pre-enablement history only exists in the cloud bill, untagged
- Enable the schemas you will need on day one of the workspace
Why: Each system schema has to be turned on per metastore, and several are off by default. Once enabled, rows start landing for new activity only. A query over system.billing.usage filtered to last quarter returns whatever has accumulated since you flipped the switch, which might be a day. The cloud provider's own billing export is the only record of the months before that, and it will not have your Databricks tags on it. The practical move is to enable every schema you expect to need on day one of a workspace, then tell stakeholders the attribution clock starts now.
Common mistake: Assuming system tables contain the workspace's full history because the UI shows older jobs.
Follow-up they will ask: The job UI shows runs from three months ago. Why can't the system table?
intermediateHalf the bill comes back with a null team tag. What produced those rows, and how do you stop it?
System tables: who spent that? · tap to open the answer
Say this first: Anything created without a custom tag — an all-purpose cluster someone spun up, a warehouse, a job from the UI — lands as unattributed usage.
Remember these
- Untagged clusters, warehouses, and jobs bill with a null team
- Serverless and platform workloads need a budget policy to attribute
- A cluster policy can require the tag before compute starts
- The null-tag query is the ongoing audit, not a one-time cleanup
Why: usage.custom_tags only contains tags that were set on the compute or the job when it ran. A cluster created by clicking in the UI, a SQL warehouse with no tags, and a job defined before the tagging standard existed all bill to null. Serverless and system workloads such as predictive optimization bill under their own SKU and often have no team tag unless a budget policy assigns one. The fix is a default tag enforced where compute is created: a budget policy, a cluster policy that requires a team tag, and asset bundles that set the tag in YAML so a UI click is no longer the path. Then query the null-tag remainder weekly, because that query is the audit.
SELECT sku_name, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE custom_tags.team IS NULL
AND usage_date >= current_date() - 7
GROUP BY sku_name
ORDER BY dbus DESCCommon mistake: Tagging new jobs and assuming the existing clusters inherit the standard.
Follow-up they will ask: Where do you enforce the tag so a UI-created cluster cannot skip it?
beginnerWhat are bronze, silver, and gold for?
Medallion Architecture · tap to open the answer
Say this first: Bronze = raw replay. Silver = conformed truth. Gold = product metrics.
Why: Bronze keeps original bytes. Silver types, dedupes, joins. Gold is the grain BI/ML consume. Arrows only flow downstream.
Common mistake: Medallion as three folders with the same messy table copied three times.
Follow-up they will ask: Who is allowed to write back into bronze?
intermediateWhy not serve bronze to a dashboard?
Medallion Architecture · tap to open the answer
Say this first: Every filter becomes a full scan of landing JSON, and metrics disagree.
Why: Bronze has no stable schema contract. Gold is small, named, and owned. Silver is what jobs share so gold doesn't re-parse.
Common mistake: Skipping silver 'to move faster'.
Follow-up they will ask: Where do quality expectations live?
beginnerWhy is bronze append-only? Storage is not free, and we already have the cleaned data in silver.
Medallion Architecture · tap to open the answer
Say this first: Bronze is the replay source — overwrite it and you can never rebuild silver and gold after you find a logic bug.
Remember these
- Bronze is what makes a logic fix a rerun instead of a re-extract
- Keep rejected rows: today's junk is tomorrow's bug report
- _metadata.file_path gives you a natural dedupe key for ingest
- Control bronze cost with retention and compaction, not overwrites
Why: Every downstream bug is fixed by rerunning the transform, and that rerun needs the original bytes, including the rows your current logic rejects or ignores. Append-only bronze also gives you the cheapest form of idempotent ingest, because the source file path is a natural key — _metadata.file_path lets you detect a file that landed twice. Cost is controlled with retention and layout rather than with overwrites: cluster or partition bronze by ingest date, compact it, and expire it on a stated schedule you can defend. Overwriting bronze nightly is how a team ends up asking a vendor to resend three weeks of data, which is usually impossible. The rule of thumb is that bronze is cheap storage buying you expensive optionality.
SELECT _metadata.file_path, count(*)
FROM ops.bronze.orders_raw
GROUP BY 1 HAVING count(*) > 0Common mistake: Overwriting bronze nightly because silver is treated as the real table.
Follow-up they will ask: What do you do with rows that fail to parse at the bronze stage?
intermediateDeduplication, currency conversion, and a seven-day rolling average. Which layer does each belong in, and why?
Medallion Architecture · tap to open the answer
Say this first: Dedup and currency conversion in silver, the rolling average in gold.
Remember these
- Silver: one row per key, typed, conformed, joined to reference data
- Gold: consumer grain, aggregation, and metric definitions
- The test is whether two teams would implement it identically
- Aggregating in silver pre-empts the next consumer's window
Why: Silver owns the correctness that every consumer needs: one row per business key, typed columns, conformed units and codes, and joins to reference data. Gold owns the shape one named consumer wants: grain, aggregation, and metric definitions. The test that separates them is whether two teams would implement the rule identically — dedup and FX normalisation are objectively right or wrong, so doing them twice is both waste and a guaranteed source of two different numbers. A seven-day rolling average is a product decision with a window someone chose, which is why it needs an owner and belongs in gold. Push the aggregate into silver and you have pre-empted the next consumer who needs a different window; push dedup into gold and every gold table reimplements it slightly differently.
Common mistake: Moving aggregation into silver because a dashboard was slow, then needing a different window a month later.
Follow-up they will ask: Where does a business rule like 'exclude internal test accounts' belong?
beginnerWhat does MERGE INTO do on Delta?
Delta MERGE · tap to open the answer
Say this first: Match keys, update/insert/delete, rewrite touched files, one atomic commit.
Why: It is not an in-place row edit on Parquet. Files that contain matched rows are rewritten. Tiny files after many MERGEs need OPTIMIZE.
Common mistake: Thinking MERGE updates a row in a single Parquet file like a database page.
Follow-up they will ask: What happens to unmatched target rows by default?
intermediateWhy did MERGE rewrite 40% of the table to change 0.1% of rows?
Delta MERGE · tap to open the answer
Say this first: Those rows lived in large files (or unclustered files) so whole files were copied.
Why: Partition / liquid clustering / Z-ORDER so a key lives in fewer files. MERGE ON a selective condition. Don't MERGE a full snapshot if you have change data.
Common mistake: OPTIMIZE after every MERGE in the same transaction as a religion without measuring file size.
Follow-up they will ask: How do you MERGE only yesterday's partition?
beginnerA MERGE finished in 40 minutes and you want to know whether that was a lot of work or a lot of waste. Where do you look?
Delta MERGE · tap to open the answer
Say this first: DESCRIBE HISTORY on the target: the MERGE commit's operationMetrics tells you rows touched and files rewritten.
Remember these
- DESCRIBE HISTORY exposes operationMetrics per commit
- numTargetFilesRemoved is the rewrite cost, not rows updated
- DESCRIBE DETAIL gives numFiles and sizeInBytes for context
- Rows changed vs files rewritten is the amplification ratio
Why: Every Delta write records an entry in the transaction log, and DESCRIBE HISTORY surfaces it with an operationMetrics map. For a MERGE that map carries numSourceRows, numTargetRowsUpdated, numTargetRowsInserted, numTargetRowsDeleted, numTargetFilesRemoved and numTargetFilesAdded. The ratio you care about is rows changed against files removed: 900 rows updated and 4,000 files removed means the job copied most of the table to edit almost nothing. Pair it with DESCRIBE DETAIL, which gives numFiles and sizeInBytes, so you can turn files removed into an estimate of bytes rewritten. That single ratio is the difference between 'the MERGE is slow' and 'the file layout is wrong', and it is the number an interviewer wants you to name.
DESCRIBE HISTORY ops.silver.orders;
-- operationMetrics: numSourceRows, numTargetRowsUpdated,
-- numTargetFilesRemoved, numTargetFilesAdded
DESCRIBE DETAIL ops.silver.orders;Common mistake: Reading only the wall clock from the job run and never opening the commit metrics.
Follow-up they will ask: If files removed is close to the table's total file count, what is your next move?
intermediateYour nightly MERGE started failing with 'Cannot perform Merge as multiple source rows matched and attempted to modify the same target row'. What changed upstream, and how do you fix it?
Delta MERGE · tap to open the answer
Say this first: The source now contains more than one row per merge key, so Delta refuses to guess which one wins — collapse the source to one row per key first.
Remember these
- At most one source row may match a given target row
- CDC batches routinely contain several versions of one key
- ROW_NUMBER over the key ordered by sequence desc collapses them
- A loose ON clause produces the same error from the target side
Why: MERGE requires that each target row match at most one source row, because an UPDATE with two candidate payloads has no defined result. A change-data feed breaks this as soon as a key is modified twice inside one batch window, which happens the first time a customer edits an order and then cancels it in the same hour. The fix is in the USING clause: rank the source by the key, ordered by the sequence or commit timestamp, and keep only the latest row. The other cause is an ON clause that is too loose, for example joining on email instead of the primary key, so one source row legitimately matches several target rows. Check which of the two it is by counting the source grouped by the key before you change anything — if the source is already unique, the ON clause is the bug.
MERGE INTO ops.silver.orders t
USING (SELECT * FROM (
SELECT *, ROW_NUMBER() OVER (PARTITION BY id ORDER BY seq DESC) rn FROM changes
) WHERE rn = 1) s ON t.id = s.id
WHEN MATCHED THEN UPDATE SET *Common mistake: Adding DISTINCT to the source, which keeps both versions of the key whenever any other column differs.
Follow-up they will ask: If two source rows for a key are identical except for the sequence number, which one should win and why?
intermediateWalk me through the difference between WHEN MATCHED THEN UPDATE SET * and listing every column by hand. When has SET * bitten you?
Delta MERGE · tap to open the answer
Say this first: SET * expands to the source's columns matched by name, so it is convenient until the source's shape changes or it overwrites columns the target owns.
Remember these
- SET * matches source to target columns by name, not position
- A new source column fails the MERGE unless evolution is enabled
- WITH SCHEMA EVOLUTION opts in per statement instead of per session
- Explicit assignments protect target-owned audit columns
Why: SET * is resolved at analysis time into an assignment per source column, matched to target columns by name rather than position, so reordering the source is harmless but renaming or dropping a column is not. The first trap is target-owned columns: if silver carries ingested_at, processed_by or a surrogate key that the source does not have, SET * leaves them alone, but if the source happens to carry a column of the same name it will overwrite them with whatever the vendor sent. The second trap is a new source column, which fails the MERGE unless schema evolution is enabled — either through the session setting spark.databricks.delta.schema.autoMerge.enabled or by writing MERGE WITH SCHEMA EVOLUTION INTO. Explicit column lists are more typing and they are also a contract: a new upstream column cannot change silver without someone editing the statement. For bronze-to-silver plumbing SET * is reasonable; for a table a finance team reads, name the columns.
MERGE WITH SCHEMA EVOLUTION INTO ops.silver.orders t
USING changes s ON t.id = s.id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *Common mistake: Enabling automatic schema merge globally on the cluster so every pipeline silently widens its tables.
Follow-up they will ask: How would you detect that a MERGE added a column to silver last night?
beginnerWhat does OPTIMIZE do?
OPTIMIZE, Z-ORDER, Liquid Clustering · tap to open the answer
Say this first: Compacts small files into larger ones so scans launch fewer tasks.
Why: It rewrites data files and commits a new Delta version. It does not change business grain. Z-ORDER / CLUSTER BY is extra: it colocates filter columns.
Common mistake: OPTIMIZE as a vacuum of old versions.
Follow-up they will ask: Does OPTIMIZE make MERGE faster, slower, or both?
intermediateZ-ORDER vs liquid clustering — when do you pick each?
OPTIMIZE, Z-ORDER, Liquid Clustering · tap to open the answer
Say this first: Z-ORDER is a rewrite you schedule. Liquid clustering is incremental on write (newer DBR).
Why: High-cardinality id Z-ORDER can be wasted. Cluster by columns you actually filter. Don't Z-ORDER every column.
Common mistake: Z-ORDER(customer_id, date, country, sku, event_id) 'to be safe'.
Follow-up they will ask: How do you prove skipping worked?
beginnerA streaming bronze job lands around 1,200 files an hour. Which two table properties attack that at write time instead of leaving it for a nightly OPTIMIZE?
OPTIMIZE, Z-ORDER, Liquid Clustering · tap to open the answer
Say this first: delta.autoOptimize.optimizeWrite, which shuffles before writing to produce fewer larger files, and delta.autoOptimize.autoCompact, which compacts small files right after a write.
Remember these
- optimizeWrite shuffles so each task writes one larger file
- autoCompact runs a small compaction after a commit
- Both are table properties, so they bind every writer
- Neither clusters data; scheduled layout work still has a job
Why: Optimized writes add a shuffle in front of the write so each task produces one healthier file instead of one file per Spark partition, which is where the 1,200-files-an-hour number usually comes from. Auto compaction then checks after a successful commit whether the table has accumulated enough small files and, if so, runs a small compaction as a follow-up commit. Both are table properties, so they apply to every writer rather than to one job's code, which matters when three pipelines append to the same bronze table. They reduce how often you need OPTIMIZE but they do not replace it: neither one reorganizes data by a clustering key, and auto compaction targets modest file sizes rather than a full bin-pack. The cost is on the write path — the extra shuffle and the follow-up commit make each micro-batch slower, which is a real trade on a latency-sensitive stream.
ALTER TABLE ops.bronze.orders_raw SET TBLPROPERTIES (
'delta.autoOptimize.optimizeWrite' = 'true',
'delta.autoOptimize.autoCompact' = 'true'
)Common mistake: Turning both on and dropping the scheduled OPTIMIZE, then wondering why selective filters still read the whole table.
Follow-up they will ask: What does the extra shuffle do to a one-minute micro-batch?
intermediateExplain how Delta decides a file cannot possibly match a WHERE clause, and what OPTIMIZE ZORDER changes about that decision.
OPTIMIZE, Z-ORDER, Liquid Clustering · tap to open the answer
Say this first: Each file's add entry in the log carries per-column min/max and null counts; the scan drops files whose range refutes the predicate, and Z-ORDER tightens those ranges.
Remember these
- Min/max and null counts live in the log's add actions per file
- Only leading columns get stats (delta.dataSkippingNumIndexedCols)
- Z-ORDER colocates values so file ranges narrow enough to refute
- date_trunc(col) in the predicate defeats skipping on col
Why: When Delta commits a data file it records statistics for the leading columns of the schema, bounded by delta.dataSkippingNumIndexedCols. At read time the planner evaluates the predicate against those ranges and skips any file it can prove holds no matching row, so the scan never opens it. If rows for a given region are scattered across every file, each file's min/max for region spans the whole alphabet and nothing can be refuted. OPTIMIZE ZORDER BY (region, order_date) rewrites the data so rows with similar values for those columns land together, which narrows each file's ranges until the predicate becomes refutable. Three things defeat it: a filter column that sits past the indexed-column cutoff, a long string column whose statistics are truncated, and wrapping the column in a function so the predicate no longer compares against the raw column. The query profile is the proof — it reports files pruned against files read for the scan.
-- stats cover only the leading columns of the schema
ALTER TABLE ops.silver.orders
SET TBLPROPERTIES ('delta.dataSkippingNumIndexedCols' = '16');
OPTIMIZE ops.silver.orders ZORDER BY (region, order_date);Common mistake: Believing Z-ORDER builds an index. It reorders rows so existing file statistics become useful; there is no secondary structure to look up.
Follow-up they will ask: Your filter column is the 40th column in the schema. What are your options?
intermediateYou added a nightly OPTIMIZE and the storage bill went up rather than down. Is that expected?
OPTIMIZE, Z-ORDER, Liquid Clustering · tap to open the answer
Say this first: Yes — OPTIMIZE writes new files and only marks the old ones removed in the log, so both copies occupy storage until VACUUM deletes the unreferenced ones.
Remember these
- Commits add and remove references; they never delete bytes
- Removed files survive until VACUUM and past the retention window
- A frequent OPTIMIZE cadence multiplies superseded copies
- Retention is a recovery decision, not a storage-cost decision
Why: A Delta commit never deletes bytes. OPTIMIZE adds the compacted files and records remove actions for the originals, which keeps time travel to earlier versions working and lets in-flight readers finish against the file list they already resolved. Those removed files stay in object storage until a VACUUM runs and they are older than delta.deletedFileRetentionDuration, so a table that is optimized nightly and never vacuumed holds every generation of itself. Running OPTIMIZE more often makes this worse, because each run creates another full set of superseded files for the data it touched. The fix is to schedule VACUUM alongside OPTIMIZE with a retention that respects your recovery requirement, and to check the VACUUM commit's metrics to confirm it actually deleted something. On Unity Catalog managed tables predictive optimization can handle both, but you should still be able to point at the commits that prove it ran.
VACUUM ops.silver.orders; -- deletes unreferenced files past retention
SHOW TBLPROPERTIES ops.silver.orders ('delta.deletedFileRetentionDuration');Common mistake: Running VACUUM with a very short retention to reclaim the space, which breaks time travel and can fail concurrent readers.
Follow-up they will ask: How do you pick a retention window you can defend to both finance and legal?
beginnerYou are creating a new fact table and someone proposes PARTITIONED BY (event_date) plus a nightly OPTIMIZE ZORDER BY (customer_id). What would you do instead?
Liquid Clustering: change your mind later · tap to open the answer
Say this first: CLUSTER BY (event_date, customer_id) — liquid clustering replaces both the Hive partitioning and the Z-ORDER.
Why: CREATE TABLE ... CLUSTER BY (cols) lets Delta organize files by those keys and maintain the layout incrementally as data lands, instead of pinning a physical directory structure and periodically re-sorting the whole table. You cannot have both: a table is either partitioned or clustered, and CLUSTER BY is rejected alongside PARTITIONED BY.
Common mistake: Treating liquid clustering as a new name for Z-ORDER and keeping the partition columns as well.
Follow-up they will ask: What actually performs the clustering after rows are written?
intermediateQuery patterns moved from customer_id to region. What does re-keying cost on a liquid-clustered table versus a partitioned one?
Liquid Clustering: change your mind later · tap to open the answer
Say this first: ALTER TABLE t CLUSTER BY (region) changes the keys with no rewrite of existing data; repartitioning a Hive-partitioned table means recreating it and rewriting every byte.
Why: Clustering keys are table metadata, so the ALTER applies to subsequent writes and to subsequent OPTIMIZE runs, which cluster new and touched data incrementally — old files stay as they are until something rewrites them. Changing PARTITIONED BY is not an ALTER at all, which is exactly why teams stayed stuck with a bad partition column for years.
Common mistake: Expecting the ALTER to re-lay-out history instantly, then declaring clustering broken when the first query looks the same.
Follow-up they will ask: What would you run if you genuinely needed history in the new layout today?
beginnerSomeone hands you a table and says it is clustered. How do you verify that, and on which keys?
Liquid Clustering: change your mind later · tap to open the answer
Say this first: DESCRIBE DETAIL reports clusteringColumns and partitionColumns, SHOW CREATE TABLE prints the CLUSTER BY clause, and DESCRIBE HISTORY tells you whether any OPTIMIZE has run.
Remember these
- DESCRIBE DETAIL: clusteringColumns, partitionColumns, numFiles, sizeInBytes
- SHOW CREATE TABLE prints the CLUSTER BY clause, including AUTO
- A declared key with no OPTIMIZE commit has not reorganized history
- Files pruned vs files read in the query profile is the real test
Why: DESCRIBE DETAIL is the one-stop answer: clusteringColumns lists the liquid clustering keys, partitionColumns is empty on a clustered table, and numFiles with sizeInBytes gives you the average file size so you can see whether the layout is healthy at all. SHOW CREATE TABLE prints the DDL including CLUSTER BY, which is useful when you want to reproduce the table or check whether it says AUTO. Declaring keys is only half of it — clustering is performed by writers and by OPTIMIZE, so DESCRIBE HISTORY matters: a table declared with CLUSTER BY three months ago that has never had an OPTIMIZE commit is clustered in name only for its historical data. Finally, check a real query in the profile and read files pruned against files read, because that is the number the clustering exists to move.
DESCRIBE DETAIL ops.silver.orders;
SHOW CREATE TABLE ops.silver.orders;
DESCRIBE HISTORY ops.silver.orders;Common mistake: Trusting the CREATE TABLE statement in the repository, which says nothing about whether maintenance ever ran.
Follow-up they will ask: DESCRIBE DETAIL shows 300,000 files averaging 3 MB on a clustered table. What do you conclude?
intermediateHow many clustering keys would you declare, and what do you say when someone asks for six?
Liquid Clustering: change your mind later · tap to open the answer
Say this first: Liquid clustering takes up to four keys, and every key you add that nobody filters on dilutes the clustering of the ones they do.
Remember these
- Up to four clustering keys; fewer usually prunes better
- Each extra key loosens the per-file ranges of the others
- Pick keys from query history, weighted by query cost
- CLUSTER BY AUTO beats guessing when the pattern is unknown
Why: Clustering spreads rows across files so that each key's values are localized, and that localization is a finite budget split across the declared keys. Four is the documented ceiling, but the practical limit is lower: with four keys, each one's min/max ranges are looser than they would be with two, so a filter on a single key prunes less. The selection rule is empirical rather than aesthetic — take the columns that actually appear in WHERE clauses and join conditions from query history, weighted by how much those queries cost, and stop there. The classic waste is including a column because it is in the primary key or because it feels important; a surrogate id nobody filters on contributes nothing but dilution. If you genuinely cannot characterize the access pattern, CLUSTER BY AUTO lets Databricks pick and adjust the keys from observed queries, which is a better answer than guessing six.
CREATE OR REPLACE TABLE ops.silver.orders
CLUSTER BY (country, event_date)
AS SELECT * FROM staging.ordersCommon mistake: Carrying over the full list of Z-ORDER columns from the old table, which was already too long for the same reason.
Follow-up they will ask: Which of the two keys in that example does the pruning for a query that filters only on event_date?
beginnerWe have a nightly notebook that loops over six hundred tables running OPTIMIZE and VACUUM. What does predictive optimization change about that?
Predictive Optimization: stop cron-ing OPTIMIZE · tap to open the answer
Say this first: Databricks runs the maintenance itself, using Unity Catalog's usage and change statistics to decide which tables actually need it rather than touching all six hundred.
Remember these
- Maintenance is scheduled by predicted benefit, not by a fixed table list
- Enabled at account, catalog, or schema level and inherited downward
- Runs on serverless compute and bills on its own line
- Replaces the fixed nightly window that grew with every new table
Why: The cron approach spends the same effort on a table that changed by four rows and a table that was half rewritten, because a loop cannot tell the difference. Predictive optimization uses the statistics Unity Catalog already keeps about how tables are written and queried, and schedules OPTIMIZE, VACUUM, and related maintenance where the predicted benefit justifies the cost. You enable it once at the account, catalog, or schema level and child objects inherit it. The operational win is that the nightly window stops being a fixed multi-hour block that grows with table count. The accounting change is that this work runs on serverless compute and shows up as its own usage, instead of hiding inside your ETL cluster's bill.
ALTER CATALOG prod ENABLE PREDICTIVE OPTIMIZATION;
ALTER SCHEMA prod.bronze INHERIT PREDICTIVE OPTIMIZATIONCommon mistake: Assuming it is just a managed cron that runs the same commands on every table on a schedule.
Follow-up they will ask: Where would you look to see what it actually ran last week?
intermediateYou enabled predictive optimization on the whole catalog, but one schema's tables are never maintained and still have a small-files problem. Why?
Predictive Optimization: stop cron-ing OPTIMIZE · tap to open the answer
Say this first: Predictive optimization only covers Unity Catalog managed tables — external tables and anything still in the Hive metastore are skipped.
Remember these
- Coverage is Unity Catalog managed tables only
- External tables are skipped because Databricks does not own their files
- hive_metastore tables are not candidates at all
- DESCRIBE DETAIL tells you MANAGED versus EXTERNAL per table
Why: The feature works on tables whose storage lifecycle Databricks owns, which means Unity Catalog managed tables. External tables point at a location you control and may be written by things Databricks cannot see, so it will not rewrite or vacuum their files. Tables still registered in hive_metastore are outside Unity Catalog entirely and are not candidates at all. Run DESCRIBE DETAIL or check the information schema to confirm whether the schema's tables are MANAGED or EXTERNAL before blaming the feature. The fix is usually a migration to managed tables, not a configuration flag — and until that happens, those tables need to keep their old maintenance job.
SELECT table_name, table_type
FROM prod.information_schema.tables
WHERE table_schema = 'bronze' AND table_type <> 'MANAGED'Common mistake: Deleting the old OPTIMIZE job catalog-wide and assuming every table is now covered.
Follow-up they will ask: What is your migration path for the external tables in that schema?
intermediateYour manager wants proof that predictive optimization is doing something before approving it for production. What do you pull?
Predictive Optimization: stop cron-ing OPTIMIZE · tap to open the answer
Say this first: The predictive optimization operations history system table, which lists every operation it ran, on which table, and what it cost.
Remember these
- system.storage.predictive_optimization_operations_history lists what ran
- Join to system.billing.usage to convert DBUs into money
- Compare bytes scanned before and after on the most-touched tables
- The old cron's cost was hidden in an ETL cluster, so compare honestly
Why: system.storage.predictive_optimization_operations_history records the maintenance operations the feature performed — the operation type, the target table, the metrics, and the usage attributed to it. Join or compare it against system.billing.usage to put a currency number next to the DBUs. On the benefit side, pick the tables it touched most and compare query profiles or bytes scanned before and after, because the claim being made is that queries stop degrading, not that they get faster than they ever were. Present both halves: the maintenance DBUs it spent and the maintenance job it replaced. If you only show the new line on the bill you will lose the argument, because the cost it removed was buried inside an ETL cluster nobody itemised.
SELECT catalog_name, table_name, operation_type, usage_unit, SUM(usage_quantity) AS dbus
FROM system.storage.predictive_optimization_operations_history
WHERE start_time >= current_date() - INTERVAL 7 DAYS
GROUP BY ALL ORDER BY dbus DESCCommon mistake: Comparing the new serverless line against zero, as if the old nightly OPTIMIZE job had been free.
Follow-up they will ask: Which tables would you expect to dominate that list, and what would it mean if a cold table topped it?
beginnerWhat problem does Auto Loader solve?
Auto Loader · tap to open the answer
Say this first: Incrementally ingest new files without listing the whole bucket every run.
Why: cloudFiles source + notification/listing + checkpoint. It is not a replacement for Delta MERGE logic — it lands bronze.
Common mistake: Every job does dbutils.fs.ls on the landing zone.
Follow-up they will ask: Where is Auto Loader's progress stored?
intermediateDirectory listing vs file notification mode — when do you need notifications?
Auto Loader · tap to open the answer
Say this first: Huge landing zones: listing is slow and expensive. Notifications scale.
Why: File events (SNS/SQS, Event Grid, Pub/Sub) tell you what landed. Listing is fine for small prefixes. Misconfigured notifications skip files or duplicate if you also copy.
Common mistake: Notification mode without the cloud event pipeline actually wired.
Follow-up they will ask: How does schema inference evolve with rescued data?
beginnerA new column shows up in the incoming JSON. What are your options for how the stream reacts, and which one deliberately stops it?
Auto Loader · tap to open the answer
Say this first: cloudFiles.schemaEvolutionMode picks between addNewColumns, rescue, failOnNewColumns and none — and both of the failing modes stop the stream, for different reasons.
Remember these
- addNewColumns fails once, widens the stored schema, then succeeds on restart
- rescue keeps the schema fixed and diverts new columns to the rescued column
- failOnNewColumns blocks until a human intervenes
- cloudFiles.schemaHints pins types so inference cannot drift
Why: With addNewColumns, the stream fails when it sees an unknown column, writes the widened schema to the schema location, and the next run picks the column up — which means it only works if something restarts the job, and people read that first failure as a bug. With rescue, the schema never changes and unexpected columns are captured in the rescued data column instead, so the stream keeps running and the new field is a JSON blob nobody notices. failOnNewColumns fails and does not update the stored schema, so it stays failed until a human decides what to do, which is what you want when a schema change downstream must be a reviewed event. none ignores new columns entirely. Schema hints are the companion option: cloudFiles.schemaHints pins the type of specific columns so inference cannot decide your amount column is a string this week and a double next week.
.option('cloudFiles.schemaEvolutionMode', 'addNewColumns')
.option('cloudFiles.schemaHints', 'order_id BIGINT, amount DECIMAL(18,2)')Common mistake: Seeing the single failure from addNewColumns, assuming data was lost, and deleting the schema location to 'start clean'.
Follow-up they will ask: Which mode would you pick for a bronze table feeding a regulated report?
intermediatecloudFiles.schemaLocation and checkpointLocation — what lives in each, and what breaks if someone points both at the same scratch path?
Auto Loader · tap to open the answer
Say this first: The schema location holds the inferred schema and its evolution history; the checkpoint holds stream offsets and the registry of files already ingested. Losing either has a different, equally bad failure.
Remember these
- schemaLocation: inferred schema plus its evolution history
- checkpointLocation: offsets plus the registry of ingested file paths
- Losing the checkpoint duplicates; losing the schema re-infers types
- Both must be durable paths, never a scratch or temp directory
Why: The schema location is where Auto Loader persists the schema it inferred, versioned as it evolves, so restarts agree with the previous run about column names and types. The checkpoint location holds the streaming offsets plus the file registry that makes ingestion exactly-once — the record of which paths have already been committed. Delete the checkpoint and the stream treats the whole landing zone as new, which duplicates bronze rather than losing it. Delete the schema location and the next run re-infers from a sample, which can change types and quietly route mismatched values into the rescued column. They must therefore be separate, durable paths that outlive clusters: a workspace temp directory or anything a cleanup policy can reach is how teams lose both at once. If a declarative pipeline owns the stream the service manages both for you, which is one of the stronger arguments for using it for ingest.
(spark.readStream.format('cloudFiles')
.option('cloudFiles.format', 'json')
.option('cloudFiles.schemaLocation', '/Volumes/ops/ingest/schema/orders')
.load(landing)
.writeStream
.option('checkpointLocation', '/Volumes/ops/ingest/ckpt/orders')
.trigger(availableNow=True).table('ops.bronze.orders_raw'))Common mistake: Deriving both paths from one variable so they end up in the same directory, then losing the schema and the offsets in a single cleanup.
Follow-up they will ask: The checkpoint is gone and bronze has no deduplication. What is your recovery plan?
intermediateThe cloud notification queue dropped events during a provider incident and a few hours of files were never ingested. How do you pick them up without resetting the checkpoint?
Auto Loader · tap to open the answer
Say this first: cloudFiles.backfillInterval makes Auto Loader periodically list the directory to catch files notifications missed, and the file registry stops anything being ingested twice.
Remember these
- Notification mode permanently misses a file whose event was lost
- cloudFiles.backfillInterval adds a periodic listing as a safety net
- The file registry makes backfills idempotent, not duplicate-prone
- includeExistingFiles only governs the stream's very first start
Why: File notification mode is efficient because it reacts to events rather than listing, which also means a lost event is a permanently missed file. cloudFiles.backfillInterval tells the stream to run an asynchronous directory listing on a cadence and pick up anything its registry has not seen, so the gap closes on the next backfill without any manual intervention. The registry is what makes this safe: files already committed are not re-ingested, so a backfill is not a duplicate risk. For a one-off recovery you can also run a separate availableNow stream with its own checkpoint pointed at the affected prefix, land it in a staging table, and MERGE it into bronze keyed on the source file path from the _metadata column. What you should not do is delete the checkpoint, which reprocesses the entire landing zone. Note that cloudFiles.includeExistingFiles is a different knob: it decides whether files already present when a stream first starts count as new.
.option('cloudFiles.useNotifications', 'true')
.option('cloudFiles.backfillInterval', '1 day')
.option('cloudFiles.includeExistingFiles', 'true')Common mistake: Running notification mode with no backfill interval and treating the cloud queue as a lossless system of record.
Follow-up they will ask: How would you detect that a file was missed before a consumer notices the gap?
beginnerWhat is Photon?
Photon · tap to open the answer
Say this first: Databricks' native vectorized engine for eligible SQL/DataFrame operators.
Why: It runs in C++ on the data plane, not a new storage format. If the plan is eligible, scans/joins/aggs get faster. UDFs and some expressions fall back to the JVM.
Common mistake: Photon as a magic switch that always 10×s every notebook.
Follow-up they will ask: Where do you see whether Photon ran?
intermediateA Python UDF in the query — what happens to Photon?
Photon · tap to open the answer
Say this first: That operator (and often the pipeline around it) falls back.
Why: Photon is a native vectorized engine that only covers operators it has implemented, and a Python UDF is opaque to it. Rows have to leave the native columnar format, be serialized to a Python worker process, evaluated one at a time, and come back, so that operator runs on the JVM path instead. Because the fallback breaks the native pipeline, neighbouring operators often drop out of Photon too, and you end up paying the Photon DBU premium for a query that is barely using it. The query profile labels which nodes ran in Photon, so check there rather than assuming. Rewriting the UDF with built-in SQL functions is the fix; a pandas UDF reduces the per-row serialization cost but is still not Photon-native.
Common mistake: Turning Photon on and keeping the UDF.
Follow-up they will ask: Does Photon run on all-purpose, jobs, and warehouses?
beginnerWhere do you turn Photon on, and is there anywhere it is already on without you choosing?
Photon · tap to open the answer
Say this first: It is a compute choice: a runtime option when you create an all-purpose or job cluster, and part of the product on SQL warehouses and serverless compute.
Remember these
- A runtime option on all-purpose and job clusters, with its own DBU rate
- SQL warehouses and serverless compute run it as part of the product
- No code, SQL or table changes are involved
- Enabled on the cluster is not the same as used by the query
Why: Photon is not a library, a SQL dialect or a table format — it is an execution engine selected by the compute you run on. On all-purpose and job clusters you choose a Photon-enabled runtime at creation time, which is also where the higher DBU rate attaches. SQL warehouses run it as part of the product, so BI queries are already on it whether anyone decided that or not, and serverless compute is likewise Photon. Nothing about your SQL or DataFrame code changes, which is exactly why teams lose track of whether it is on. The place to verify is the query profile, not the cluster configuration page, because being on a Photon runtime does not mean a given query actually ran in Photon — unsupported operators fall back to the JVM on the same cluster.
Common mistake: Pointing at the cluster configuration as proof the workload is accelerated, without opening a single query profile.
Follow-up they will ask: Where in the profile would you confirm a specific query used it?
intermediateYou are told Photon is on. In the query profile, how do you prove how much of the query actually ran in Photon and find the exact fallback?
Photon · tap to open the answer
Say this first: Photon operators appear as Photon-prefixed nodes in the profile and plan; the fallback is the plain Spark node, and what matters is how much time is attributed to it.
Remember these
- Photon-prefixed node names in the profile mark native execution
- explain on a Photon cluster prints why parts were unsupported
- Judge a fallback by the time attributed to it, not its presence
- High coverage plus slow query means I/O, spill or skew, not Photon
Why: In the query profile, nodes executed natively carry Photon-prefixed names such as PhotonScan and PhotonGroupingAgg, and the profile attributes time and rows to each node, so you can see both what ran natively and what it cost. A fallback shows up as an ordinary Spark operator in the middle of the plan, usually with a transition around it. Running explain on a Photon-enabled cluster is the complementary check, because it prints the reasons Photon could not support parts of the query — that message names the specific expression or operator, which is what you need to fix it. The discipline is to look at time, not presence: a non-Photon node that holds 2% of the query's time is noise, and one that holds 80% is the whole investigation. If the profile shows high Photon coverage and the query is still slow, stop blaming the engine and go look at bytes scanned, file counts, spill and skew.
spark.sql(query).explain('formatted')
# Photon explain reports which operators it could not support, and whyCommon mistake: Spotting one non-Photon node and declaring the query un-accelerated, when that node holds a sliver of the runtime.
Follow-up they will ask: Coverage is high, the query is still slow. What do you look at next?
intermediatePhoton bills a higher DBU rate. How do you decide whether it pays for a specific workload?
Photon · tap to open the answer
Say this first: Run the workload both ways and compare DBUs consumed, not wall clock — Photon pays when runtime falls by more than the rate multiplier.
Remember these
- A speedup below the rate multiplier is a net cost increase
- Compare DBUs from system.billing.usage, not just wall clock
- Scan and aggregation heavy SQL clears the bar; UDF-heavy Python rarely does
- Decide per workload; it is a per-cluster choice
Why: Because Photon raises the DBU rate per hour, a speedup smaller than the multiplier is a cost increase dressed as a performance win. So the test is arithmetic: run the same workload on the same data with and without Photon, and compare total DBUs from system.billing.usage alongside wall clock. Scan- and aggregation-heavy SQL over large Delta tables usually clears the bar comfortably, because that is exactly the operator set Photon implements natively. Workloads dominated by Python UDFs, RDD code or arbitrary notebook Python usually do not clear it, because there is little for Photon to accelerate and you pay the premium on all of it. Small queries are their own case: when startup and listing dominate, the engine barely matters. Decide per workload rather than per account, since you choose it per cluster and warehouses are on it regardless — and if latency rather than cost is the goal, say so explicitly, because then a modest speedup at a higher rate can still be the right call.
SELECT sku_name, sum(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date > current_date() - INTERVAL 30 DAYS
GROUP BY sku_name ORDER BY dbus DESCCommon mistake: Comparing only wall-clock time, which makes a 20% faster query on a much higher rate look like a success.
Follow-up they will ask: The workload is 30% faster on Photon. Is that enough?
beginnerAll-purpose cluster vs jobs cluster vs SQL warehouse — pick one sentence each.
Compute types · tap to open the answer
Say this first: All-purpose: interactive notebooks, stays on. Jobs: start, run, terminate. Warehouse: BI/Photon, not notebooks.
Why: Wrong compute is the expensive mistake that never shows up as a Spark bug. ETL on all-purpose burns idle hours. Tableau on an all-purpose cluster fights notebooks for slots.
Common mistake: One mega cluster for explore, ETL, and BI.
Follow-up they will ask: Which product should a nightly MERGE use?
intermediateWhy is an all-purpose cluster a bad host for production ETL?
Compute types · tap to open the answer
Say this first: It stays on, keeps leftover libraries/cache, and you pay for idle.
Why: A forgotten notebook pins a 32-worker cluster overnight. Job clusters get a clean Spark version and die when the task ends. Cluster policies should block oversized all-purpose.
Common mistake: Scheduling Workflows on the team's shared interactive cluster 'so it's warm'.
Follow-up they will ask: What still belongs on all-purpose?
beginnerA cluster set to autoscale 2–20 workers sat at 20 for an hour after the big job finished. Why didn't it shrink?
Compute types · tap to open the answer
Say this first: Scale-down needs sustained idleness and nodes it can safely release; running tasks, cached data, and attached sessions keep workers alive.
Remember these
- Scale-down requires sustained idleness, not just a finished stage
- Cached data and in-flight tasks pin workers in place
- Cluster event log records every upscale and downscale with a reason
- An all-purpose cluster that never shrinks bills worker-hours all day
Why: Autoscaling reacts to pending work and to idleness over a period, not to stage boundaries, so it will not track each stage's parallelism like a dial. A worker holding a cached dataset or shuffle output another stage still needs is not a candidate for removal, and on an all-purpose cluster one notebook left attached with cached DataFrames can hold the cluster wide open. The cluster event log records the upscale and downscale decisions with a reason, which is where you check rather than guessing. The bill is worker-hours, so an all-purpose cluster that never shrinks back is the surprise invoice nobody can attribute to a job. Clearing the cache, detaching the notebook, or setting a short auto-termination are the levers.
Common mistake: Expecting autoscaling to follow each stage's parallelism up and down like a throttle.
Follow-up they will ask: Why is autoscaling a weaker fit for a short job cluster than for a long interactive one?
intermediateFinance sees two line items for the same cluster: DBUs on the Databricks invoice and instance hours on the cloud bill. Which one does Photon change?
Compute types · tap to open the answer
Say this first: Photon raises DBU consumption per hour; the cloud VM cost is unchanged, so it only pays off if the runtime falls enough to shrink total hours.
Remember these
- Two bills on classic compute: cloud VMs plus Databricks DBUs
- All-purpose carries a higher DBU rate than jobs compute
- Photon raises DBUs per hour, so it must cut runtime to pay off
- Serverless folds the infrastructure cost into a single DBU rate
Why: Every classic workload costs twice: your cloud provider charges for the VMs, disks, and network running in your account, and Databricks charges DBUs at a rate set by the product and the features in use. All-Purpose Compute carries a higher DBU rate than Jobs Compute for the same instance, and enabling Photon increases DBU consumption again. So the Photon decision is arithmetic, not faith — a job must get enough faster to offset the higher rate. Serverless changes the shape of the bill entirely by folding the infrastructure into one Databricks rate, so the cloud VM line disappears for that workload. You settle the argument in system.billing.usage, grouped by SKU, next to the runtimes from the job run history.
SELECT sku_name, sum(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY sku_name
ORDER BY dbus DESCCommon mistake: Turning Photon on everywhere because faster must be cheaper, without measuring the runtime change.
Follow-up they will ask: How would you prove Photon paid for itself on a specific job?
intermediateA job on spot workers fails every few runs with executors disappearing mid-stage. What is your sequence of fixes?
Compute types · tap to open the answer
Say this first: Pin the driver to on-demand, let workers run spot with on-demand fallback, and shorten the shuffle so re-computation after a reclaim is cheap.
Remember these
- Spot reclaim removes the executor and its shuffle files
- A spot driver means the whole run dies, not just a task
- Cluster event log distinguishes a reclaim from an OOM
- Long stages lose more work per reclaim than short ones
Why: A reclaimed spot instance takes its executor and the shuffle files that executor produced, so Spark has to recompute the lost tasks and may hit fetch failures in the downstream stage. Losing the driver is unrecoverable: the whole run dies, which is why the driver should never be spot for anything with an SLA. The cluster event log names the loss explicitly as a spot termination, which distinguishes it from an OOM that people usually blame first. The deeper exposure is stage length, because the amount of work you lose scales with how much a stage had already done — a long single shuffle is the worst shape for spot. Checkpointing, splitting the job into tasks, or running the SLA-critical step on on-demand are the structural answers.
Common mistake: Diagnosing an OOM and raising executor memory when the event log says the instance was reclaimed.
Follow-up they will ask: Which single node should never be spot, and why?
beginnerA classic cluster takes four minutes to start and a serverless warehouse is ready in seconds. What is physically different?
Serverless compute: where the VMs went · tap to open the answer
Say this first: Classic compute provisions new VMs in your own cloud account on demand; serverless draws machines from a pool Databricks already has running.
Remember these
- Classic boots new VMs in your cloud account; that is the minutes
- Serverless attaches capacity from a Databricks-managed warm pool
- Data stays in your storage; only the compute location changed
- You pay for the warm pool through a higher DBU rate
Why: When you start a classic cluster, the control plane asks your cloud provider for instances in your VPC, those VMs boot, pull the runtime image, and the driver and executors register with each other. All of that is real provisioning time, and the cluster event log narrates each step. Serverless compute skips provisioning entirely: Databricks keeps a warm pool of capacity in its own account and attaches some of it to your workload, so startup is seconds. Your data has not moved — it is still in your storage account, read through Unity Catalog credentials. What moved is where the compute runs and who pays for keeping it warm, which is reflected in a higher DBU rate.
Common mistake: Saying serverless means 'no servers', instead of 'someone else already started them'.
Follow-up they will ask: Who is paying for that pool to sit warm when nobody is querying?
intermediateServerless has a higher DBU rate than classic. Give me a workload where it is still cheaper and one where it is not.
Serverless compute: where the VMs went · tap to open the answer
Say this first: Bursty work wins because you stop paying the moment it ends; a cluster pinned busy all day loses, because you pay the premium rate for every hour anyway.
Remember these
- Serverless raises the rate and cuts the billed idle time
- Bursty and intermittent workloads recover the premium easily
- A saturated all-day job has no idle to recover, so it costs more
- Decide from measured utilisation, not from the DBU price alone
Why: Cost is rate multiplied by time, and serverless changes both terms. The rate goes up, but the billed time collapses to actual work because there is no provisioning window and no idle cluster waiting for the next query. An analyst warehouse serving sporadic dashboard hits spends most of the day idle on classic, and that idle is the bill; on serverless it is near zero. A long-running nightly ETL that saturates a cluster for six straight hours has almost no idle to recover, so the higher rate is close to pure markup. The honest test is to pull the actual utilisation from system.billing.usage and the job run timeline rather than arguing from the price list.
SELECT sku_name, date_trunc('DAY', usage_start_time) AS d, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_start_time >= current_date() - INTERVAL 30 DAYS
GROUP BY ALL ORDER BY dCommon mistake: Comparing the two DBU rates and stopping, without counting the idle hours classic bills for.
Follow-up they will ask: How would you estimate the idle fraction of an existing warehouse before migrating it?
intermediateA job that installs a custom wheel and runs an init script fails when you switch it to serverless. What is going on?
Serverless compute: where the VMs went · tap to open the answer
Say this first: Serverless trades configurability for speed: cluster-scoped init scripts, arbitrary instance types, and some library installation paths are constrained.
Remember these
- Standardised machines are why startup is fast; that costs configurability
- Cluster-scoped init scripts and instance type choice are constrained
- Dependencies come through supported environment and artefact paths
- Work that needs root or native system packages stays on classic
Why: The warm pool only works because the machines are standardised, so the hooks that let you reshape a classic cluster are not all available. Cluster-scoped init scripts, choosing a specific instance family, attaching an instance profile, and installing anything that needs root are the usual casualties. Python dependencies are expected to come through supported paths — environment specifications and workspace or Unity Catalog volume artefacts — rather than a script that apt-installs a native library at boot. The practical move is to read what the init script actually does: most of them set an environment variable, install a package, or configure logging, and each of those has a supported equivalent. If the script genuinely needs root or a native system library, that workload stays on classic.
Common mistake: Treating serverless as a drop-in swap for the compute block in an existing job definition.
Follow-up they will ask: Your init script sets three environment variables — what replaces it?
beginnerThe BI team wants to point Tableau at Databricks. Why hand them a SQL warehouse instead of the cluster they already have access to?
SQL Warehouses: compute shaped for BI · tap to open the answer
Say this first: A warehouse is a SQL endpoint built for many short concurrent queries: Photon on by default, result caching, auto-stop, and extra clusters when concurrency rises.
Why: An all-purpose cluster runs notebooks and BI inside the same Spark application, so one runaway cell starves the dashboard and there is no per-query profile for the analyst. A warehouse gives JDBC/ODBC endpoints, Query History with a profile per statement, and scale-out for concurrency instead of fighting notebooks for slots.
Common mistake: Handing BI the JDBC URL of the team's interactive cluster because it is already warm.
Follow-up they will ask: Classic, Pro, or Serverless — which would you pick and what changes?
intermediateOne analyst is fine, but at 9am with forty people the dashboards crawl. Do you bump the T-shirt size?
SQL Warehouses: compute shaped for BI · tap to open the answer
Say this first: No — size is scale-up for one query's horsepower; concurrency needs more clusters, so raise max clusters.
Why: Cluster size (2X-Small through 4X-Large) decides how much compute a single query can use; min/max clusters decides how many queries run in parallel before they queue. If Query History shows long queue time against short execution time, it is a concurrency problem, and upsizing just makes each idle cluster more expensive.
Common mistake: Jumping from Small to 2X-Large, watching the bill grow, and finding the queue still there.
Follow-up they will ask: Where in Query History do you separate queue time from execution time?
beginnerA dashboard is pointed at a warehouse that is currently stopped. What does the user experience, and what starts it?
SQL Warehouses: compute shaped for BI · tap to open the answer
Say this first: The first query starts the warehouse on demand, so the user waits through startup — seconds on Serverless, minutes on Classic or Pro.
Remember these
- A query against a stopped warehouse starts it rather than failing
- Classic and Pro wait for VMs in your account; Serverless starts in seconds
- Query History separates start and queue time from execution
- Disabling auto stop trades a short wait for 24/7 DBUs
Why: A stopped warehouse is not an error state; an incoming query through the JDBC or ODBC endpoint triggers a start and then runs. On Classic and Pro that start means waiting for VMs in your own cloud account, which is why the first dashboard load of the morning feels like the platform is broken. On Serverless the compute comes from a pool Databricks already maintains, so the same start takes seconds. Query History and the warehouse monitoring tab show the start and any queue time, so you can tell a slow query from a slow start. The predictable wrong reaction is to disable auto stop, which converts a one-time morning wait into a bill that runs all night.
Common mistake: Turning auto stop off so nobody ever waits, then being unable to explain the invoice.
Follow-up they will ask: How do you make 9am fast without paying for the whole night?
intermediateThe CFO asks why Serverless costs more per DBU than Classic. Answer it.
SQL Warehouses: compute shaped for BI · tap to open the answer
Say this first: The Serverless rate includes the compute infrastructure Databricks runs for you; on Classic and Pro you pay Databricks less per DBU and your cloud provider separately for the VMs.
Remember these
- Classic and Pro cost DBUs plus a separate cloud VM bill
- Serverless has one rate that already includes the infrastructure
- Short auto stop is only painless when startup is seconds
- Serverless compute runs outside your account, which is a residency question
Why: Classic and Pro warehouses launch instances in your own cloud account, so the real cost is the Databricks DBU rate plus a cloud invoice for VMs, disks, and the idle minutes you failed to stop. Serverless runs in the Databricks account from a warm pool, so there is one rate and no second bill, and startup is seconds rather than VM provisioning. Which is cheaper depends on utilisation: bursty BI with genuine idle gaps usually wins on Serverless because you can set a short auto stop without punishing users, while a warehouse busy twelve hours a day is a closer comparison. The axis that is not about money is where the compute runs, which is a data-residency conversation for some organisations. Compare them with DBU hours per warehouse from system.billing.usage against query counts from Query History, not with list prices side by side.
Common mistake: Comparing per-DBU rates without adding the cloud VM cost and the idle hours to the Classic side.
Follow-up they will ask: What would make you keep a Classic warehouse despite the startup time?
beginnerA dashboard runs the same twelve-table aggregate on every page load. Would you make it a view, a materialized view, or a streaming table?
Streaming tables and materialized views · tap to open the answer
Say this first: A materialized view, because it stores the computed result and the dashboard reads that instead of recomputing the aggregate on every load.
Remember these
- A VIEW recomputes on every read; an MV stores the result
- Streaming tables ingest incrementally using a checkpoint
- Repeated expensive aggregation is the MV case
- Materialized views require serverless compute
Why: A plain VIEW is a saved query: every reader pays the full cost of the aggregate again. A materialized view stores the result as a table and refreshes it on a schedule or on demand, so the dashboard reads precomputed rows. A streaming table is for the ingest side — it consumes new rows from a source incrementally using a checkpoint, which is what you want for appending raw data, not for serving an aggregate. The decision rule is how the result is produced and consumed: incremental ingest means streaming table, repeated expensive aggregation means materialized view, cheap or rarely used logic means a plain view. Materialized views on Databricks require serverless compute, which is worth knowing before you promise one.
CREATE MATERIALIZED VIEW gold.daily_revenue AS
SELECT order_date, region, SUM(amount) AS revenue
FROM silver.orders GROUP BY order_date, regionCommon mistake: Calling a materialized view 'a view that is faster', with no mention that its data has a refresh time.
Follow-up they will ask: What does the dashboard show between refreshes?
intermediateA materialized view that refreshed in two minutes for months now takes ninety. Nobody changed the data volume. What happened?
Streaming tables and materialized views · tap to open the answer
Say this first: It stopped refreshing incrementally and fell back to a full recompute, almost always because the definition changed or it now contains something the planner cannot reason about incrementally.
Remember these
- Incremental refresh happens only when the planner can prove it is safe
- A definition change, current_timestamp(), or rand() forces full recompute
- Refresh history and the pipeline event log record which path ran
- Nothing errors — the result stays correct and only the cost moves
Why: Incremental refresh is not a setting; it is something the planner proves it can do for a given query shape. When it can, it reads only the changed rows from the sources and updates the stored result. When it cannot, it recomputes the whole thing, and on a large source that is the ninety minutes. The usual triggers are a changed definition — any edit can invalidate incremental state — a non-deterministic expression like current_timestamp() or rand(), or a join or aggregate pattern outside what incremental refresh supports. The refresh history and the pipeline event log for the materialized view record which path each refresh took, so you check rather than guess. The reason this hurts is that nothing fails: the view is correct, the dashboard is fine, and only the bill changes.
DESCRIBE HISTORY gold.daily_revenue;
-- and inspect the materialized view's pipeline event log
-- for the refresh type recorded on each runCommon mistake: Assuming incremental refresh is guaranteed because the view was created as a materialized view.
Follow-up they will ask: Where exactly would you look to confirm the last refresh was full rather than incremental?
intermediateSomeone adds a column defined as datediff(current_date(), order_date) to a materialized view. What is your review comment?
Streaming tables and materialized views · tap to open the answer
Say this first: A time-dependent expression makes the result a function of when it ran, so incremental refresh cannot be proven safe and every refresh becomes a full recompute.
Remember these
- Time-dependent expressions make old output rows wrong without input changes
- current_timestamp(), current_date(), and rand() all defeat incremental refresh
- Keep the MV a pure function of source columns
- Compute age or recency in a thin view on top of the MV
Why: Incremental refresh works by reasoning that old output rows are still correct if their input rows did not change. current_date() and current_timestamp() break that reasoning: yesterday's output row is wrong today even though no input changed. The planner cannot verify the result incrementally, so it recomputes everything. The same applies to rand() and to user-defined functions the planner cannot analyse. The fix is to keep the stored result a pure function of the source columns — store order_date and compute the age at read time in the dashboard or in a thin view on top. That keeps the expensive aggregation precomputed and leaves the one cheap time-dependent expression to the reader.
-- store the date, not the age
CREATE MATERIALIZED VIEW gold.orders_rollup AS
SELECT order_date, customer_id, SUM(amount) AS amount
FROM silver.orders GROUP BY order_date, customer_idCommon mistake: Adding a 'days since order' column for convenience and silently turning every refresh into a full rebuild.
Follow-up they will ask: Where would you put the days-since calculation instead?
beginnerWhat is Delta Live Tables (Lakeflow Declarative Pipelines)?
Delta Live Tables (Lakeflow) · tap to open the answer
Say this first: You declare tables and expectations; DLT infers the graph and runs in order.
Why: dlt.read / @dlt.table instead of a 12-task Workflow of notebooks. Expectations drop or fail bad rows. Still Delta under the hood.
Common mistake: DLT as a replacement for Spark.
Follow-up they will ask: How does DLT know bronze runs before gold?
intermediateExpectations vs a filter in a notebook job — why bother?
Delta Live Tables (Lakeflow) · tap to open the answer
Say this first: Expectations are cataloged, metrics are visible, and bad rows can fail the pipeline.
Why: A silent filter hides data quality. DLT event log shows expectation pass rate. That's what on-call wants at 3am.
Common mistake: expect_or_drop everything so the pipeline never fails.
Follow-up they will ask: When is a Workflow of notebooks still better than DLT?
beginnerWhat actually changes when you flip a pipeline from development to production mode?
Delta Live Tables (Lakeflow) · tap to open the answer
Say this first: Compute lifecycle and retry behaviour: development keeps the cluster around and does not retry failures, production terminates the cluster and retries.
Remember these
- Development reuses compute between updates for fast iteration
- Development disables retries so the first error is the one you see
- Production terminates compute per update and retries failures
- The mode does not change the target catalog or schema
Why: In development mode the pipeline's compute is reused between updates for a period rather than being torn down, so the next iteration starts in seconds instead of waiting for a cluster, and failures are not retried so you see the error immediately instead of watching three attempts. In production mode the compute is provisioned for the update and terminated when it ends, and failures are retried according to the service's policy, which is what you want for a scheduled pipeline that hits a transient storage error. That is the whole difference. It is not a different code path, and it is not a different dataset — the catalog and schema a pipeline writes to come from its configuration, so a development-mode pipeline pointed at production targets will happily write to them. If you want a separate sandbox, that is a configuration or Asset Bundle target, not the mode toggle.
Common mistake: Assuming development mode writes somewhere safe, then discovering a half-built table in the production schema.
Follow-up they will ask: How would you give each developer their own isolated target without editing the pipeline by hand?
intermediateIn a declarative pipeline, when do you declare a streaming table and when a materialized view?
Delta Live Tables (Lakeflow) · tap to open the answer
Say this first: Streaming tables process only new rows from an append-only source and keep state; materialized views keep a query's result correct, recomputing when they cannot do it incrementally.
Remember these
- Streaming tables need an append-only source and carry checkpoints
- Materialized views keep a query result correct, recomputing if needed
- Streaming aggregates cannot revise rows they already emitted
- Wrong choice means either full recompute or permanently stale results
Why: A streaming table reads its source as a stream and maintains a checkpoint, so each update processes only what arrived since the last one. That is the right shape for bronze ingest from a landing zone and for silver tables that append or apply changes, because cost tracks new data rather than table size. A materialized view is defined by a query over its sources, and the service keeps its result consistent on every update — incrementally when the query shape allows it, by recomputing when it does not. That is the right shape for gold aggregates and for anything joining a dimension that can change retroactively, because a streaming aggregate cannot revise a row it has already emitted. Picking wrongly is expensive in both directions: a materialized view over a 10 TB bronze table may recompute from scratch on every run, while a streaming table over a source that gets updated in place will serve stale answers forever. The decision question is whether the source is append-only and whether late restatements have to be reflected.
CREATE OR REFRESH STREAMING TABLE bronze_orders AS
SELECT * FROM STREAM read_files('/Volumes/landing/orders', format => 'json');
CREATE OR REFRESH MATERIALIZED VIEW gold_daily AS
SELECT order_date, sum(amount) FROM silver_orders GROUP BY order_date;Common mistake: Declaring everything as a streaming table because the word streaming sounds cheaper, then being unable to restate a corrected aggregate.
Follow-up they will ask: Your source occasionally restates yesterday's rows. Which do you pick for silver?
intermediateMap expect, expect_or_drop and expect_or_fail onto an SLA. Which one goes on a required business key?
Delta Live Tables (Lakeflow) · tap to open the answer
Say this first: expect records and passes the row, expect_or_drop removes it, expect_or_fail aborts the update — a required key belongs on fail or on a quarantine pattern, never a silent drop.
Remember these
- All three write pass/fail counts to the event log
- expect_or_drop is a silent data-loss channel if nobody alerts on the rate
- expect_or_fail is for violations that make the output wrong, not thin
- Quarantine tables keep the SLA and keep the rejected rows
Why: All three record pass and fail counts into the pipeline event log, which is the part that matters operationally: an expectation is a metric first and an action second. expect lets the bad row through and only counts it, which suits a soft rule you want to watch before you enforce. expect_or_drop removes the row from the target and counts the drop, which is appropriate for genuinely unusable records as long as somebody is watching the rate. expect_or_fail aborts the table update so nothing downstream publishes, which is the only correct choice when a violation means the output would be wrong rather than incomplete. A required business key sits in that last category, because dropping it means revenue rows quietly disappear and the pipeline stays green — that is how you get an 18% gap nobody notices. The middle path many teams land on is a quarantine pattern: a negated expectation writing failures into their own table so the main pipeline keeps its SLA and the bad rows are still recoverable.
@dlt.expect('amount_present', 'amount IS NOT NULL')
@dlt.expect_or_drop('valid_ts', 'event_ts IS NOT NULL')
@dlt.expect_or_fail('valid_id', 'id IS NOT NULL')Common mistake: Using expect_or_drop everywhere so the pipeline never goes red, which converts every data problem into a silent row loss.
Follow-up they will ask: How do you alert on a drop rate that creeps from 0.1% to 4% over a month?
beginnerA teammate's production pipeline is one notebook with 40 cells on a schedule. Sell me on breaking it up.
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: A failure in cell 37 re-runs all 40 cells, and you get no per-step retry, no parallelism, and no per-step timing.
Why: A Databricks Job is a DAG of tasks wired with depends_on, so independent steps fan out in parallel, each task carries its own retry policy and timeout, and the run timeline gives you duration per task. Inside one notebook Spark sees a single linear sequence, so recovery means rerunning the expensive ingest you already completed.
Common mistake: Calling the monolith 'simpler' because it is one file, while every failure costs a full rerun.
Follow-up they will ask: Where would you put the task boundaries in that notebook?
intermediateA six-task job failed on task 4; tasks 5 and 6 show Skipped and someone is about to hit Run now. What do you tell them?
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: Fix the cause and use Repair run — it re-executes the failed task and its skipped downstream branch while tasks 1 to 3 stay successful.
Why: Downstream tasks skip rather than fail when a dependency fails, so the run keeps an accurate record of what completed. Repair run reuses that state and restarts from the failure point, which is why task granularity matters: coarse tasks make repair redo work that was already fine.
Common mistake: Clicking Run now and paying for the first three tasks again — or double-writing output that is not idempotent.
Follow-up they will ask: How would you make that task safe to retry more than once?
intermediateTask A computes the window to process. How does task B get that value without writing it to a table or calling now() again?
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: Task values — dbutils.jobs.taskValues.set in A, .get in B keyed by the producing task, or a {{tasks.<key>.values.<name>}} reference in a task parameter.
Remember these
- Task values are small JSON, scoped to one job run
- get takes the producing taskKey plus a debugValue for interactive runs
- {{tasks.<key>.values.<name>}} injects them into non-notebook tasks
- One window per run is what makes retries and repairs deterministic
Why: Task values are small JSON values scoped to a single job run, set by one task and read by any downstream task that names the producer. The producer calls dbutils.jobs.taskValues.set with a key and a value; the consumer calls get with the producing taskKey, the key, and a debugValue so the notebook still runs interactively outside a job. For tasks that are not notebooks — a SQL task or a Python wheel — the same value can be injected through a parameter using the {{tasks.<task_key>.values.<name>}} reference, which keeps the plumbing out of the code. Why this matters beyond convenience: if each task computes its own window from the current time, two tasks in one run can disagree, and a retry an hour later processes a different slice than the original attempt. Deriving the window once and passing it forward makes the run's input deterministic, which is the precondition for a retry or a repair being safe.
dbutils.jobs.taskValues.set(key='window_end', value='2026-10-24T00:00:00')
# downstream task
end = dbutils.jobs.taskValues.get(
taskKey='bronze_ingest', key='window_end', debugValue='')Common mistake: Having every task call current_timestamp() independently, so a slow run straddles a day boundary and the tasks process different windows.
Follow-up they will ask: Where would you compute the window if two separate jobs have to agree on it?
intermediateYou wired four silver tasks to depend only on bronze, but the run timeline shows them starting one after another. Where do you look?
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: The dependency graph permits parallelism; the compute has to supply it — check the depends_on lists, then the cluster the tasks share and how it was sized.
Remember these
- An accidental depends_on between siblings serializes them silently
- Tasks sharing a job cluster compete for the same finite cores
- Separate new clusters show as long pending before short running
- Pools, policies and instance capacity cap parallelism from outside the job
Why: Start with the graph itself, because the most common cause is an accidental chain: one silver task listing another silver task in depends_on makes them sequential no matter what you intended, and the job's task graph view shows that immediately. If the graph is genuinely a fan-out, look at compute. Tasks sharing a job cluster run as separate Spark applications' worth of work on finite cores, so if the first task's stages consume every slot the others are effectively queued behind it. If each task was instead given its own new cluster, the timeline shows long pending segments before short execution segments — that is provisioning time, not contention. Then check for a ceiling outside the job: a cluster policy, a pool with a maximum, or cloud instance capacity, all of which show up as tasks waiting with no error. The run timeline is the evidence either way, since it separates each task's pending segment from its running segment.
Common mistake: Concluding Databricks does not parallelize tasks, when the graph has a stray dependency or the shared cluster has no spare slots.
Follow-up they will ask: The tasks do overlap but each one is slower than when it ran alone. What now?
beginnerLeadership announces a migration to Lakeflow and asks for a project plan. What do you tell them the project actually is?
Lakeflow: ingest, transform, orchestrate · tap to open the answer
Say this first: Lakeflow is the name over Connect, Declarative Pipelines, and Jobs — mostly a unification of Delta Live Tables and Workflows, plus managed ingestion.
Remember these
- Connect, Declarative Pipelines, and Jobs are the three parts
- Declarative Pipelines is the new name for Delta Live Tables
- Jobs is the new name for Workflows
- Existing pipelines and jobs do not have to be rewritten
Why: Three products that used to be separate sit under one name. Lakeflow Connect is managed connectors that pull from SaaS systems and databases. Lakeflow Declarative Pipelines is what was Delta Live Tables: you declare tables and the engine runs the graph. Lakeflow Jobs is what was Databricks Workflows: a DAG of tasks with retries and repair. Existing DLT pipelines and jobs keep working under the new names. The new thing to adopt, if you need it, is Connect, which replaces the extractor you would otherwise write for Salesforce or SQL Server. There is no flag day where pipelines must be rewritten.
Common mistake: Scheduling a migration project to move working DLT pipelines onto Lakeflow.
Follow-up they will ask: So what would you actually adopt Lakeflow Connect for?
intermediateYou have CSVs landing in a bucket and a Salesforce object to ingest. Which Lakeflow piece handles which, and why are they different?
Lakeflow: ingest, transform, orchestrate · tap to open the answer
Say this first: Files landing in object storage are Auto Loader. A SaaS API or a database is Lakeflow Connect, because someone has to speak that system's incremental protocol.
Remember these
- Auto Loader is for files that appear in object storage
- Connect is for APIs and databases, including CDC
- Connect owns auth, incremental state, and rate limits for that source
- Pick by source shape, not by which product is newer
Why: Auto Loader watches cloud storage, tracks which files it has seen in its checkpoint, and incrementally reads new ones. It knows nothing about Salesforce's API, cursors, or rate limits. Connect is a managed connector for a specific source: it handles authentication, incremental extraction, and change data capture, and lands the result as tables. Using Connect for files adds a service you do not need. Using a hand-rolled notebook for Salesforce means you own pagination, schema drift, and the retry policy forever. The split is by source shape, not by which product is newer.
Common mistake: Replacing a working Auto Loader path with Connect because Lakeflow is the new standard.
Follow-up they will ask: Where does the connector's incremental state live, and what happens if you reset it?
intermediateA declarative pipeline has three tables and nobody specified an order. How does it know bronze runs before gold?
Lakeflow: ingest, transform, orchestrate · tap to open the answer
Say this first: The engine reads the queries, sees which tables each one reads, and infers the dependency graph.
Remember these
- Dependencies come from which tables each query reads
- There is no depends_on between tables inside one pipeline
- The engine owns checkpoints, incremental state, and retries
- Expectations are constraints declared on the table, not a downstream check
Why: Each declaration is a table name plus the query that produces it. If silver reads bronze and gold reads silver, that is an edge, and the engine topological-sorts the graph before it runs anything. You do not write depends_on between tables inside the pipeline. The engine also owns the checkpoint and the incremental state for streaming tables, and it retries a failed table rather than you coding that. Expectations declared on a table — CONSTRAINT ... EXPECT — are evaluated as the data flows, and a failed expectation can drop the row, fail the row, or just record it. The graph you see in the pipeline UI is that inferred graph, not a diagram someone drew.
CREATE OR REFRESH STREAMING TABLE bronze.orders AS
SELECT * FROM STREAM read_files('/mnt/raw/orders', format => 'json');
CREATE OR REFRESH MATERIALIZED VIEW silver.orders AS
SELECT * FROM bronze.orders
WHERE order_id IS NOT NULLCommon mistake: Adding a job task per table to force an order the pipeline already infers.
Follow-up they will ask: What happens to gold if silver's expectation is set to fail the pipeline?
beginnerThe team says they have CI/CD because the notebooks are in Git. A job's schedule changed last week and nobody can say who did it. What is missing?
Asset Bundles: your workspace in git · tap to open the answer
Say this first: Git-backed notebooks version the code. The job definition, schedule, cluster spec, and permissions still live only in the workspace UI.
Remember these
- Git-backed notebooks version code, not the job around the code
- Schedule, cluster, parameters, and run-as live on the job
- A bundle is databricks.yml plus resource YAML in the same repo
- A schedule change becomes a pull request instead of a click
Why: A notebook in a repo tells you who changed the code. It does not tell you who changed the schedule, the cluster size, the parameters, or the run-as identity, because those are properties of the job, and the job was created by clicking. Two workspaces then drift: dev has a schedule somebody paused and prod has a cluster size somebody bumped, and there is no diff. A Databricks Asset Bundle puts the job, the pipeline, and the related resources in YAML next to the notebook, so a schedule change is a pull request. Until the job itself is in the repo, 'our code is in git' describes half the system.
Common mistake: Pointing at the notebook repo as evidence that production is reproducible.
Follow-up they will ask: Name three job settings that a notebook repo does not capture.
intermediateWalk through what happens from a merged pull request to a job existing in production.
Asset Bundles: your workspace in git · tap to open the answer
Say this first: CI runs databricks bundle validate on the pull request and databricks bundle deploy -t prod on merge, as a service principal.
Remember these
- validate runs on the pull request and changes nothing
- deploy -t prod applies the YAML to the production workspace
- Resources removed from YAML are removed from the workspace
- The deploy identity should be a service principal
Why: validate checks the YAML against the target without changing the workspace: bad references, missing variables, and permission problems fail here, on the pull request. deploy -t prod applies the bundle to the production workspace, creating or updating the jobs and pipelines the YAML describes and deleting bundle-managed resources that were removed from the file. run is for a developer exercising a job in dev, not for production. The identity that deploys matters as much as the command: a service principal owns the deployment, so production does not depend on the laptop or the employment of whoever last clicked. A human deploying from their laptop to prod is the failure mode the pipeline exists to remove.
databricks bundle validate -t prod
databricks bundle deploy -t prod
databricks bundle run -t dev daily_revenueCommon mistake: Deploying to production from a laptop with a personal token because the pipeline is not finished yet.
Follow-up they will ask: What does deploy do with a job that is in the workspace but no longer in the YAML?
intermediateDev and prod need different warehouses, different catalogs, and different schedules. How does one YAML describe both?
Asset Bundles: your workspace in git · tap to open the answer
Say this first: Targets. Each target supplies its own workspace host, mode, and variable values, and the resources reference those variables.
Remember these
- A target binds a workspace, a mode, and variable values
- Resources reference ${var.catalog} rather than a fixed name
- mode: development prefixes names and pauses schedules
- Prod runs as a service principal on the real schedule
Why: A target is an environment: a workspace URL, a mode, and a set of variable values. The job YAML says ${var.catalog} and ${var.warehouse_id} instead of a hardcoded name, and the dev target resolves them to the dev catalog while prod resolves them to prod. mode: development on the dev target prefixes resource names with the deploying user and pauses schedules, so two engineers deploying the same bundle do not overwrite each other's jobs or send real emails. Prod has no such prefix, a real schedule, and a run-as service principal. ${workspace.current_user.userName} and ${resources.jobs.x.id} cover the substitutions that should follow the workspace rather than a variable you set.
targets:
dev:
mode: development
workspace:
host: https://dev.cloud.databricks.com
variables:
catalog: dev_sales
prod:
workspace:
host: https://prod.cloud.databricks.com
variables:
catalog: prod_salesCommon mistake: Copying the YAML per environment and editing names by hand, which is the drift the bundle was meant to end.
Follow-up they will ask: What does mode: development change besides the name prefix?
beginnerA teammate wants to classify a million support tickets by calling the model API from a Python loop over a collected DataFrame. What do you suggest instead?
AI functions and Vector Search in SQL · tap to open the answer
Say this first: ai_query is a SQL expression the engine batches and distributes, so the rows stay in the table and the calls go out in batches.
Remember these
- ai_query is a SQL expression, not a notebook loop
- The engine batches rows and sends them to a serving endpoint
- Rows stay distributed instead of being collected to the driver
- The result is a normal column you can write with CREATE TABLE AS
Why: collect() pulls the million rows to the driver, and a Python loop then sends them one at a time, with you owning retries, rate limits, and the write-back. ai_query('endpoint', text_col) is an ordinary expression in a SELECT, so it runs where the data already is. The engine groups rows into batches and sends those to the serving endpoint, which is both faster and simpler than the loop. The result comes back as a column you can filter, group, or write with CREATE TABLE AS. You still pay per call, but you stop paying the engineering cost of the loop.
CREATE TABLE support.ticket_labels AS
SELECT ticket_id,
ai_query('ticket-classifier', comment) AS label
FROM support.tickets
WHERE created_date >= current_date() - 7Common mistake: Calling ai_query a Python UDF and expecting to manage the HTTP calls yourself.
Follow-up they will ask: If it is just SQL, why does the cost still scale with the number of rows?
intermediateYou only need a sentiment label, not a free-form answer. Which function do you reach for, and why does the choice matter?
AI functions and Vector Search in SQL · tap to open the answer
Say this first: Use a task function such as ai_analyze_sentiment or ai_classify, because the result is a constrained value you can group by rather than prose you have to parse.
Remember these
- Task functions return a constrained type for one job
- ai_classify, ai_extract, ai_summarize, and ai_forecast are SQL expressions
- Free-form model text is hard to group, join, or test
- ai_query with a response schema is the structured option
Why: ai_query with a prompt returns whatever the model writes, which changes shape between rows and between reruns. Task functions target one job: ai_classify picks from a label set you supply, ai_extract pulls fields into a structure, ai_summarize, ai_translate, ai_analyze_sentiment, ai_similarity, ai_forecast, and ai_parse_document each return a predictable type. A sentiment column that is always one of a few values can be grouped and joined. A paragraph cannot. When you do need ai_query, pass a response schema so the endpoint returns structured fields instead of text you split later.
SELECT ticket_id,
ai_classify(comment, ARRAY('billing', 'outage', 'other')) AS topic,
ai_analyze_sentiment(comment) AS sentiment
FROM support.ticketsCommon mistake: Using ai_query with a long prompt and then parsing the prose with string functions.
Follow-up they will ask: What does a response schema change about the value you get back?
intermediateThe classification query scanned 200 million rows and the invoice was painful. The WHERE clause looked selective. What is the most likely shape of the SQL?
AI functions and Vector Search in SQL · tap to open the answer
Say this first: The filter was applied to the model's output, so the endpoint was called for every row and only then were rows discarded.
Remember these
- Cost is one inference per row that reaches the function
- A filter on the model's output runs after the calls
- Filter on real columns first, then call the model
- The query profile shows how many rows entered the AI operator
Why: ai_query costs one inference per row that reaches it, and the engine does not know the model will reject most of them. WHERE ai_query(...) = 'urgent' evaluates the model for the whole table and filters afterwards. WHERE created_date >= ... AND region = 'EMEA' followed by ai_query in the SELECT list evaluates the model only for the rows that survive. The same applies inside a view: a downstream filter on the label does not push through the model call. Look at the query profile for the number of rows entering the AI operator. That number times the per-token price is the bill.
-- calls the model for every row
SELECT * FROM tickets WHERE ai_classify(comment, ARRAY('urgent','other')) = 'urgent';
-- calls it only for this week's rows
SELECT ai_classify(comment, ARRAY('urgent','other'))
FROM tickets WHERE created_date >= current_date() - 7Common mistake: Assuming the optimizer pushes a predicate through a model call the way it pushes one through a Parquet scan.
Follow-up they will ask: How would you check the profile to confirm how many calls you paid for?
Practice by topic
- Databricks Architecture
- Compute types
- SQL Warehouses
- Unity Catalog
- Delta Lake
- Time Travel & VACUUM
- Medallion Architecture
- Delta MERGE
- OPTIMIZE & Clustering
- Liquid Clustering
- Auto Loader
- Delta Live Tables
- Workflows
- Photon
- UniForm and Iceberg
- Row filters and column masks
- Delta Sharing and Clean Rooms
- System tables
- Predictive Optimization
- Serverless compute
- Streaming tables and materialized views
- Lakeflow
- Asset Bundles and CI/CD
- AI functions and Vector Search
