Databricks beginner interview questions

Short: Control plane is Databricks SaaS (workspace, jobs, identity). Data plane is compute and storage in your cloud account.

Detailed: You click notebooks in the control plane. Clusters, executors, and table files run in your VPC next to S3/ADLS/GCS. Databricks does not keep your Parquet in the SaaS plane.

Common mistake: Thinking the workspace is where Spark executors run.

Follow-up: Where does a Jobs cluster actually start?

Lesson · Simulation

Short: No. Tables stay in your bucket. The workspace orchestrates.

Detailed: Reads and writes hit s3://, abfss://, or gs://. The control plane sends API calls and gets job status / small action results. A collect() still OOMs the driver inside your VPC.

Common mistake: Uploading extracts into the workspace 'because it's Databricks'.

Follow-up: What belongs in Unity Catalog versus the storage account?

Lesson · Simulation

Short: Parquet files plus a transaction log (_delta_log) that gives ACID on object storage.

Detailed: Every write commits JSON (and checkpoints). Time travel is 'read version N'. OPTIMIZE and MERGE use the log to know which files are current.

Common mistake: Delta as a file extension you rename to .delta.

Follow-up: What happens if you read the Parquet files and ignore the log?

Lesson · Simulation

Short: Optimistic concurrency: each commit names the files added/removed. Conflicts abort.

Detailed: Concurrent overwrites of the same partition can still fail — you retry or isolate. The log is the source of truth, not 'whoever finished last silently'.

Common mistake: Writing Parquet next to Delta and expecting ACID.

Follow-up: How do you time-travel to yesterday 8am?

Lesson · Simulation

Short: catalog.schema.table (plus volumes for files).

Detailed: It is the governance plane: grants, lineage, audit. Table files still live in cloud storage. hive_metastore is the old two-level world.

Common mistake: Unity Catalog as a replacement for Delta files.

Follow-up: Where do you grant SELECT — catalog, schema, or table?

Lesson · Simulation

Short: Workspace access ≠ UC privilege on that catalog/schema/table.

Detailed: They need USE CATALOG, USE SCHEMA, and SELECT (or a share). Cluster must be UC-enabled. A personal compute with the wrong catalog still fails.

Common mistake: Making the table public because 'they're in the workspace'.

Follow-up: What's the difference between a storage credential and an external location?

Lesson · Simulation

Short: Bronze = raw replay. Silver = conformed truth. Gold = product metrics.

Detailed: Bronze keeps original bytes. Silver types, dedupes, joins. Gold is the grain BI/ML consume. Arrows only flow downstream.

Common mistake: Medallion as three folders with the same messy table copied three times.

Follow-up: Who is allowed to write back into bronze?

Lesson · Simulation

Short: Every filter becomes a full scan of landing JSON, and metrics disagree.

Detailed: Bronze has no stable schema contract. Gold is small, named, and owned. Silver is what jobs share so gold doesn't re-parse.

Common mistake: Skipping silver 'to move faster'.

Follow-up: Where do quality expectations live?

Lesson · Simulation

Short: Match keys, update/insert/delete, rewrite touched files, one atomic commit.

Detailed: It is not an in-place row edit on Parquet. Files that contain matched rows are rewritten. Tiny files after many MERGEs need OPTIMIZE.

Common mistake: Thinking MERGE updates a row in a single Parquet file like a database page.

Follow-up: What happens to unmatched target rows by default?

Lesson · Simulation

Short: Those rows lived in large files (or unclustered files) so whole files were copied.

Detailed: Partition / liquid clustering / Z-ORDER so a key lives in fewer files. MERGE ON a selective condition. Don't MERGE a full snapshot if you have change data.

Common mistake: OPTIMIZE after every MERGE in the same transaction as a religion without measuring file size.

Follow-up: How do you MERGE only yesterday's partition?

Lesson · Simulation

Short: Compacts small files into larger ones so scans launch fewer tasks.

Detailed: It rewrites data files and commits a new Delta version. It does not change business grain. Z-ORDER / CLUSTER BY is extra: it colocates filter columns.

Common mistake: OPTIMIZE as a vacuum of old versions.

Follow-up: Does OPTIMIZE make MERGE faster, slower, or both?

Lesson · Simulation

Short: Z-ORDER is a rewrite you schedule. Liquid clustering is incremental on write (newer DBR).

Detailed: High-cardinality id Z-ORDER can be wasted. Cluster by columns you actually filter. Don't Z-ORDER every column.

Common mistake: Z-ORDER(customer_id, date, country, sku, event_id) 'to be safe'.

Follow-up: How do you prove skipping worked?

Lesson · Simulation

Short: Incrementally ingest new files without listing the whole bucket every run.

Detailed: cloudFiles source + notification/listing + checkpoint. It is not a replacement for Delta MERGE logic — it lands bronze.

Common mistake: Every job does dbutils.fs.ls on the landing zone.

Follow-up: Where is Auto Loader's progress stored?

Lesson · Simulation

Short: Huge landing zones: listing is slow and expensive. Notifications scale.

Detailed: File events (SNS/SQS, Event Grid, Pub/Sub) tell you what landed. Listing is fine for small prefixes. Misconfigured notifications skip files or duplicate if you also copy.

Common mistake: Notification mode without the cloud event pipeline actually wired.

Follow-up: How does schema inference evolve with rescued data?

Lesson · Simulation

Short: Databricks' native vectorized engine for eligible SQL/DataFrame operators.

Detailed: It runs in C++ on the data plane, not a new storage format. If the plan is eligible, scans/joins/aggs get faster. UDFs and some expressions fall back to the JVM.

Common mistake: Photon as a magic switch that always 10×s every notebook.

Follow-up: Where do you see whether Photon ran?

Lesson · Simulation

Short: That operator (and often the pipeline around it) falls back.

Detailed: SQL UI / explain: Photon vs JVM. Rewrite with built-ins. Pandas UDFs are still not Photon-native.

Common mistake: Turning Photon on and keeping the UDF.

Follow-up: Does Photon run on all-purpose, jobs, and warehouses?

Lesson · Simulation

Short: All-purpose: interactive notebooks, stays on. Jobs: start, run, terminate. Warehouse: BI/Photon, not notebooks.

Detailed: Wrong compute is the expensive mistake that never shows up as a Spark bug. ETL on all-purpose burns idle hours. Tableau on an all-purpose cluster fights notebooks for slots.

Common mistake: One mega cluster for explore, ETL, and BI.

Follow-up: Which product should a nightly MERGE use?

Lesson · Simulation

Short: It stays on, keeps leftover libraries/cache, and you pay for idle.

Detailed: A forgotten notebook pins a 32-worker cluster overnight. Job clusters get a clean Spark version and die when the task ends. Cluster policies should block oversized all-purpose.

Common mistake: Scheduling Workflows on the team's shared interactive cluster 'so it's warm'.

Follow-up: What still belongs on all-purpose?

Lesson · Simulation

Short: You declare tables and expectations; DLT infers the graph and runs in order.

Detailed: dlt.read / @dlt.table instead of a 12-task Workflow of notebooks. Expectations drop or fail bad rows. Still Delta under the hood.

Common mistake: DLT as a replacement for Spark.

Follow-up: How does DLT know bronze runs before gold?

Lesson · Simulation

Short: Expectations are cataloged, metrics are visible, and bad rows can fail the pipeline.

Detailed: A silent filter hides data quality. DLT event log shows expectation pass rate. That's what on-call wants at 3am.

Common mistake: expect_or_drop everything so the pipeline never fails.

Follow-up: When is a Workflow of notebooks still better than DLT?

Lesson · Simulation