Databricks beginner interview questions

Short: Control plane is Databricks SaaS (workspace, jobs, identity). Data plane is compute and storage in your cloud account.

Detailed: You click notebooks in the control plane. Clusters, executors, and table files run in your VPC next to S3/ADLS/GCS. Databricks does not keep your Parquet in the SaaS plane.

Common mistake: Thinking the workspace is where Spark executors run.

Follow-up: Where does a Jobs cluster actually start?

Lesson · Simulation

Short: No. Tables stay in your bucket. The workspace orchestrates.

Detailed: Reads and writes hit s3://, abfss://, or gs://. The control plane sends API calls and gets job status / small action results. A collect() still OOMs the driver inside your VPC.

Common mistake: Uploading extracts into the workspace 'because it's Databricks'.

Follow-up: What belongs in Unity Catalog versus the storage account?

Lesson · Simulation

Short: Parquet files plus a transaction log (_delta_log) that gives ACID on object storage.

Detailed: Every write commits JSON (and checkpoints). Time travel is 'read version N'. OPTIMIZE and MERGE use the log to know which files are current.

Common mistake: Delta as a file extension you rename to .delta.

Follow-up: What happens if you read the Parquet files and ignore the log?

Lesson · Simulation

Short: Optimistic concurrency: each commit names the files added/removed. Conflicts abort.

Detailed: Concurrent overwrites of the same partition can still fail — you retry or isolate. The log is the source of truth, not 'whoever finished last silently'.

Common mistake: Writing Parquet next to Delta and expecting ACID.

Follow-up: How do you time-travel to yesterday 8am?

Lesson · Simulation

Short: Read the older version: SELECT * FROM gold.revenue VERSION AS OF 812, or TIMESTAMP AS OF '2026-09-12 09:00:00'.

Detailed: Every Delta write appends an ordered JSON commit to _delta_log, and the superseded data files are still referenced by earlier versions, so reading an old version is a metadata read against a different file list — nothing is restored and nothing is written. DESCRIBE HISTORY gold.revenue lists versions, timestamps, and the operation behind each commit so you can pick the right one.

Common mistake: Assuming the data is gone and asking the platform team for a bucket-level restore.

Follow-up: Does that query change the table in any way?

Lesson · Simulation

Short: Use RESTORE TABLE t TO VERSION AS OF 812 — it writes a new forward commit, so the rollback itself is auditable.

Detailed: RESTORE does not rewind the log; it adds a new version whose file list matches the target version, so DESCRIBE HISTORY shows a RESTORE operation and downstream readers just see another new version. Hand-rolling it with INSERT OVERWRITE loses that audit trail and splits the read and write into steps that can interleave with other writers.

Common mistake: Trying to delete JSON files out of _delta_log to 'undo' a commit.

Follow-up: What happens to a structured streaming reader on that table after a RESTORE?

Lesson · Simulation

Short: catalog.schema.table (plus volumes for files).

Detailed: It is the governance plane: grants, lineage, audit. Table files still live in cloud storage. hive_metastore is the old two-level world.

Common mistake: Unity Catalog as a replacement for Delta files.

Follow-up: Where do you grant SELECT — catalog, schema, or table?

Lesson · Simulation

Short: Workspace access ≠ UC privilege on that catalog/schema/table.

Detailed: They need USE CATALOG, USE SCHEMA, and SELECT (or a share). Cluster must be UC-enabled. A personal compute with the wrong catalog still fails.

Common mistake: Making the table public because 'they're in the workspace'.

Follow-up: What's the difference between a storage credential and an external location?

Lesson · Simulation

Short: Bronze = raw replay. Silver = conformed truth. Gold = product metrics.

Detailed: Bronze keeps original bytes. Silver types, dedupes, joins. Gold is the grain BI/ML consume. Arrows only flow downstream.

Common mistake: Medallion as three folders with the same messy table copied three times.

Follow-up: Who is allowed to write back into bronze?

Lesson · Simulation

Short: Every filter becomes a full scan of landing JSON, and metrics disagree.

Detailed: Bronze has no stable schema contract. Gold is small, named, and owned. Silver is what jobs share so gold doesn't re-parse.

Common mistake: Skipping silver 'to move faster'.

Follow-up: Where do quality expectations live?

Lesson · Simulation

Short: Match keys, update/insert/delete, rewrite touched files, one atomic commit.

Detailed: It is not an in-place row edit on Parquet. Files that contain matched rows are rewritten. Tiny files after many MERGEs need OPTIMIZE.

Common mistake: Thinking MERGE updates a row in a single Parquet file like a database page.

Follow-up: What happens to unmatched target rows by default?

Lesson · Simulation

Short: Those rows lived in large files (or unclustered files) so whole files were copied.

Detailed: Partition / liquid clustering / Z-ORDER so a key lives in fewer files. MERGE ON a selective condition. Don't MERGE a full snapshot if you have change data.

Common mistake: OPTIMIZE after every MERGE in the same transaction as a religion without measuring file size.

Follow-up: How do you MERGE only yesterday's partition?

Lesson · Simulation

Short: Compacts small files into larger ones so scans launch fewer tasks.

Detailed: It rewrites data files and commits a new Delta version. It does not change business grain. Z-ORDER / CLUSTER BY is extra: it colocates filter columns.

Common mistake: OPTIMIZE as a vacuum of old versions.

Follow-up: Does OPTIMIZE make MERGE faster, slower, or both?

Lesson · Simulation

Short: Z-ORDER is a rewrite you schedule. Liquid clustering is incremental on write (newer DBR).

Detailed: High-cardinality id Z-ORDER can be wasted. Cluster by columns you actually filter. Don't Z-ORDER every column.

Common mistake: Z-ORDER(customer_id, date, country, sku, event_id) 'to be safe'.

Follow-up: How do you prove skipping worked?

Lesson · Simulation

Short: CLUSTER BY (event_date, customer_id) — liquid clustering replaces both the Hive partitioning and the Z-ORDER.

Detailed: CREATE TABLE ... CLUSTER BY (cols) lets Delta organize files by those keys and maintain the layout incrementally as data lands, instead of pinning a physical directory structure and periodically re-sorting the whole table. You cannot have both: a table is either partitioned or clustered, and CLUSTER BY is rejected alongside PARTITIONED BY.

Common mistake: Treating liquid clustering as a new name for Z-ORDER and keeping the partition columns as well.

Follow-up: What actually performs the clustering after rows are written?

Lesson · Simulation

Short: ALTER TABLE t CLUSTER BY (region) changes the keys with no rewrite of existing data; repartitioning a Hive-partitioned table means recreating it and rewriting every byte.

Detailed: Clustering keys are table metadata, so the ALTER applies to subsequent writes and to subsequent OPTIMIZE runs, which cluster new and touched data incrementally — old files stay as they are until something rewrites them. Changing PARTITIONED BY is not an ALTER at all, which is exactly why teams stayed stuck with a bad partition column for years.

Common mistake: Expecting the ALTER to re-lay-out history instantly, then declaring clustering broken when the first query looks the same.

Follow-up: What would you run if you genuinely needed history in the new layout today?

Lesson · Simulation

Short: Incrementally ingest new files without listing the whole bucket every run.

Detailed: cloudFiles source + notification/listing + checkpoint. It is not a replacement for Delta MERGE logic — it lands bronze.

Common mistake: Every job does dbutils.fs.ls on the landing zone.

Follow-up: Where is Auto Loader's progress stored?

Lesson · Simulation

Short: Huge landing zones: listing is slow and expensive. Notifications scale.

Detailed: File events (SNS/SQS, Event Grid, Pub/Sub) tell you what landed. Listing is fine for small prefixes. Misconfigured notifications skip files or duplicate if you also copy.

Common mistake: Notification mode without the cloud event pipeline actually wired.

Follow-up: How does schema inference evolve with rescued data?

Lesson · Simulation

Short: Databricks' native vectorized engine for eligible SQL/DataFrame operators.

Detailed: It runs in C++ on the data plane, not a new storage format. If the plan is eligible, scans/joins/aggs get faster. UDFs and some expressions fall back to the JVM.

Common mistake: Photon as a magic switch that always 10×s every notebook.

Follow-up: Where do you see whether Photon ran?

Lesson · Simulation

Short: That operator (and often the pipeline around it) falls back.

Detailed: SQL UI / explain: Photon vs JVM. Rewrite with built-ins. Pandas UDFs are still not Photon-native.

Common mistake: Turning Photon on and keeping the UDF.

Follow-up: Does Photon run on all-purpose, jobs, and warehouses?

Lesson · Simulation

Short: All-purpose: interactive notebooks, stays on. Jobs: start, run, terminate. Warehouse: BI/Photon, not notebooks.

Detailed: Wrong compute is the expensive mistake that never shows up as a Spark bug. ETL on all-purpose burns idle hours. Tableau on an all-purpose cluster fights notebooks for slots.

Common mistake: One mega cluster for explore, ETL, and BI.

Follow-up: Which product should a nightly MERGE use?

Lesson · Simulation

Short: It stays on, keeps leftover libraries/cache, and you pay for idle.

Detailed: A forgotten notebook pins a 32-worker cluster overnight. Job clusters get a clean Spark version and die when the task ends. Cluster policies should block oversized all-purpose.

Common mistake: Scheduling Workflows on the team's shared interactive cluster 'so it's warm'.

Follow-up: What still belongs on all-purpose?

Lesson · Simulation

Short: A warehouse is a SQL endpoint built for many short concurrent queries: Photon on by default, result caching, auto-stop, and extra clusters when concurrency rises.

Detailed: An all-purpose cluster runs notebooks and BI inside the same Spark application, so one runaway cell starves the dashboard and there is no per-query profile for the analyst. A warehouse gives JDBC/ODBC endpoints, Query History with a profile per statement, and scale-out for concurrency instead of fighting notebooks for slots.

Common mistake: Handing BI the JDBC URL of the team's interactive cluster because it is already warm.

Follow-up: Classic, Pro, or Serverless — which would you pick and what changes?

Lesson · Simulation

Short: No — size is scale-up for one query's horsepower; concurrency needs more clusters, so raise max clusters.

Detailed: Cluster size (2X-Small through 4X-Large) decides how much compute a single query can use; min/max clusters decides how many queries run in parallel before they queue. If Query History shows long queue time against short execution time, it is a concurrency problem, and upsizing just makes each idle cluster more expensive.

Common mistake: Jumping from Small to 2X-Large, watching the bill grow, and finding the queue still there.

Follow-up: Where in Query History do you separate queue time from execution time?

Lesson · Simulation

Short: You declare tables and expectations; DLT infers the graph and runs in order.

Detailed: dlt.read / @dlt.table instead of a 12-task Workflow of notebooks. Expectations drop or fail bad rows. Still Delta under the hood.

Common mistake: DLT as a replacement for Spark.

Follow-up: How does DLT know bronze runs before gold?

Lesson · Simulation

Short: Expectations are cataloged, metrics are visible, and bad rows can fail the pipeline.

Detailed: A silent filter hides data quality. DLT event log shows expectation pass rate. That's what on-call wants at 3am.

Common mistake: expect_or_drop everything so the pipeline never fails.

Follow-up: When is a Workflow of notebooks still better than DLT?

Lesson · Simulation

Short: A failure in cell 37 re-runs all 40 cells, and you get no per-step retry, no parallelism, and no per-step timing.

Detailed: A Databricks Job is a DAG of tasks wired with depends_on, so independent steps fan out in parallel, each task carries its own retry policy and timeout, and the run timeline gives you duration per task. Inside one notebook Spark sees a single linear sequence, so recovery means rerunning the expensive ingest you already completed.

Common mistake: Calling the monolith 'simpler' because it is one file, while every failure costs a full rerun.

Follow-up: Where would you put the task boundaries in that notebook?

Lesson · Simulation

Short: Fix the cause and use Repair run — it re-executes the failed task and its skipped downstream branch while tasks 1 to 3 stay successful.

Detailed: Downstream tasks skip rather than fail when a dependency fails, so the run keeps an accurate record of what completed. Repair run reuses that state and restarts from the failure point, which is why task granularity matters: coarse tasks make repair redo work that was already fine.

Common mistake: Clicking Run now and paying for the first three tasks again — or double-writing output that is not idempotent.

Follow-up: How would you make that task safe to retry more than once?

Lesson · Simulation