Interview/Databricks

Databricks interview questions: Databricks architecture

Interactive Databricks interview questions on Databricks architecture. Practice with the matching lesson. Databricks control plane is the SaaS UI and jobs service. Data plane is compute in your cloud account, next to your data.

Lesson · Simulation

What is the Databricks control plane versus the data plane?

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Databricks architecture

Compliance blocked a workspace rollout

Symptoms

  • Security could not point to where tables live
  • Vendor questionnaire said 'Databricks stores data'

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 9 of 9 questions.

beginner

What is the Databricks control plane versus the data plane?

Databricks architecture · tap to open the answer

Say this first: Control plane is Databricks SaaS (workspace, jobs, identity). Data plane is compute and storage in your cloud account.

Why: You click notebooks in the control plane. Clusters, executors, and table files run in your VPC next to S3/ADLS/GCS. Databricks does not keep your Parquet in the SaaS plane.

Common mistake: Thinking the workspace is where Spark executors run.

Follow-up they will ask: Where does a Jobs cluster actually start?

Lesson · Simulation

intermediate

Does Databricks store your fact tables?

Databricks architecture · tap to open the answer

Say this first: No. Tables stay in your bucket. The workspace orchestrates.

Why: Reads and writes hit s3://, abfss://, or gs://. The control plane sends API calls and gets job status / small action results. A collect() still OOMs the driver inside your VPC.

Common mistake: Uploading extracts into the workspace 'because it's Databricks'.

Follow-up they will ask: What belongs in Unity Catalog versus the storage account?

Lesson · Simulation

senior

Security asks: can PII leak through the control plane?

Databricks architecture · tap to open the answer

Say this first: Not as table files. Risk is notebooks, logs, collect(), and who can run compute that reads the lake.

Why: Files stay in the data plane. Notebook outputs, driver pulls, and mis-shared workspaces can still leak samples. Private link, cluster policies, and no collect() on PII are the controls. Unity Catalog governs names and grants; it is not a second copy of the files.

Senior depth: Commands down, status up, files never leave the data plane. Circle the driver in the customer VPC.

Common mistake: Answering only 'data never leaves our VPC' and stopping.

Follow-up they will ask: What would you put on the architecture diagram in an interview?

Lesson · Simulation

beginner

A new engineer asks why a cluster takes four minutes to start if Databricks is a SaaS product. What do you tell them?

Databricks architecture · tap to open the answer

Say this first: The control plane only sends an API call; the VMs are provisioned in your own cloud account, and that provisioning is the wait.

Remember these

  • Control plane sends the request; your cloud account provisions the VMs
  • Cluster event log is the timeline for every start and failure
  • Capacity errors in the event log come from the cloud provider, not Databricks
  • Pools and serverless remove provisioning from the critical path

Why: Clicking Start calls the Databricks control plane, which asks your cloud provider for instances in your VPC or VNet subnets. Those VMs boot, pull the Databricks Runtime image, and the driver and executors register with each other before the cluster is usable. The cluster event log narrates exactly that: STARTING, instance acquisition, driver healthy, running. If the node type is scarce in that region the event log shows a capacity error instead, which is a cloud-provider problem, not a Databricks one. Instance pools and serverless cut the wait because the machines are already running before you ask.

Common mistake: Assuming Databricks keeps machines warm for your workspace by default.

Follow-up they will ask: What would you read first if a cluster never reached the running state?

Lesson · Simulation

intermediate

Draw the line for me: name things the control plane stores and things that never leave your cloud account.

Databricks architecture · tap to open the answer

Say this first: Control plane holds notebook source, job and cluster definitions, query history and Unity Catalog metadata; table files, executor heap, and the disk cache stay in your account.

Remember these

  • Notebook source, job config, query history: control plane
  • Parquet, _delta_log, executor memory, disk cache: your account
  • Notebook and SQL result previews are persisted control-plane side
  • Unity Catalog stores metadata and grants, not data files

Why: Databricks-managed storage holds notebook source and revision history, job and cluster configuration, query history, cluster event logs, and the Unity Catalog metadata and grants. Your cloud account holds the Parquet data files, the _delta_log, the executor JVM heap, and the local SSD disk cache on the workers. The detail that catches people out is command output: notebook results and SQL result previews are persisted in the control plane up to a size limit, so a display() of PII rows does put a sample outside your account. That is why the honest answer is 'table files never leave the data plane' rather than 'nothing leaves'. Unity Catalog is names and permissions in the control plane; it is not a second copy of the bytes.

Common mistake: Saying 'nothing leaves our account' and forgetting notebook outputs and query result previews.

Follow-up they will ask: Where does a display() of 10,000 PII rows end up?

Lesson · Simulation

advanced

With serverless compute enabled, where does the Spark driver actually run, and what does that change for a security review?

Databricks architecture · tap to open the answer

Say this first: In a Databricks-managed account rather than your VPC, so the controls shift from your network rules to serverless connectivity settings and Unity Catalog credentials.

Remember these

  • Serverless compute runs in the Databricks account, not your VPC
  • Storage access goes through Unity Catalog credentials, not instance profiles
  • Your VPC security groups no longer govern that traffic
  • Network connectivity configuration plus private endpoints are the replacement

Why: Classic compute puts the driver and executors on VMs in your subnets, reaching storage with an instance profile or managed identity you attached. Serverless warehouses, jobs, and notebooks run that same Spark work on Databricks-managed compute, and it reaches your storage through Unity Catalog storage credentials rather than cluster-attached cloud identities. So the questionnaire line 'all compute runs in our VPC' stops being true the moment serverless is switched on, while 'table files stay in our bucket' remains true. The controls that replace your security groups are the serverless network connectivity configuration, private endpoints from serverless to your storage, and the fact that data access is only possible through Unity Catalog. Init scripts and instance profiles are not the extension points they were, which is itself part of the isolation story.

-- Which warehouses are serverless?
SELECT name, warehouse_type FROM system.compute.warehouses

Senior depth: Serverless moves the execution boundary, not the data boundary. I would keep two documented models side by side — classic compute governed by our VPC, serverless governed by the connectivity configuration and Unity Catalog — and say which products use which. Then I would make the choice enforceable rather than cultural: restrict who can create serverless resources, require private connectivity to storage, and remove direct path access so Unity Catalog is the only route to the data on either model.

Common mistake: Repeating 'our compute never leaves our VPC' in a review after enabling serverless warehouses.

Follow-up they will ask: Which identity reads storage when a serverless warehouse queries a managed table?

Lesson · Simulation

advanced

A workspace admin cannot create the metastore or add a new user and insists they already have admin rights. Explain what is happening.

Databricks architecture · tap to open the answer

Say this first: Metastores, identities, and credential configurations are account-level objects; workspace admin is a different scope from account admin.

Remember these

  • Account console: metastore, identities, workspaces, credentials
  • Workspace admin: compute, policies, jobs, secrets, object ACLs
  • One metastore per region, assigned to workspaces afterwards
  • Identity federation pushes account groups down into workspaces

Why: Databricks splits administration in two. The account console owns the Unity Catalog metastore, workspace provisioning, users, groups, service principals, and the network and credential configurations. Workspace admin owns compute, cluster policies, jobs, secrets, and workspace object ACLs. A metastore is created once per region in the account console and then assigned to workspaces, so no amount of workspace entitlement produces that button. With identity federation, groups are managed at the account level and assigned down into workspaces, which is why 'add this person' is also an account-level action. The fix is to involve an account admin or get that role, not to grant more workspace permissions.

Senior depth: The practical consequence is that the account is the real security boundary and the workspace is an operational one. I keep account admin small and audited, manage groups once at the account level through SCIM so a leaver is removed everywhere at once, and never let a team solve a permission problem by creating a new workspace. Anything account-level — metastore assignment, credentials, network config — goes through Terraform so the configuration is reviewable instead of clicked.

Common mistake: Hunting for a Create Metastore action inside Catalog Explorer in the workspace.

Follow-up they will ask: Where would you manage group membership when ten workspaces share one metastore?

Lesson · Simulation

senior

Why would you run three workspaces instead of one workspace with three catalogs?

Databricks architecture · tap to open the answer

Say this first: A workspace is the blast radius for compute, secrets, and schedules; a catalog is the governance boundary for data — you split workspaces to isolate people and compute, not tables.

Remember these

  • Catalogs separate data; workspaces separate compute, secrets, and admins
  • One metastore can be shared by many workspaces in a region
  • Catalog bindings stop a dev workspace writing a prod catalog
  • More workspaces means more configuration to keep identical in code

Why: One metastore can serve many workspaces in a region, so separating data does not require separating workspaces: catalogs, grants, and catalog-to-workspace bindings already do that. What a separate workspace buys you is a separate set of cluster policies, secret scopes, job schedules, network configuration, and admins — the right boundary between production and development, or between a regulated business unit and everyone else. Bindings then make the isolation mutual: a prod catalog bound to prod workspaces cannot be written from a dev notebook even if someone over-grants. The cost is configuration drift, because every workspace now needs the same policies and permissions, which is why multi-workspace setups live or die on Terraform and Databricks Asset Bundles. Splitting per team without that discipline is how organisations end up with ten unmanaged workspaces and a legacy hive metastore in each.

Senior depth: I decide it by asking what I am trying to contain. If the answer is 'who can read this table', that is a catalog and a grant. If the answer is 'a dev cannot attach to compute that holds prod credentials, and a dev job cannot run on the prod schedule', that is a workspace. Then I bind prod catalogs to prod workspaces so the two boundaries reinforce each other, and define every workspace from the same Terraform module so the fourth workspace is not a snowflake.

Common mistake: Creating a workspace per team for data isolation that catalogs and grants already provide.

Follow-up they will ask: What makes a prod and dev split worth a second workspace rather than a second catalog?

Lesson · Simulation

architect

Compliance requires that no cluster has a public IP and that traffic to object storage does not cross the internet. What are you configuring?

Databricks architecture · tap to open the answer

Say this first: Secure cluster connectivity for no public IPs, a customer-managed VPC with private subnets, private or gateway endpoints to storage, and private link to the control plane.

Remember these

  • Secure cluster connectivity: no public IPs, outbound tunnel only
  • Customer-managed VPC gives you subnets, routes, and security groups
  • Private or gateway endpoints keep storage traffic off the internet
  • Serverless is governed separately, not by your VPC rules

Why: With secure cluster connectivity the cluster VMs get no public IP address and no inbound ports; instead each instance opens an outbound tunnel to the control plane relay, so the control plane never initiates a connection into your network. Bringing your own VPC lets you own the subnets, route tables, and security groups the compute runs in. Storage traffic is kept off the internet with a gateway or private endpoint to the object store so it stays on the cloud backbone. Private link covers the other two hops: users to the workspace on the front end, and compute to the control plane on the back end. Serverless needs its own answer, because none of your VPC controls apply to it — that is a network connectivity configuration plus private connectivity from serverless to your storage. Customer-managed keys cover the encryption question that usually arrives in the same conversation.

Senior depth: I present it as four hops and ask which of them compliance cares about: user to workspace, control plane to compute, compute to storage, and compute to the internet. Secure cluster connectivity and private link close the first two, endpoints and the credential model close the third, and an egress policy closes the fourth. Then I state the serverless exception out loud rather than letting someone discover it in an audit, because the whole value of this diagram is that it is the answer we can defend in writing.

Common mistake: Assuming private link to the control plane also covers serverless compute automatically.

Follow-up they will ask: Which part of this design does nothing for a serverless SQL warehouse?

Lesson · Simulation