Databricks interview questions: Serverless compute: where the VMs went

Say this first: Classic compute provisions new VMs in your own cloud account on demand; serverless draws machines from a pool Databricks already has running.

Why: When you start a classic cluster, the control plane asks your cloud provider for instances in your VPC, those VMs boot, pull the runtime image, and the driver and executors register with each other. All of that is real provisioning time, and the cluster event log narrates each step. Serverless compute skips provisioning entirely: Databricks keeps a warm pool of capacity in its own account and attaches some of it to your workload, so startup is seconds. Your data has not moved — it is still in your storage account, read through Unity Catalog credentials. What moved is where the compute runs and who pays for keeping it warm, which is reflected in a higher DBU rate.

Common mistake: Saying serverless means 'no servers', instead of 'someone else already started them'.

Follow-up they will ask: Who is paying for that pool to sit warm when nobody is querying?

Say this first: Bursty work wins because you stop paying the moment it ends; a cluster pinned busy all day loses, because you pay the premium rate for every hour anyway.

Why: Cost is rate multiplied by time, and serverless changes both terms. The rate goes up, but the billed time collapses to actual work because there is no provisioning window and no idle cluster waiting for the next query. An analyst warehouse serving sporadic dashboard hits spends most of the day idle on classic, and that idle is the bill; on serverless it is near zero. A long-running nightly ETL that saturates a cluster for six straight hours has almost no idle to recover, so the higher rate is close to pure markup. The honest test is to pull the actual utilisation from system.billing.usage and the job run timeline rather than arguing from the price list.

SELECT sku_name, date_trunc('DAY', usage_start_time) AS d, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_start_time >= current_date() - INTERVAL 30 DAYS
GROUP BY ALL ORDER BY d

Common mistake: Comparing the two DBU rates and stopping, without counting the idle hours classic bills for.

Follow-up they will ask: How would you estimate the idle fraction of an existing warehouse before migrating it?

Say this first: Serverless trades configurability for speed: cluster-scoped init scripts, arbitrary instance types, and some library installation paths are constrained.

Why: The warm pool only works because the machines are standardised, so the hooks that let you reshape a classic cluster are not all available. Cluster-scoped init scripts, choosing a specific instance family, attaching an instance profile, and installing anything that needs root are the usual casualties. Python dependencies are expected to come through supported paths — environment specifications and workspace or Unity Catalog volume artefacts — rather than a script that apt-installs a native library at boot. The practical move is to read what the init script actually does: most of them set an environment variable, install a package, or configure logging, and each of those has a supported equivalent. If the script genuinely needs root or a native system library, that workload stays on classic.

Common mistake: Treating serverless as a drop-in swap for the compute block in an existing job definition.

Follow-up they will ask: Your init script sets three environment variables — what replaces it?

Say this first: Serverless egress leaves a Databricks-managed network, so you configure network connectivity and private endpoints rather than relying on being inside the VPC.

Why: The implicit assumption in the classic setup was that the executor's source address was one your firewall already trusted. Serverless compute runs outside your VPC, so that assumption disappears and the connection is refused or times out with nothing useful in the Spark logs. The replacement is explicit: a network connectivity configuration attached to the workspace, with private endpoints to the resources serverless needs to reach, and firewall rules written against that path instead of against your subnet CIDRs. For public endpoints you may also need to allow the documented egress addresses on the other side. The reason this surprises teams is that the failure looks like an application bug — a timeout in a notebook cell — rather than the networking change it actually is.

Senior depth: I treat 'can this workload reach its dependencies' as a first-class migration criterion, right next to cost. Before moving anything I list the outbound dependencies per job — storage, metastores, SaaS APIs, on-prem databases over a gateway — and mark which ones are reachable only because of VPC placement. Those jobs need a connectivity configuration and private endpoints designed first, or they stay on classic. Doing it the other way round means discovering network requirements one production failure at a time, which is both slower and much more visible.

Common mistake: Debugging the HTTP client for a day before checking where the request is now coming from.

Follow-up they will ask: What would you check first to prove the failure is network rather than code?

Say this first: Compare DBUs per run rather than per hour: if the job already saturated its cluster with little idle, you paid the premium rate for the same work.

Why: Pull system.billing.usage filtered to the job for a window before and after the change and normalise by run count, so you are comparing cost per execution rather than cost per day. Then put the job run timeline next to it: if classic runs show the cluster busy end to end with a short start-up tail, there was never much idle to recover and the higher rate goes straight through to the bill. The other common cause is that serverless removed the startup wait, so the job now finishes faster and somebody increased its frequency, which is a real behavioural change hiding inside a cost regression. Check whether concurrency changed too, since serverless scales out more readily and a job that used to queue may now run several tasks in parallel at full rate. Decide from that evidence: revert, resize, or keep it because the latency improvement is worth the money.

SELECT usage_metadata.job_id, COUNT(DISTINCT usage_metadata.job_run_id) AS runs,
       SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_metadata.job_id IS NOT NULL
  AND usage_start_time >= current_date() - INTERVAL 14 DAYS
GROUP BY ALL

Senior depth: Cost regressions after a compute change are usually a behaviour change wearing a cost change's clothes. So the first thing I establish is whether the unit of work is the same — same run count, same data volume, same concurrency — before I touch the compute decision. If the work really is identical and serverless is more expensive, that is a correct result for a saturated workload and I move it back without embarrassment. If the work grew because the platform got faster, that is a product decision, and I would make it explicitly rather than let it land as an unexplained invoice line.

Common mistake: Reverting on the daily total without checking whether run count or concurrency changed.

Follow-up they will ask: If the latency win is genuinely worth it, how would you say so in cost terms?

Say this first: Rank by idle fraction and startup overhead: spiky warehouses and short frequent jobs move first, saturated long-running ETL moves last or never.

Why: Build the list from data rather than opinion. For warehouses, compare billed uptime against actual query time to get an idle fraction; anything mostly idle is a strong candidate because serverless deletes exactly that cost. For jobs, compare cluster startup time against total run duration in the run timeline; a five-minute job paying four minutes of boot is wasting most of its wall clock. Then subtract the blockers — init scripts, specific instance types, instance profiles, VPC-only network dependencies — because those decide feasibility regardless of the economics. Move in that order, measure cost per run before and after on each wave, and keep the classic definition until the new one has survived a full cycle. The jobs left at the bottom of the list are the ones where classic is genuinely the right answer.

Senior depth: I frame it as a portfolio with evidence per workload, because 'serverless everywhere' and 'serverless nowhere' are both decisions made without data. The ordering that works is cheapest proof first: the idle warehouses, where the saving is obvious and the blast radius is a dashboard. That buys credibility for the harder conversations about network-coupled jobs. I also write down the criteria that would move something back — cost per run above a threshold, a dependency that needs root — so the migration has an exit condition instead of becoming a one-way mandate.

Common mistake: Running a workspace-wide migration as a single change because serverless is the strategic default.

Follow-up they will ask: What would make you move a job back after it migrated cleanly?

Say this first: No — a serverless warehouse still bills while it is running, and the idle timeout is what ends the meter when queries stop arriving.

Why: Serverless removes provisioning time, not the meter. A serverless SQL warehouse stays up for its configured idle period after the last query so the next one is instant, and those minutes are billed at the serverless rate. Set the timeout too high and a dashboard that polls every few minutes keeps the warehouse alive all day, which is the classic idle bill with a faster start. Because startup is seconds rather than minutes, you can afford a much shorter timeout than you would on classic — the cost of a cold start is now small. The thing to watch for is an auto-refreshing dashboard or a monitoring query on a short interval, which quietly converts an on-demand warehouse into an always-on one.

SELECT warehouse_id, COUNT(*) AS queries, MAX(end_time) AS last_query
FROM system.query.history
WHERE start_time >= current_date() - INTERVAL 1 DAYS
GROUP BY warehouse_id

Senior depth: The setting that was correct on classic is usually wrong on serverless, and nobody revisits it during migration because the warehouse still works. I would shorten the timeout aggressively, then measure: billed uptime versus query time per warehouse, and p50 query latency to confirm cold starts are not hurting anyone. The failure I actually look for is the automated poller — a dashboard refresh or a synthetic health check on a tight interval — because that single consumer can make every timeout setting irrelevant and it never shows up in a conversation about user behaviour.

Common mistake: Leaving a long idle timeout from the classic configuration because startup used to be expensive.

Follow-up they will ask: How short would you set it, and how would you tell if you went too far?

Say this first: Standardise network connectivity, identity, budget policies and tagging centrally; leave warehouse sizing and per-job compute choice to the teams who see the latency.

Why: The things that must be central are the ones where a wrong local decision is invisible to the person making it: the network connectivity configuration and private endpoints, which identities may create serverless resources, and the budget policies and tags that make serverless spend attributable. Serverless usage lands on shared SKUs, so without a tagging standard you get a large unattributable line and no way to have a cost conversation with anyone. What should stay local is sizing and workload placement, because the team running the job is the one feeling the latency and the one who can tell whether the premium is buying anything. Give them the cost-per-run data and let them choose. Then review quarterly: which teams' serverless spend grew, and did their workload grow with it.

SELECT custom_tags['team'] AS team, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE sku_name ILIKE '%SERVERLESS%'
  AND usage_start_time >= date_trunc('MONTH', current_date())
GROUP BY ALL ORDER BY dbus DESC

Senior depth: The architectural risk with serverless is not technical, it is that the friction which used to constrain spend is gone. On classic, creating a big cluster was visible and slow; on serverless anyone can summon capacity in seconds. So the controls have to move from scarcity to accountability: tags and budget policies attached at creation, a small set of blessed configurations, and monthly spend per team that someone actually reads. I would rather have fewer technical restrictions and a working attribution chain than a thicket of policies that teams route around.

Common mistake: Enabling serverless broadly without budget policies or tags, then being unable to tell any team what they spent.

Follow-up they will ask: What does an untagged serverless charge tell you about your governance?