Question 1

A classic cluster takes four minutes to start and a serverless warehouse is ready in seconds. What is physically different?

Accepted Answer

Classic compute provisions new VMs in your own cloud account on demand; serverless draws machines from a pool Databricks already has running. When you start a classic cluster, the control plane asks your cloud provider for instances in your VPC, those VMs boot, pull the runtime image, and the driver and executors register with each other. All of that is real provisioning time, and the cluster event log narrates each step. Serverless compute skips provisioning entirely: Databricks keeps a warm pool of capacity in its own account and attaches some of it to your workload, so startup is seconds. Your data has not moved — it is still in your storage account, read through Unity Catalog credentials. What moved is where the compute runs and who pays for keeping it warm, which is reflected in a higher DBU rate. Classic boots new VMs in your cloud account; that is the minutes. Serverless attaches capacity from a Databricks-managed warm pool. Data stays in your storage; only the compute location changed. You pay for the warm pool through a higher DBU rate.

Question 2

Serverless has a higher DBU rate than classic. Give me a workload where it is still cheaper and one where it is not.

Accepted Answer

Bursty work wins because you stop paying the moment it ends; a cluster pinned busy all day loses, because you pay the premium rate for every hour anyway. Cost is rate multiplied by time, and serverless changes both terms. The rate goes up, but the billed time collapses to actual work because there is no provisioning window and no idle cluster waiting for the next query. An analyst warehouse serving sporadic dashboard hits spends most of the day idle on classic, and that idle is the bill; on serverless it is near zero. A long-running nightly ETL that saturates a cluster for six straight hours has almost no idle to recover, so the higher rate is close to pure markup. The honest test is to pull the actual utilisation from system.billing.usage and the job run timeline rather than arguing from the price list. Serverless raises the rate and cuts the billed idle time. Bursty and intermittent workloads recover the premium easily. A saturated all-day job has no idle to recover, so it costs more. Decide from measured utilisation, not from the DBU price alone.

Question 3

A job that installs a custom wheel and runs an init script fails when you switch it to serverless. What is going on?

Accepted Answer

Serverless trades configurability for speed: cluster-scoped init scripts, arbitrary instance types, and some library installation paths are constrained. The warm pool only works because the machines are standardised, so the hooks that let you reshape a classic cluster are not all available. Cluster-scoped init scripts, choosing a specific instance family, attaching an instance profile, and installing anything that needs root are the usual casualties. Python dependencies are expected to come through supported paths — environment specifications and workspace or Unity Catalog volume artefacts — rather than a script that apt-installs a native library at boot. The practical move is to read what the init script actually does: most of them set an environment variable, install a package, or configure logging, and each of those has a supported equivalent. If the script genuinely needs root or a native system library, that workload stays on classic. Standardised machines are why startup is fast; that costs configurability. Cluster-scoped init scripts and instance type choice are constrained. Dependencies come through supported environment and artefact paths. Work that needs root or native system packages stays on classic.

Question 4

A serverless job needs to call a REST API that only accepts traffic from your VPC. On classic it worked because the cluster was already inside. What now?

Accepted Answer

Serverless egress leaves a Databricks-managed network, so you configure network connectivity and private endpoints rather than relying on being inside the VPC. The implicit assumption in the classic setup was that the executor's source address was one your firewall already trusted. Serverless compute runs outside your VPC, so that assumption disappears and the connection is refused or times out with nothing useful in the Spark logs. The replacement is explicit: a network connectivity configuration attached to the workspace, with private endpoints to the resources serverless needs to reach, and firewall rules written against that path instead of against your subnet CIDRs. For public endpoints you may also need to allow the documented egress addresses on the other side. The reason this surprises teams is that the failure looks like an application bug — a timeout in a notebook cell — rather than the networking change it actually is. Serverless egress does not originate from your VPC subnets. A network connectivity configuration plus private endpoints is the replacement. Firewall rules written against subnet CIDRs stop matching. The symptom is a connection timeout, which reads like an app bug.

Question 5

A team migrated a job to serverless and the bill went up, not down. Walk me through diagnosing that instead of just reverting.

Accepted Answer

Compare DBUs per run rather than per hour: if the job already saturated its cluster with little idle, you paid the premium rate for the same work. Pull system.billing.usage filtered to the job for a window before and after the change and normalise by run count, so you are comparing cost per execution rather than cost per day. Then put the job run timeline next to it: if classic runs show the cluster busy end to end with a short start-up tail, there was never much idle to recover and the higher rate goes straight through to the bill. The other common cause is that serverless removed the startup wait, so the job now finishes faster and somebody increased its frequency, which is a real behavioural change hiding inside a cost regression. Check whether concurrency changed too, since serverless scales out more readily and a job that used to queue may now run several tasks in parallel at full rate. Decide from that evidence: revert, resize, or keep it because the latency improvement is worth the money. Normalise by cost per run, not cost per day. A saturated classic cluster had little idle to recover. Faster runs often lead to more frequent runs, which is a real cost change. Easier scale-out can raise concurrency and therefore spend.

Question 6

You have thirty jobs and twelve SQL warehouses. How do you decide which ones move to serverless first?

Accepted Answer

Rank by idle fraction and startup overhead: spiky warehouses and short frequent jobs move first, saturated long-running ETL moves last or never. Build the list from data rather than opinion. For warehouses, compare billed uptime against actual query time to get an idle fraction; anything mostly idle is a strong candidate because serverless deletes exactly that cost. For jobs, compare cluster startup time against total run duration in the run timeline; a five-minute job paying four minutes of boot is wasting most of its wall clock. Then subtract the blockers — init scripts, specific instance types, instance profiles, VPC-only network dependencies — because those decide feasibility regardless of the economics. Move in that order, measure cost per run before and after on each wave, and keep the classic definition until the new one has survived a full cycle. The jobs left at the bottom of the list are the ones where classic is genuinely the right answer. Idle fraction and startup-to-runtime ratio rank the candidates. Feasibility blockers are init scripts, instance types, and network placement. Migrate in waves and measure cost per run on each one. Some saturated long-running workloads should stay on classic.

Question 7

Someone says auto-stop does not matter any more because the warehouse is serverless. Are they right?

Accepted Answer

No — a serverless warehouse still bills while it is running, and the idle timeout is what ends the meter when queries stop arriving. Serverless removes provisioning time, not the meter. A serverless SQL warehouse stays up for its configured idle period after the last query so the next one is instant, and those minutes are billed at the serverless rate. Set the timeout too high and a dashboard that polls every few minutes keeps the warehouse alive all day, which is the classic idle bill with a faster start. Because startup is seconds rather than minutes, you can afford a much shorter timeout than you would on classic — the cost of a cold start is now small. The thing to watch for is an auto-refreshing dashboard or a monitoring query on a short interval, which quietly converts an on-demand warehouse into an always-on one. Serverless bills while running; the idle timeout ends the meter. Fast startup means you can set a much shorter timeout than on classic. A frequently polling dashboard keeps the warehouse alive all day. Check billed uptime against actual query time, not just the setting.

Question 8

You are setting serverless policy for the whole account. What do you standardise, and what do you deliberately leave to teams?

Accepted Answer

Standardise network connectivity, identity, budget policies and tagging centrally; leave warehouse sizing and per-job compute choice to the teams who see the latency. The things that must be central are the ones where a wrong local decision is invisible to the person making it: the network connectivity configuration and private endpoints, which identities may create serverless resources, and the budget policies and tags that make serverless spend attributable. Serverless usage lands on shared SKUs, so without a tagging standard you get a large unattributable line and no way to have a cost conversation with anyone. What should stay local is sizing and workload placement, because the team running the job is the one feeling the latency and the one who can tell whether the premium is buying anything. Give them the cost-per-run data and let them choose. Then review quarterly: which teams' serverless spend grew, and did their workload grow with it. Centralise network connectivity, identity, budget policies, and tagging. Serverless SKUs are shared, so tags are the only attribution key. Leave sizing and compute choice to the team that feels the latency. Review spend against workload growth per team, not in aggregate.