Interview/Databricks

Databricks interview questions: Compute types

Interactive Databricks interview questions on Compute types. Practice with the matching lesson. All-purpose clusters for notebooks, job clusters that terminate, SQL warehouses for BI. Mixing them is how the bill explodes.

Lesson · Simulation

All-purpose cluster vs jobs cluster vs SQL warehouse — pick one sentence each.

Answer it out loud, then reveal. Play steps through like the simulators.

Production scenario

Compute types

Overnight 32-worker all-purpose

Symptoms

  • No Workflows ran
  • One notebook detached
  • Bill 8×

All questions on this page

Indexed as FAQ. Open any item if you prefer a list to Play.

Showing 9 of 9 questions.

beginner

All-purpose cluster vs jobs cluster vs SQL warehouse — pick one sentence each.

Compute types · tap to open the answer

Say this first: All-purpose: interactive notebooks, stays on. Jobs: start, run, terminate. Warehouse: BI/Photon, not notebooks.

Why: Wrong compute is the expensive mistake that never shows up as a Spark bug. ETL on all-purpose burns idle hours. Tableau on an all-purpose cluster fights notebooks for slots.

Common mistake: One mega cluster for explore, ETL, and BI.

Follow-up they will ask: Which product should a nightly MERGE use?

Lesson · Simulation

intermediate

Why is an all-purpose cluster a bad host for production ETL?

Compute types · tap to open the answer

Say this first: It stays on, keeps leftover libraries/cache, and you pay for idle.

Why: A forgotten notebook pins a 32-worker cluster overnight. Job clusters get a clean Spark version and die when the task ends. Cluster policies should block oversized all-purpose.

Common mistake: Scheduling Workflows on the team's shared interactive cluster 'so it's warm'.

Follow-up they will ask: What still belongs on all-purpose?

Lesson · Simulation

senior

How do you read a Databricks bill that exploded with 'no new jobs'?

Compute types · tap to open the answer

Say this first: Look at all-purpose uptime, DBU SKU (Photon), and warehouses left in running.

Why: System tables / account console: idle clusters, SQL warehouses without auto-stop, Photon on workloads that don't benefit. Spark UI will look fine.

Senior depth: Auto-terminate all-purpose, job clusters only for Workflows, warehouses auto-stop, Photon where the plan is native.

Common mistake: Tuning spark.sql.shuffle.partitions to cut DBUs.

Follow-up they will ask: What policy would you add first?

Lesson · Simulation

beginner

A cluster set to autoscale 2–20 workers sat at 20 for an hour after the big job finished. Why didn't it shrink?

Compute types · tap to open the answer

Say this first: Scale-down needs sustained idleness and nodes it can safely release; running tasks, cached data, and attached sessions keep workers alive.

Remember these

  • Scale-down requires sustained idleness, not just a finished stage
  • Cached data and in-flight tasks pin workers in place
  • Cluster event log records every upscale and downscale with a reason
  • An all-purpose cluster that never shrinks bills worker-hours all day

Why: Autoscaling reacts to pending work and to idleness over a period, not to stage boundaries, so it will not track each stage's parallelism like a dial. A worker holding a cached dataset or shuffle output another stage still needs is not a candidate for removal, and on an all-purpose cluster one notebook left attached with cached DataFrames can hold the cluster wide open. The cluster event log records the upscale and downscale decisions with a reason, which is where you check rather than guessing. The bill is worker-hours, so an all-purpose cluster that never shrinks back is the surprise invoice nobody can attribute to a job. Clearing the cache, detaching the notebook, or setting a short auto-termination are the levers.

Common mistake: Expecting autoscaling to follow each stage's parallelism up and down like a throttle.

Follow-up they will ask: Why is autoscaling a weaker fit for a short job cluster than for a long interactive one?

Lesson · Simulation

intermediate

Finance sees two line items for the same cluster: DBUs on the Databricks invoice and instance hours on the cloud bill. Which one does Photon change?

Compute types · tap to open the answer

Say this first: Photon raises DBU consumption per hour; the cloud VM cost is unchanged, so it only pays off if the runtime falls enough to shrink total hours.

Remember these

  • Two bills on classic compute: cloud VMs plus Databricks DBUs
  • All-purpose carries a higher DBU rate than jobs compute
  • Photon raises DBUs per hour, so it must cut runtime to pay off
  • Serverless folds the infrastructure cost into a single DBU rate

Why: Every classic workload costs twice: your cloud provider charges for the VMs, disks, and network running in your account, and Databricks charges DBUs at a rate set by the product and the features in use. All-Purpose Compute carries a higher DBU rate than Jobs Compute for the same instance, and enabling Photon increases DBU consumption again. So the Photon decision is arithmetic, not faith — a job must get enough faster to offset the higher rate. Serverless changes the shape of the bill entirely by folding the infrastructure into one Databricks rate, so the cloud VM line disappears for that workload. You settle the argument in system.billing.usage, grouped by SKU, next to the runtimes from the job run history.

SELECT sku_name, sum(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY sku_name
ORDER BY dbus DESC

Common mistake: Turning Photon on everywhere because faster must be cheaper, without measuring the runtime change.

Follow-up they will ask: How would you prove Photon paid for itself on a specific job?

Lesson · Simulation

intermediate

A job on spot workers fails every few runs with executors disappearing mid-stage. What is your sequence of fixes?

Compute types · tap to open the answer

Say this first: Pin the driver to on-demand, let workers run spot with on-demand fallback, and shorten the shuffle so re-computation after a reclaim is cheap.

Remember these

  • Spot reclaim removes the executor and its shuffle files
  • A spot driver means the whole run dies, not just a task
  • Cluster event log distinguishes a reclaim from an OOM
  • Long stages lose more work per reclaim than short ones

Why: A reclaimed spot instance takes its executor and the shuffle files that executor produced, so Spark has to recompute the lost tasks and may hit fetch failures in the downstream stage. Losing the driver is unrecoverable: the whole run dies, which is why the driver should never be spot for anything with an SLA. The cluster event log names the loss explicitly as a spot termination, which distinguishes it from an OOM that people usually blame first. The deeper exposure is stage length, because the amount of work you lose scales with how much a stage had already done — a long single shuffle is the worst shape for spot. Checkpointing, splitting the job into tasks, or running the SLA-critical step on on-demand are the structural answers.

Common mistake: Diagnosing an OOM and raising executor memory when the event log says the instance was reclaimed.

Follow-up they will ask: Which single node should never be spot, and why?

Lesson · Simulation

advanced

On a shared cluster a user's RDD code and a direct path read both fail, but the same notebook works on their personal cluster. What does access mode have to do with it?

Compute types · tap to open the answer

Say this first: Standard (shared) access mode enforces Unity Catalog isolation between users, so ungovernable operations like arbitrary RDD and JVM access are blocked by design.

Remember these

  • Standard access mode sandboxes user code so reads stay attributable
  • Dedicated access mode binds compute to one principal and allows the full API
  • No-isolation shared cannot attribute reads, so Unity Catalog access is off
  • A mode error looks like a permission error but no grant will fix it

Why: Unity Catalog has to attribute every read to a principal, which is impossible if user code can reach the JVM, the filesystem, or RDDs directly, so a multi-user cluster runs that code in an isolated sandbox and permits only the governed surface: Spark SQL, the DataFrame API, Unity Catalog tables, and volumes. Dedicated (single-user) access mode binds the cluster to one principal, so the full API surface is available and is still attributable — that is the mode for ML libraries and Scala jars that need it. The legacy no-isolation mode has no per-user attribution at all, which is why it cannot be used for Unity Catalog table access. This is the single most common cause of 'it works for me but not on the shared cluster', and re-granting privileges never fixes it because the privilege was never the problem.

Senior depth: I treat access mode as part of the governance design rather than a cluster checkbox. Standard is the default for anything multi-user, because that is the only mode where one analyst cannot read another's credentials or escape to the filesystem. Dedicated exists for the workloads that genuinely need the full API, and I give those a narrow service principal rather than a human owner. Then I pin the mode in cluster policies, because if people can create no-isolation compute the entire audit story has a hole in it.

Common mistake: Granting more privileges to chase an access-mode restriction.

Follow-up they will ask: Which access mode would you require for production jobs that must show up in lineage?

Lesson · Simulation

advanced

Twelve small jobs each spend three minutes provisioning VMs, and serverless is not available to you. What do you do?

Compute types · tap to open the answer

Say this first: Back them with an instance pool holding idle instances, so cluster creation takes pre-warmed machines instead of asking the cloud provider.

Remember these

  • Pools hold idle VMs so cluster creation skips cloud provisioning
  • Preloading the runtime version removes image download from startup
  • Idle pool instances cost cloud money but not DBUs
  • Multiple instance types in a pool reduce capacity failures

Why: An instance pool keeps a set of idle VMs held in your account; a cluster that references the pool takes its driver and workers from those, which removes cloud provisioning from the start path and leaves only runtime initialisation. Preloading the Databricks Runtime version on the pool removes the image download too, so starts drop to a fraction of what they were. The cost model matters in the interview: Databricks does not charge DBUs for idle pool instances, but your cloud provider still charges for the VMs, so pool size and idle timeout are a direct trade of money for latency. Pools also soften capacity errors, because you can list several instance types for the pool to draw from. The job run timeline is the proof — the pending segment in front of the first Spark stage should nearly vanish.

Senior depth: The question behind the question is what you are willing to pay for latency. A pool converts a per-run wait into a continuous VM cost, so I size it from the concurrency of the schedule, not from the biggest job, and set a real idle timeout so the pool deflates overnight. I measure it the same way either side: pending seconds per run from the job timeline against VM hours from the cloud bill. If serverless becomes available for job compute, it answers the same problem without me running a pool at all, which is usually the better end state.

Common mistake: Keeping a warm all-purpose cluster instead, which bills both DBUs and VMs for every idle hour.

Follow-up they will ask: Who pays for an idle pool instance, and how do you size the pool?

Lesson · Simulation

architect

You own the platform and every team sizes its own compute. Write the guardrail.

Compute types · tap to open the answer

Say this first: Cluster policies: fix the access mode, cap workers and node types, force auto-termination, pin the runtime, and require tags so spend is attributable.

Remember these

  • Policies can fix, default, or range-restrict any compute setting
  • Auto-termination and max workers are the two rules that cut the bill
  • Required custom tags make DBUs attributable per team in system tables
  • Separate policies for all-purpose and jobs compute

Why: A cluster policy is a set of rules applied at compute creation that can fix a value, supply a default, or restrict a range, and policies are defined per compute type so all-purpose and jobs get different ones. The rules that pay for themselves immediately are autotermination_minutes, a maximum worker count, an allowed node-type list, the access mode, and custom tags. Tags are the one people skip and the one that ends the argument, because tagged DBUs land in system.billing.usage and you can show a team its own spend instead of debating the workspace total. Pinning the runtime version keeps jobs off an unsupported DBR and keeps behaviour comparable across teams. The last decision is political rather than technical: whether anyone is allowed to schedule production work on all-purpose compute, and the answer should be no.

SELECT custom_tags.team, sum(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - INTERVAL 7 DAYS
GROUP BY 1
ORDER BY dbus DESC

Senior depth: Guardrails only work if the feedback loop is short, so I pair policies with attribution. Policies stop the obviously wrong compute from existing — no untagged clusters, no missing auto-termination, no production on all-purpose — and tagged usage in system tables gives every team a weekly number they own. I would rather enforce four rules everyone understands than twenty that teams route around by asking for exceptions. Then the only remaining conversation is sizing, which is the one teams are actually qualified to have.

Common mistake: Writing a policy that caps worker count but leaves auto-termination as an optional field.

Follow-up they will ask: What would you do about teams that already have oversized clusters created before the policy?

Lesson · Simulation