Overnight 32-worker all-purpose
Symptoms
- No Workflows ran
- One notebook detached
- Bill 8×
Interactive Databricks interview questions on Compute types. Same topic as /learn/databricks/compute. All-purpose clusters for notebooks, job clusters that terminate, SQL warehouses for BI. Mixing them is how the bill explodes.
Question 1 of 3
All-purpose cluster vs jobs cluster vs SQL warehouse — pick one sentence each.
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 1 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
All-purpose cluster vs jobs cluster vs SQL warehouse — pick one sentence each.
Compute types · tap to open the answer
Short: All-purpose: interactive notebooks, stays on. Jobs: start, run, terminate. Warehouse: BI/Photon, not notebooks.
Detailed: Wrong compute is the expensive mistake that never shows up as a Spark bug. ETL on all-purpose burns idle hours. Tableau on an all-purpose cluster fights notebooks for slots.
Common mistake: One mega cluster for explore, ETL, and BI.
Follow-up: Which product should a nightly MERGE use?
Why is an all-purpose cluster a bad host for production ETL?
Compute types · tap to open the answer
Short: It stays on, keeps leftover libraries/cache, and you pay for idle.
Detailed: A forgotten notebook pins a 32-worker cluster overnight. Job clusters get a clean Spark version and die when the task ends. Cluster policies should block oversized all-purpose.
Common mistake: Scheduling Workflows on the team's shared interactive cluster 'so it's warm'.
Follow-up: What still belongs on all-purpose?
How do you read a Databricks bill that exploded with 'no new jobs'?
Compute types · tap to open the answer
Short: Look at all-purpose uptime, DBU SKU (Photon), and warehouses left in running.
Detailed: System tables / account console: idle clusters, SQL warehouses without auto-stop, Photon on workloads that don't benefit. Spark UI will look fine.
Senior: Auto-terminate all-purpose, job clusters only for Workflows, warehouses auto-stop, Photon where the plan is native.
Common mistake: Tuning spark.sql.shuffle.partitions to cut DBUs.
Follow-up: What policy would you add first?