Five-hour rerun to redo a four-minute failure
Symptoms
- The nightly run fails on the vendor export near the end and starts over from the top
- Total pipeline time doubles on every failure
- Nobody can say which part of the pipeline was slow
Interactive Databricks interview questions on Workflows: a DAG of tasks, not one notebook. Practice with the matching lesson. A job is a DAG of tasks with real parallelism, per-task retries, and a repair run that skips what already passed.
Question 1 of 10
A teammate's production pipeline is one notebook with 40 cells on a schedule. Sell me on breaking it up.
Answer it out loud, then reveal. Play steps through like the simulators.
Case 1 of 4 · symptoms
Symptoms
Indexed as FAQ. Open any item if you prefer a list to Play.
Showing 10 of 10 questions.
A teammate's production pipeline is one notebook with 40 cells on a schedule. Sell me on breaking it up.
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: A failure in cell 37 re-runs all 40 cells, and you get no per-step retry, no parallelism, and no per-step timing.
Why: A Databricks Job is a DAG of tasks wired with depends_on, so independent steps fan out in parallel, each task carries its own retry policy and timeout, and the run timeline gives you duration per task. Inside one notebook Spark sees a single linear sequence, so recovery means rerunning the expensive ingest you already completed.
Common mistake: Calling the monolith 'simpler' because it is one file, while every failure costs a full rerun.
Follow-up they will ask: Where would you put the task boundaries in that notebook?
A six-task job failed on task 4; tasks 5 and 6 show Skipped and someone is about to hit Run now. What do you tell them?
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: Fix the cause and use Repair run — it re-executes the failed task and its skipped downstream branch while tasks 1 to 3 stay successful.
Why: Downstream tasks skip rather than fail when a dependency fails, so the run keeps an accurate record of what completed. Repair run reuses that state and restarts from the failure point, which is why task granularity matters: coarse tasks make repair redo work that was already fine.
Common mistake: Clicking Run now and paying for the first three tasks again — or double-writing output that is not idempotent.
Follow-up they will ask: How would you make that task safe to retry more than once?
The nightly job runs on the team's all-purpose cluster because it is already warm. Make the cost argument.
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: All-purpose DBUs are a more expensive SKU than jobs compute, and you also pay for the cluster's idle hours; a job cluster exists for the run and terminates.
Why: In Databricks pricing, Jobs Compute is a cheaper DBU rate than All-Purpose Compute for the same instance type, and a job cluster starts per run and shuts down at the end so uptime matches work. Shared interactive clusters also leak state — installed libraries, cached data, someone else's runaway cell — into production runs, which is how a green pipeline fails only on Mondays.
Senior depth: There are two separate costs: the DBU rate and uptime you did not need. I would move scheduled work to job clusters or serverless, add a cluster policy so nobody can schedule production onto all-purpose, and keep interactive clusters for humans. If per-run startup latency is the genuine objection, serverless job compute answers it without the idle bill — the warm shared cluster answers it by paying for the cluster all night.
Common mistake: Arguing only about startup convenience and never comparing the DBU rate or the idle hours on the bill.
Follow-up they will ask: When is serverless job compute the better answer than a job cluster?
Two runs of the same hourly job overlapped and the target table got duplicate rows. What is wrong beyond 'the job was slow'?
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: Concurrent runs were allowed against a non-idempotent write, so two runs processed overlapping input and both appended.
Why: A job's max concurrent runs setting decides whether a scheduled run starts while the previous one is still going; at 1 the new run is skipped or queued instead of racing. The durable fix is making the write idempotent — MERGE on a business key, or a replaceWhere overwrite bounded to the run's window — so an overlap or a retry converges instead of duplicating.
Senior depth: Overlap is the symptom; the non-idempotent append is the bug. I would set max concurrent runs to 1 with a timeout and an alert so a slow run becomes visible rather than doubled, pass the window explicitly through job parameters and task values instead of each task calling now(), and make the sink a MERGE or a bounded replaceWhere so re-running a window is a no-op. Then define the job in Databricks Asset Bundles so concurrency, retries, and schedule are code, not something each workspace gets clicked differently.
Common mistake: Just lengthening the schedule interval, which hides the race until one run is slow again.
Follow-up they will ask: How would you pass the window boundaries between tasks so each task knows exactly what it processes?
Task A computes the window to process. How does task B get that value without writing it to a table or calling now() again?
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: Task values — dbutils.jobs.taskValues.set in A, .get in B keyed by the producing task, or a {{tasks.<key>.values.<name>}} reference in a task parameter.
Remember these
Why: Task values are small JSON values scoped to a single job run, set by one task and read by any downstream task that names the producer. The producer calls dbutils.jobs.taskValues.set with a key and a value; the consumer calls get with the producing taskKey, the key, and a debugValue so the notebook still runs interactively outside a job. For tasks that are not notebooks — a SQL task or a Python wheel — the same value can be injected through a parameter using the {{tasks.<task_key>.values.<name>}} reference, which keeps the plumbing out of the code. Why this matters beyond convenience: if each task computes its own window from the current time, two tasks in one run can disagree, and a retry an hour later processes a different slice than the original attempt. Deriving the window once and passing it forward makes the run's input deterministic, which is the precondition for a retry or a repair being safe.
dbutils.jobs.taskValues.set(key='window_end', value='2026-10-24T00:00:00')
# downstream task
end = dbutils.jobs.taskValues.get(
taskKey='bronze_ingest', key='window_end', debugValue='')Common mistake: Having every task call current_timestamp() independently, so a slow run straddles a day boundary and the tasks process different windows.
Follow-up they will ask: Where would you compute the window if two separate jobs have to agree on it?
You wired four silver tasks to depend only on bronze, but the run timeline shows them starting one after another. Where do you look?
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: The dependency graph permits parallelism; the compute has to supply it — check the depends_on lists, then the cluster the tasks share and how it was sized.
Remember these
Why: Start with the graph itself, because the most common cause is an accidental chain: one silver task listing another silver task in depends_on makes them sequential no matter what you intended, and the job's task graph view shows that immediately. If the graph is genuinely a fan-out, look at compute. Tasks sharing a job cluster run as separate Spark applications' worth of work on finite cores, so if the first task's stages consume every slot the others are effectively queued behind it. If each task was instead given its own new cluster, the timeline shows long pending segments before short execution segments — that is provisioning time, not contention. Then check for a ceiling outside the job: a cluster policy, a pool with a maximum, or cloud instance capacity, all of which show up as tasks waiting with no error. The run timeline is the evidence either way, since it separates each task's pending segment from its running segment.
Common mistake: Concluding Databricks does not parallelize tasks, when the graph has a stray dependency or the shared cluster has no spare slots.
Follow-up they will ask: The tasks do overlap but each one is slower than when it ran alone. What now?
You want a task that posts to Slack only when something upstream failed. Yours never fires. Why?
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: The default run_if is ALL_SUCCESS, so when its parent fails the notifier is marked upstream-failed and skipped along with everything else downstream.
Remember these
Why: A dependent task's run_if condition decides when it is eligible to start given its parents' outcomes, and the default requires every parent to have succeeded. That is correct for a gold task — you do not want it running on stale silver — and wrong for a failure handler, which by definition only matters when a parent failed. AT_LEAST_ONE_FAILED fires the task when at least one parent failed; ALL_DONE fires it once the parents have finished regardless of outcome, which is the right choice for cleanup that must happen either way. Worth separating from this: plain alerting does not need a task at all, because jobs have built-in notifications and webhook destinations for failure and timeout. Use a run_if task when the response is an action — rolling back a staged table, releasing a lock, writing an incident record — rather than a message. One caveat to say out loud: a task that runs on failure must not assume its parent's outputs exist.
- task_key: rollback_staging
depends_on:
- task_key: gold_publish
run_if: AT_LEAST_ONE_FAILED
notebook_task:
notebook_path: /Repos/etl/rollbackSenior depth: run_if is how you express compensation in the DAG rather than in a try/except somebody forgets to maintain. I keep notification in the job's built-in alerting, because a task that posts to Slack can itself fail, and reserve run_if tasks for state-changing recovery — unlocking, rolling back a staging table, recording the incident. The design rule is that a handler must be safe with partial or missing upstream output, since that is the exact situation it runs in.
Common mistake: Wiring the notifier as a dependent of the task it watches and leaving run_if at its default, so the alert is skipped precisely when it is needed.
Follow-up they will ask: How does a repair run interact with a task whose run_if is AT_LEAST_ONE_FAILED?
A task appends to a gold table and occasionally fails partway through. Someone set max_retries to 3. What did they just build?
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: An automatic duplicator — a retry re-runs the whole task, so a partially completed append plus a retry writes some rows twice.
Remember these
Why: Delta gives each individual write atomicity, but a task is usually several writes, and a retry restarts the task from the beginning with no knowledge of what the previous attempt committed. So a failure after the second of three appends, followed by a retry, double-writes the first two. Retries are a reliability feature that assumes idempotency; adding them to a non-idempotent task converts an occasional failure into permanent silent duplication, which is harder to find than the original error. Fix the write first. A MERGE on the business key converges no matter how many times it runs. A bounded overwrite with replaceWhere scoped to the run's window replaces that window's rows instead of adding to them. For append-shaped writes, Delta's txnAppId and txnVersion write options let the table recognise and ignore a duplicate commit from the same logical transaction. Once the write is idempotent, retries do what people expected them to do all along.
(df.write.format('delta').mode('append')
.option('txnAppId', 'nightly_etl')
.option('txnVersion', run_id)
.saveAsTable('ops.gold.metrics'))Senior depth: Retries are a statement that the task is idempotent, so the review question for any max_retries change is 'what happens if this runs twice'. I would make the sink converge first — MERGE on a key, or replaceWhere bounded by a window passed in as a parameter so the retry processes the identical slice — and only then turn retries on. For multi-table tasks the honest options are to split them so each task owns one convergent write, or to accept a reconciliation check as the safety net and say so out loud.
Common mistake: Adding retries to make a flaky task green, and finding the duplicates weeks later in a reconciliation.
Follow-up they will ask: Which of those three idempotency techniques fits a task that writes to four tables?
Jobs are configured by hand in three workspaces and staging no longer matches production. How do you fix that?
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: Define the jobs as code in a Databricks Asset Bundle with per-target overrides, and deploy them instead of clicking them.
Remember these
Why: A bundle is a databricks.yml that declares resources — jobs, pipelines, clusters — plus targets such as dev, staging and prod that override whatever differs between environments. You validate it, deploy it with the CLI against a target, and the workspace's job definition becomes an artifact of a reviewed commit rather than a record of who last edited it in the UI. The drift this eliminates is the dangerous kind: schedule, max concurrent runs, retry counts, cluster spec, notebook path and parameters are precisely the settings that get adjusted during an incident and never copied back. It also gives each developer their own deployed copy under a dev target, which is a better isolation story than sharing one job. The migration is not a rewrite — you can start from an existing job's definition and bring it under the bundle. The point to make in an interview is that a retry policy or a concurrency limit then becomes a pull request, with a diff and a reviewer, instead of a checkbox nobody can audit.
resources:
jobs:
nightly_etl:
max_concurrent_runs: 1
schedule:
quartz_cron_expression: '0 0 2 * * ?'
timezone_id: UTCSenior depth: The failure mode is not YAML preference, it is that nobody can say what production is currently configured to do. Bundles fix that by making the deployed definition derive from a commit, so an incident edit becomes visible as drift rather than becoming the new truth. I would pair it with a rule that UI edits in production are allowed during an incident and must be followed by a PR that reconciles them, and I would deploy from CI so the workspace state and the repository cannot disagree for long.
Common mistake: Keeping the production job in the UI as the source of truth and using the bundle only for dev, which guarantees the two diverge again.
Follow-up they will ask: Someone fixes production by editing the job in the UI during an incident. What is your process for that?
One job has grown to 120 tasks owned by five teams. What is your restructuring, and what do you lose?
Workflows: a DAG of tasks, not one notebook · tap to open the answer
Say this first: Split along ownership and SLA boundaries into separate jobs, wire them with a run-job task or table-arrival triggers, and accept that you lose the single timeline and one-click repair.
Remember these
Why: A 120-task job has one owner for alerting, one schedule, one concurrency limit and one repair surface, which is the wrong shape when five teams have different SLAs and on-call rotations. The split follows ownership: each team gets a job for its own tasks, with its own retries, notifications and permissions, and the cross-team dependency is expressed explicitly with a run-job task so the parent waits for the child, or with a table-arrival trigger so the downstream job fires when its input is actually ready. What you give up is real. The single run timeline becomes several, so an end-to-end view needs stitching together. A repair no longer covers the whole graph — you repair each job, in order. And tasks that used to share one job cluster now live in different jobs, so you either accept more provisioning or move to serverless. The rule I would state is to keep tasks in one job when they share compute and fail as a unit, and split when they have different owners, different SLAs, or different schedules.
- task_key: run_gold_job
run_job_task:
job_id: 418273
depends_on:
- task_key: silver_publishSenior depth: The boundary I draw is the on-call boundary: whoever gets paged for a task should own the job it lives in. Cross-job dependencies then get expressed through the data rather than through orchestration wherever possible — a consumer triggered by its input table being updated is more robust than a chain of run-job tasks, because the producer can be rescheduled without touching the consumer. The cost I would flag up front is observability: end-to-end latency now has to be measured from table freshness rather than read off one run timeline, so that measurement has to be built before the split, not after.
Common mistake: Splitting by technical layer so every job spans all five teams, which multiplies the coordination instead of reducing it.
Follow-up they will ask: How would you keep an end-to-end SLA visible once the pipeline spans four jobs?