Databricks interview questions: Asset Bundles: your workspace in git

Say this first: Git-backed notebooks version the code. The job definition, schedule, cluster spec, and permissions still live only in the workspace UI.

Why: A notebook in a repo tells you who changed the code. It does not tell you who changed the schedule, the cluster size, the parameters, or the run-as identity, because those are properties of the job, and the job was created by clicking. Two workspaces then drift: dev has a schedule somebody paused and prod has a cluster size somebody bumped, and there is no diff. A Databricks Asset Bundle puts the job, the pipeline, and the related resources in YAML next to the notebook, so a schedule change is a pull request. Until the job itself is in the repo, 'our code is in git' describes half the system.

Common mistake: Pointing at the notebook repo as evidence that production is reproducible.

Follow-up they will ask: Name three job settings that a notebook repo does not capture.

Say this first: CI runs databricks bundle validate on the pull request and databricks bundle deploy -t prod on merge, as a service principal.

Why: validate checks the YAML against the target without changing the workspace: bad references, missing variables, and permission problems fail here, on the pull request. deploy -t prod applies the bundle to the production workspace, creating or updating the jobs and pipelines the YAML describes and deleting bundle-managed resources that were removed from the file. run is for a developer exercising a job in dev, not for production. The identity that deploys matters as much as the command: a service principal owns the deployment, so production does not depend on the laptop or the employment of whoever last clicked. A human deploying from their laptop to prod is the failure mode the pipeline exists to remove.

databricks bundle validate -t prod
databricks bundle deploy -t prod
databricks bundle run -t dev daily_revenue

Common mistake: Deploying to production from a laptop with a personal token because the pipeline is not finished yet.

Follow-up they will ask: What does deploy do with a job that is in the workspace but no longer in the YAML?

Say this first: Targets. Each target supplies its own workspace host, mode, and variable values, and the resources reference those variables.

Why: A target is an environment: a workspace URL, a mode, and a set of variable values. The job YAML says ${var.catalog} and ${var.warehouse_id} instead of a hardcoded name, and the dev target resolves them to the dev catalog while prod resolves them to prod. mode: development on the dev target prefixes resource names with the deploying user and pauses schedules, so two engineers deploying the same bundle do not overwrite each other's jobs or send real emails. Prod has no such prefix, a real schedule, and a run-as service principal. ${workspace.current_user.userName} and ${resources.jobs.x.id} cover the substitutions that should follow the workspace rather than a variable you set.

targets:
  dev:
    mode: development
    workspace:
      host: https://dev.cloud.databricks.com
    variables:
      catalog: dev_sales
  prod:
    workspace:
      host: https://prod.cloud.databricks.com
    variables:
      catalog: prod_sales

Common mistake: Copying the YAML per environment and editing names by hand, which is the drift the bundle was meant to end.

Follow-up they will ask: What does mode: development change besides the name prefix?

Say this first: The job is bundle-managed, and the next deploy overwrites the workspace with whatever is in git.

Why: Deploy treats the YAML as the desired state. A change made in the UI is not in git, so the following deploy writes the git version back over it. That includes the schedule, the cluster, the task list, and parameters. The bundle can also remove resources that disappeared from YAML, so a job deleted from the file disappears from the workspace on the next prod deploy. The rule is that a bundle-managed job is edited in the repo and only in the repo. An emergency UI edit has to be copied back into YAML in the same change, or it will not survive the night. This is the trade you accepted to get a reviewable definition.

Senior depth: Bundle-deployed resources carry deployment metadata, and the bundle's own state records what it owns. If the job is in the bundle's resource list, assume the next deploy will win. Check the job's source before an incident edit, not after.

Common mistake: Editing the production job in the UI and considering the incident finished.

Follow-up they will ask: How can you tell whether a given job is bundle-managed before you edit it?

Say this first: When that person leaves, their identity is disabled and every job running as them stops. Production should run as a service principal.

Why: The run-as identity is whose permissions the job uses. If it is a human, the job can read whatever they can read, which is usually too much, and it dies when their account is deactivated, which is a strange way to discover a bus factor. A service principal is an application identity with exactly the grants the job needs: USE CATALOG, USE SCHEMA, SELECT, and MODIFY on the tables it writes. The bundle sets run_as on the prod target so it cannot be forgotten per job. CI also deploys as a service principal, separate from the one that runs the jobs if you want a tighter audit trail. Personal access tokens on a laptop are neither of these and should not appear in the pipeline.

targets:
  prod:
    run_as:
      service_principal_name: 00000000-0000-0000-0000-000000000000

Senior depth: Prefer two. The deployer can change the workspace definition; the runtime principal can only touch data. One identity that can do both is a larger blast radius and a muddier audit log.

Common mistake: Using a shared human admin account as run-as so that 'someone always has access'.

Follow-up they will ask: Should the principal that deploys be the same one the jobs run as?

Say this first: mode: development prefixes every resource with the deploying user and pauses schedules, so the two deploys do not own the same objects.

Why: Without development mode, both engineers deploy a job named daily_revenue to the same workspace, and the second deploy updates or replaces the first engineer's job. With the mode set, each gets [user] daily_revenue, schedules are paused so dev does not fire overnight, and the deployment is marked as development. Prod must not use this mode, or production jobs ship with a username prefix and no schedule. The prefix also makes cleanup honest: databricks bundle destroy -t dev removes that user's deployment rather than the team's shared jobs. A shared dev workspace without this mode is how one engineer's experiment becomes another engineer's missing job.

Senior depth: Destroy removes the resources tracked by that bundle deployment in that target. It does not remove tables the job created, and it does not remove jobs that were never part of the bundle. In dev that is one user's prefixed resources. Pointed at prod, it removes production.

Common mistake: Turning development mode on for the prod target because it seemed safer.

Follow-up they will ask: What does bundle destroy remove, and what does it leave alone?

Say this first: Fix the YAML and rerun deploy. Deploy is desired-state, so a second successful run converges. Do not patch the remainder by hand in the UI.

Why: A failed deploy can leave the workspace partially updated, because resources are applied in order and a later one failed validation or permissions. The recovery is to correct the cause — usually a permission the service principal lacks, or a variable prod does not define — and deploy again. The second deploy updates what is left and leaves what already matches untouched. Hand-editing the failed job in the UI forks it from git, and the next green deploy will overwrite that edit or fight it. Read the deploy log for the specific resource that failed rather than rerunning blindly, and prefer validate against the prod target locally before pushing the fix.

Senior depth: On the service principal, and in code if you manage grants with the bundle or Terraform. Granting it to your own user fixes your laptop deploy and leaves CI broken.

Common mistake: Finishing the failed deploy by clicking the remaining changes into the workspace.

Follow-up they will ask: The failure was a missing permission. Where do you grant it so the next deploy works?

Say this first: The bundle owns everything whose drift would change production behaviour: jobs, pipelines, and their permissions. Tables of data do not belong in a deploy that can destroy them.

Why: Jobs, pipelines, job clusters, schedules, and the grants the runtime principal needs are definition, and definition belongs in reviewable YAML. A catalog full of terabytes does not: deploy and destroy semantics that are correct for a job are dangerous for a table, and data lifecycle is a different change process. Schemas are a judgement call — a bundle-managed schema is convenient and also something a bad destroy could drop, so many teams create schemas out of band and only reference them. The split to write down is: the bundle is allowed to create and delete the orchestration, and it is not allowed to be the thing that drops prod.gold. That boundary is what lets you give CI an aggressive deploy role without giving it the keys to the data.

Senior depth: If the answer is 'drop the gold catalog', the boundary is wrong. Move data lifecycle out of the bundle until the worst case is 'prod jobs disappear and can be redeployed from git'.

Common mistake: Managing production tables in the same bundle that manages jobs, because it is convenient to have one tool.

Follow-up they will ask: What is the worst thing a mistyped bundle destroy -t prod could do in your setup?