Question 1

The team says they have CI/CD because the notebooks are in Git. A job's schedule changed last week and nobody can say who did it. What is missing?

Accepted Answer

Git-backed notebooks version the code. The job definition, schedule, cluster spec, and permissions still live only in the workspace UI. A notebook in a repo tells you who changed the code. It does not tell you who changed the schedule, the cluster size, the parameters, or the run-as identity, because those are properties of the job, and the job was created by clicking. Two workspaces then drift: dev has a schedule somebody paused and prod has a cluster size somebody bumped, and there is no diff. A Databricks Asset Bundle puts the job, the pipeline, and the related resources in YAML next to the notebook, so a schedule change is a pull request. Until the job itself is in the repo, 'our code is in git' describes half the system. Git-backed notebooks version code, not the job around the code. Schedule, cluster, parameters, and run-as live on the job. A bundle is databricks.yml plus resource YAML in the same repo. A schedule change becomes a pull request instead of a click.

Question 2

Walk through what happens from a merged pull request to a job existing in production.

Accepted Answer

CI runs databricks bundle validate on the pull request and databricks bundle deploy -t prod on merge, as a service principal. validate checks the YAML against the target without changing the workspace: bad references, missing variables, and permission problems fail here, on the pull request. deploy -t prod applies the bundle to the production workspace, creating or updating the jobs and pipelines the YAML describes and deleting bundle-managed resources that were removed from the file. run is for a developer exercising a job in dev, not for production. The identity that deploys matters as much as the command: a service principal owns the deployment, so production does not depend on the laptop or the employment of whoever last clicked. A human deploying from their laptop to prod is the failure mode the pipeline exists to remove. validate runs on the pull request and changes nothing. deploy -t prod applies the YAML to the production workspace. Resources removed from YAML are removed from the workspace. The deploy identity should be a service principal.

Question 3

Dev and prod need different warehouses, different catalogs, and different schedules. How does one YAML describe both?

Accepted Answer

Targets. Each target supplies its own workspace host, mode, and variable values, and the resources reference those variables. A target is an environment: a workspace URL, a mode, and a set of variable values. The job YAML says ${var.catalog} and ${var.warehouse_id} instead of a hardcoded name, and the dev target resolves them to the dev catalog while prod resolves them to prod. mode: development on the dev target prefixes resource names with the deploying user and pauses schedules, so two engineers deploying the same bundle do not overwrite each other's jobs or send real emails. Prod has no such prefix, a real schedule, and a run-as service principal. ${workspace.current_user.userName} and ${resources.jobs.x.id} cover the substitutions that should follow the workspace rather than a variable you set. A target binds a workspace, a mode, and variable values. Resources reference ${var.catalog} rather than a fixed name. mode: development prefixes names and pauses schedules. Prod runs as a service principal on the real schedule.

Question 4

You fix a failing job in the prod UI, save it, and the next morning your fix is gone. What happened?

Accepted Answer

The job is bundle-managed, and the next deploy overwrites the workspace with whatever is in git. Deploy treats the YAML as the desired state. A change made in the UI is not in git, so the following deploy writes the git version back over it. That includes the schedule, the cluster, the task list, and parameters. The bundle can also remove resources that disappeared from YAML, so a job deleted from the file disappears from the workspace on the next prod deploy. The rule is that a bundle-managed job is edited in the repo and only in the repo. An emergency UI edit has to be copied back into YAML in the same change, or it will not survive the night. This is the trade you accepted to get a reviewable definition. Deploy makes the workspace match git, not the other way around. A UI edit to a bundle-managed job is overwritten on the next deploy. A job removed from YAML is deleted from the workspace. An emergency UI fix has to be copied back into the repo immediately.

Question 5

Production jobs run as the engineer who created them. Why is that a bigger problem than it sounds, and what replaces it?

Accepted Answer

When that person leaves, their identity is disabled and every job running as them stops. Production should run as a service principal. The run-as identity is whose permissions the job uses. If it is a human, the job can read whatever they can read, which is usually too much, and it dies when their account is deactivated, which is a strange way to discover a bus factor. A service principal is an application identity with exactly the grants the job needs: USE CATALOG, USE SCHEMA, SELECT, and MODIFY on the tables it writes. The bundle sets run_as on the prod target so it cannot be forgotten per job. CI also deploys as a service principal, separate from the one that runs the jobs if you want a tighter audit trail. Personal access tokens on a laptop are neither of these and should not appear in the pipeline. Run-as is the identity whose permissions the job executes with. A human run-as dies when that account is disabled. A service principal gets only the grants that job needs. The bundle sets run_as on the prod target so it is not per-job folklore.

Question 6

Two engineers deploy the same bundle to the shared dev workspace and each one's jobs disappear. What setting prevents that?

Accepted Answer

mode: development prefixes every resource with the deploying user and pauses schedules, so the two deploys do not own the same objects. Without development mode, both engineers deploy a job named daily_revenue to the same workspace, and the second deploy updates or replaces the first engineer's job. With the mode set, each gets [user] daily_revenue, schedules are paused so dev does not fire overnight, and the deployment is marked as development. Prod must not use this mode, or production jobs ship with a username prefix and no schedule. The prefix also makes cleanup honest: databricks bundle destroy -t dev removes that user's deployment rather than the team's shared jobs. A shared dev workspace without this mode is how one engineer's experiment becomes another engineer's missing job. development mode prefixes resources with the deploying user. It pauses schedules so dev deployments do not run overnight. Two engineers can deploy the same bundle without colliding. Prod must not use development mode.

Question 7

A deploy to prod failed halfway. Some jobs updated, one did not, and the pipeline is red. What do you do, and what do you not do?

Accepted Answer

Fix the YAML and rerun deploy. Deploy is desired-state, so a second successful run converges. Do not patch the remainder by hand in the UI. A failed deploy can leave the workspace partially updated, because resources are applied in order and a later one failed validation or permissions. The recovery is to correct the cause — usually a permission the service principal lacks, or a variable prod does not define — and deploy again. The second deploy updates what is left and leaves what already matches untouched. Hand-editing the failed job in the UI forks it from git, and the next green deploy will overwrite that edit or fight it. Read the deploy log for the specific resource that failed rather than rerunning blindly, and prefer validate against the prod target locally before pushing the fix. A failed deploy can leave some resources updated and some not. A corrected second deploy converges to the YAML. A UI patch during the incident will be overwritten. validate against the prod target before rerunning.

Question 8

How do you decide what belongs in the bundle and what stays managed elsewhere?

Accepted Answer

The bundle owns everything whose drift would change production behaviour: jobs, pipelines, and their permissions. Tables of data do not belong in a deploy that can destroy them. Jobs, pipelines, job clusters, schedules, and the grants the runtime principal needs are definition, and definition belongs in reviewable YAML. A catalog full of terabytes does not: deploy and destroy semantics that are correct for a job are dangerous for a table, and data lifecycle is a different change process. Schemas are a judgement call — a bundle-managed schema is convenient and also something a bad destroy could drop, so many teams create schemas out of band and only reference them. The split to write down is: the bundle is allowed to create and delete the orchestration, and it is not allowed to be the thing that drops prod.gold. That boundary is what lets you give CI an aggressive deploy role without giving it the keys to the data. Jobs, pipelines, schedules, and runtime grants belong in the bundle. Tables of data do not belong to a tool with destroy semantics. Schemas are a judgement call because a destroy could drop them. CI should be able to redeploy orchestration without being able to drop data.