Databricks CI/CD with Git Folders and Automation Bundles
Introduction
This lesson covers the Implementing CI/CD section of the official exam outline (about 10% of the scored questions). Databricks CI/CD combines Git folders and Declarative Automation Bundles to promote reviewed Lakeflow Jobs and pipeline code across environments. The Lakeflow Jobs task graph lesson provides the runtime workflow this lesson packages for deployment. That works for a prototype, but a production data platform needs the same discipline as any other software: code in version control, reviews before merging, automated checks, and repeatable deployments to development, test and production environments without anyone editing a job by hand.
Databricks gives you two building blocks for this. Git folders (formerly Repos) connect a workspace folder to a Git repository so that you can branch, commit, push and pull from the workspace. Declarative Automation Bundles (formerly Databricks Asset Bundles, often shortened to bundles) describe jobs, pipelines and other resources as YAML files that the Databricks CLI validates, deploys and runs. Together they let a team treat a Lakeflow Job like code.
After this lesson you will be able to explain how Git folders support a branch-and-pull-request workflow, describe the structure of a bundle and its databricks.yml file, use targets and variables to promote the same code from dev to prod, and choose the right CLI command at each stage of a CI/CD pipeline.
Learning objectives
- Explain what a Databricks Git folder is and which Git operations it supports in the workspace.
- Describe a branch, commit, push and pull request workflow for notebooks and pipeline code.
- Describe the structure of a Declarative Automation Bundle and the role of
databricks.yml. - Use targets, variables and overrides to configure dev, test and prod from one code base.
- Explain how a bundle deploys Lakeflow Jobs and Lakeflow Spark Declarative Pipelines.
- Use
databricks bundle validate,deploy,runanddestroyin a CI/CD workflow.
1. Why CI/CD matters for data engineering
Continuous integration (CI) means every change is merged into a shared branch often, and each merge triggers automatic checks such as linting, unit tests and a validation of the deployment configuration. Continuous delivery or deployment (CD) means a validated change is released to an environment by an automated process rather than by a person clicking in the UI.
Without CI/CD, typical failures look like this: someone edits a production job directly and nobody can tell what changed; a notebook works in a personal folder but fails in production because a parameter differs; two engineers overwrite each other's work in the same shared notebook. The goal of this section of the exam is to know the Databricks tools that remove these problems.
| Concern | Without CI/CD | With Git folders and bundles |
|---|---|---|
| Change history | Notebook revision history only, no review | Git commits, pull requests and reviews |
| Environments | Jobs copied and edited by hand | One definition, several targets |
| Deployment | Manual clicks | databricks bundle deploy in a pipeline |
| Rollback | Guess the previous settings | Redeploy a reviewed previous commit after checking state and data compatibility |
| Consistency | Differences between dev and prod | Same reviewed code; target-specific settings and permissions are explicit |
2. Databricks Git folders
A Git folder is a folder in the workspace that is linked to a remote Git repository (GitHub, GitLab, Azure DevOps, Bitbucket and others are supported). The workspace holds a clone of the repository. Git folders were called Repos in earlier documentation; the exam guide now uses the name Git folders, so recognize both.
What you can do from a Git folder in the workspace UI:
- Clone a remote repository into the workspace, using a Git credential (a personal access token or a linked account) stored for your user.
- Create and switch branches.
- Commit and push changes, and pull changes made by others. Before committing you can review the changed files and write a commit message.
- Resolve merge conflicts in simple cases, and discard or reset local changes.
What you do not do in the workspace: a pull request is created and reviewed on the Git provider's side (for example GitHub), not inside Databricks. The Git folder is your working copy; the provider is where review, approval and merge happen. After the merge, a pipeline or another engineer pulls the updated branch.
A recommended team workflow:
- Each engineer has a personal Git folder (in their user area) cloned from the shared repository.
- The engineer creates a feature branch, edits notebooks or Python files, runs them on a development cluster or serverless compute, and commits.
- The engineer pushes the branch and opens a pull request on the Git provider.
- Automated checks run on the pull request (see section 6). A teammate reviews it.
- After approval the branch is merged into
main. A CI/CD pipeline then deploys the merged code to the test and production workspaces.
An important rule: production should not run from a user's personal Git folder. A personal folder can sit on any branch with uncommitted changes. Production jobs should run a version-controlled, deployed copy: either a Git source configured on the job (a specific branch, tag or commit) or the files deployed by a bundle.
Concrete example. Contoso Retail has a repository retail-data-platform on GitHub. Amira clones it into her Git folder, creates the branch feature/fix-returns-join, repairs a join in the silver notebook, tests it on a small date range and pushes. A teammate reviews the pull request, a GitHub Action validates the bundle, and the merge to main triggers the deployment to the test workspace.
Git source on a job
A job task can also point to a remote Git repository instead of a workspace path. You configure the repository URL and a branch, tag or commit, and each run uses that version of the notebook. Pinning an immutable commit SHA fixes the source revision; a tag can be moved unless protected by the provider. pointing to a branch gives you "latest of that branch" behavior. All tasks using remote Git in a job run use the resolved commit for that run. This versions source, not data, dependencies or credentials. For bundle-managed jobs, prefer the workspace files deployed from the checked-out repository; mixing remote git_source with bundle-relative source paths can resolve different content.
3. Declarative Automation Bundles: the idea
A bundle is a project that contains:
- the source files: notebooks, Python files, SQL files, pipeline definitions, tests;
- the configuration (YAML) that declares the resources to create in a workspace: jobs, pipelines, and other supported objects such as dashboards, models or volumes;
- one or more targets: the environments where the resources are deployed.
It is declarative: you describe the desired state (a job with these tasks and this schedule) and the CLI makes the workspace match it. Because the description is plain text, it lives in Git, goes through pull requests, and can be validated in CI.
The product was named Databricks Asset Bundles until recently. The exam guide uses the new name, Declarative Automation Bundles, and the Databricks CLI commands keep the bundle word: databricks bundle ....
A bundle is created with databricks bundle init (from a template) and organized like this:
retail-data-platform/
databricks.yml # the root configuration file (exactly one)
resources/
ingest_job.yml # job definition
sales_pipeline.yml # pipeline definition
src/
ingest_orders.py
transform/
silver_orders.sql
sales_pipeline.sql
tests/
test_transformations.pyUse a current supported Databricks CLI; bundles require at least version 0.218.0, and newer resource fields may require a newer CLI schema. Pin a reviewed CLI version in real CI. These examples use illustrative workspace hosts, warehouse ids and service-principal ids: replace them before deployment. The target workspace must support serverless notebook jobs, and the SQL warehouse, Unity Catalog schemas/volumes, source tables and privileges must already exist.
The databricks.yml file
Every bundle has exactly one databricks.yml at the root. It declares the bundle name, includes other YAML files, defines variables and targets, and may declare resources directly.
bundle:
name: retail-data-platform
include:
- resources/*.yml
variables:
catalog:
description: Unity Catalog catalog for this environment
default: dev_retail
warehouse_id:
description: SQL warehouse used by SQL tasks
targets:
dev:
mode: development
default: true
workspace:
host: https://adb-1111111111111111.1.azuredatabricks.net
variables:
warehouse_id: "1111111111111111"
test:
workspace:
host: https://adb-2222222222222222.2.azuredatabricks.net
variables:
catalog: test_retail
warehouse_id: "2222222222222222"
prod:
mode: production
workspace:
host: https://adb-3333333333333333.3.azuredatabricks.net
root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}
variables:
catalog: prod_retail
warehouse_id: "3333333333333333"
git:
branch: main
run_as:
service_principal_name: 4f7c0d2e-1111-2222-3333-444455556666Read it from top to bottom:
bundle.namenames the project. Deployment identity is tied to the workspace/root path; its default includes the deploying user, bundle name and target. A stable shared production root prevents per-user deployment copies, but two bundles must not share the same state path.includepulls in other YAML files so resources stay in small readable files.variablesdeclares named values with optional defaults. They are referenced as${var.catalog}.targetslists the environments. Each target can set the workspace host, override variables, and change resource settings.
4. Targets, variables and overrides
A target is a named deployment environment such as dev, test or prod. The same resource definitions are deployed to each target, with only the differences declared as overrides.
Common uses of targets and variables:
| Need | Mechanism |
|---|---|
| Different workspace per environment | workspace.host in each target |
| Different catalog or schema per environment | A variable with a different value per target |
| Smaller cluster in dev, bigger in prod | Override a resource setting under targets.<name>.resources |
| Pause schedules in dev | Override trigger / schedule pause_status for the dev target |
| Deploy production under a service identity | run_as with a service principal |
Development and production modes
A target can set mode:
mode: developmentmarks deployed resources as development resources: names receive the default prefix[dev <user>](for example[dev amira] ingest_orders), schedules and triggers are paused by default, and default deployment paths separate users. Explicit resource settings or presets can override defaults; a shared root path can defeat that isolation. These defaults reduce accidental scheduled runs, provided they have not been overridden.mode: productionenables safety checks: for example, it validates an explicitly configured Git branch and applies production identity/path checks and does not add the development prefix. Production deployments should run as a service principal, not as a person.
Variables
Variables avoid copying values around. You declare them once, give a default, and override them per target or on the command line:
databricks bundle deploy -t prod --var="catalog=prod_retail"A variable is referenced in resource files as ${var.catalog}. There are also built-in references such as ${bundle.target} and ${workspace.current_user.userName}.
Variable precedence, highest first: command-line --var, BUNDLE_VAR_ environment variables, .databricks/bundle/<target>/variable-overrides.json, target variables, then the variable default. Keep the same variable inputs during validation, deployment and execution. Variables configure deployment; job run parameters carry per-run values. A bundle substitution is not automatically expanded inside arbitrary notebook source code.
Resource definitions
A job resource in resources/ingest_job.yml might look like this:
resources:
jobs:
ingest_orders_job:
name: ingest_orders_${bundle.target}
tasks:
- task_key: bronze_orders
notebook_task:
notebook_path: ../src/ingest_orders.py
base_parameters:
catalog: ${var.catalog}
- task_key: silver_orders
depends_on:
- task_key: bronze_orders
sql_task:
file:
path: ../src/transform/silver_orders.sql
warehouse_id: ${var.warehouse_id}
parameters:
catalog: ${var.catalog}
max_concurrent_runs: 1
trigger:
pause_status: PAUSED
file_arrival:
url: /Volumes/${var.catalog}/landing/orders/The notebook must be a real Databricks notebook (a .py source notebook starts with # Databricks notebook source) and explicitly read its catalog widget. The SQL file reads the named parameter :catalog; use IDENTIFIER when constructing a catalog-qualified table name rather than interpolating raw text. This sample keeps the ingestion trigger paused in every target until explicitly activated, and does not reference an undefined notebook environment.
A pipeline resource is similar:
resources:
pipelines:
sales_pipeline:
name: sales_pipeline_${bundle.target}
catalog: ${var.catalog}
schema: sales
libraries:
- file:
path: ../src/transform/sales_pipeline.sqlThe illustrated source paths are relative to their resource YAML file. Ensure every referenced file exists and use the intended target catalog/schema inside the source queries. The same files describe the Lakeflow Job (formerly Workflows) and the Lakeflow Spark Declarative Pipeline (formerly Delta Live Tables). When you deploy, the CLI uploads the source files to the workspace and creates or updates the job and the pipeline. Deployment state tracks resource ids by their resource keys; renaming a key can create a replacement and remove the old object, so inspect the planned changes first. Changing the visible name is different from changing the key. Deletion or redeployment does not undo data writes.
5. The Databricks CLI workflow
The Databricks CLI is the tool that runs bundle commands. A CI/CD pipeline uses a service principal credential and these commands:
| Command | What it does | When to use it |
|---|---|---|
databricks bundle init |
Creates a new bundle from a template | Starting a project |
databricks bundle validate |
Checks that the configuration is syntactically and structurally valid and returns configuration/identity information | On every pull request, before deploying |
databricks bundle deploy -t <target> |
Uploads files and creates or updates the declared resources in the workspace | Pipeline step after merge, or by a developer to dev |
databricks bundle run -t <target> <resource_key> |
Starts a deployed job or pipeline update and can follow the run | Smoke test after deploy, integration test, or manual run |
databricks bundle summary |
Shows deployed resources and links | Checking what is deployed |
databricks bundle destroy -t <target> |
Deletes the deployed resources and files of that target | Cleaning a temporary or feature environment |
If the bundle has a default target (default: true), you can omit -t for that target. For a non-default target, pass its name explicitly (for example -t prod).
Important behaviors for the exam:
validatedoes not change anything in the workspace.deploydoes.- Within the same deployment state/root path, repeated
deployreconciles the declared resource ids rather than creating copies. Changing the root path or state identity can create a separate deployment; do not delete or casually move deployment state. run <resource_key>starts the deployed job/pipeline. It does not deploy local configuration edits first; it fails if the resource has never been deployed. Avoid assuming the latest local/Git edit has been released.destroyremoves what the bundle deployed to that target, and it asks for confirmation unless you pass an auto-approve option. Use it carefully, never casually againstprod.- The CLI authenticates with the workspace through a profile, environment variables or OAuth. In CI use a service principal with OAuth machine-to-machine credentials stored as secrets in the CI system, or a supported workload identity federation configuration. Separate deployment authentication from
run_as, which controls execution permissions; neither automatically grants Unity Catalog or warehouse access. Review Unity Catalog privileges for execution identities.
A typical pipeline
# .github/workflows/deploy.yml (trusted pushes and same-repository PRs)
on:
pull_request:
push:
branches: [main]
jobs:
validate:
if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
env:
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}
steps:
- uses: actions/checkout@v4
- uses: databricks/setup-cli@main
- run: databricks bundle validate -t test
deploy-test:
if: github.ref == 'refs/heads/main'
needs: validate
runs-on: ubuntu-latest
environment: test
env:
DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}
steps:
- uses: actions/checkout@v4
- uses: databricks/setup-cli@main
- run: databricks bundle deploy -t test
- run: databricks bundle run -t test ingest_orders_jobBoth CI jobs require their own authentication; step-level environment variables do not carry over to another job. Configure secrets for the test workspace to match targets.test.workspace.host, and grant the service principal only the needed permissions. Fork PRs do not receive these deployment secrets; run unprivileged local checks for them and reserve authenticated checks for trusted reviewed code. databricks/setup-cli@main is shown for readability; pin setup actions and the CLI to reviewed immutable versions in a maintained workflow. Protect the production environment with its release approval policy.
The flow is: pull request -> validate (and unit tests) -> merge -> deploy to test -> run an integration test -> approval -> deploy to prod.
6. Testing in CI
A bundle fits naturally with automated tests:
- Unit tests for Python transformation functions (for example with
pytest) can run in the CI runner using a compatible local Spark session and Java/PySpark dependencies, or pure-Python helpers/mocks where appropriate. Unity Catalog,dbutilsand Databricks-only APIs require integration coverage rather than a claim that all notebook code runs locally. - Integration tests deploy to a test target and
runa job against test data, then assert on the results. - Validation (
bundle validate) checks configuration and variable resolution before deployment; it does not prove that SQL, source code, runtime dependencies, privileges or data logic will succeed. Treat unknown-field warnings seriously rather than assuming every typo is fatal.
Keep transformation logic in functions that take and return data frames. A function like clean_orders(df) is easy to test; logic buried in a notebook cell is not.
Worked example: promoting a pipeline from dev to prod
Scenario. A logistics company has a bundle with a job load_shipments that reads files from a volume and writes to shipments_silver. In dev the table must go to dev_logistics.silver, in production to prod_logistics.silver. The production job must run every hour as a service principal, while in dev nobody wants a schedule running. Today, an engineer changes the catalog name by hand before each release and once forgot to revert it.
Step 1: Remove hard-coded names. Set the notebook task parameter to ${var.catalog} and read it through a widget in the notebook, and declare catalog in databricks.yml with the default dev_logistics.
Step 2: Declare the targets. Add dev (mode development, default) and prod (mode production) targets. In prod set variables.catalog: prod_logistics and run_as to a service principal.
Step 3: Control the schedule. Define the hourly cron schedule in the job. Leave pause_status unspecified in dev so development-mode defaults pause it; do not explicitly set it to UNPAUSED in shared settings. Set schedule.pause_status: UNPAUSED under the production resource override when ready. Mode is a preset, not an unconditional safety barrier.
Step 4: Automate. In the CI system, run databricks bundle validate -t prod on pull requests, and after merge run databricks bundle deploy -t prod using the service principal credentials stored as secrets.
Result. The same code is deployed everywhere. Only the variable and the mode change, no one edits a catalog by hand, and the production job runs under a non-personal identity.
Why the alternatives are worse. Keeping two copies of the notebook ("dev" and "prod") causes drift. Editing the job in the production UI loses the history and breaks the next deployment. Running production from a personal Git folder ties it to one person's branch state and credentials.
Common pitfalls and exam tips
- Git folders = Repos. The old name may still appear in options. The pull request itself is created on the Git provider.
- Bundles = Databricks Asset Bundles. The configuration file is
databricks.yml; resources are declared in YAML. validatechanges nothing;deploydoes. Order: validate, deploy, run.-tselects the target. Omission chooses the configured default, which could even be production; make the production target explicit in release automation.- Variables, not copies. Differences between environments belong in variables and target overrides.
- Development mode prefixes names and pauses schedules; production mode is meant for service principals and stricter checks.
- Use a service principal for CI. Authenticate with OAuth M2M secrets or supported workload identity federation, and distinguish deployment permissions from run_as.
destroyis destructive. It removes the deployed resources of the chosen target.
Exam tip: When a question says "the same job must be deployed to dev, test and prod with different catalogs", the answer is a bundle with targets and variables, not three manually edited jobs and not three copies of the code. When it says "review changes before they reach production", the answer involves Git branches and pull requests on the Git provider.
Mini case study
Situation. A bank's data team has two engineers who edit the same shared notebook folder in the production workspace. Last week an unreviewed change broke the nightly job. The team wants (1) changes reviewed before production, (2) the same job in staging and production with different catalogs, (3) automatic checks, and (4) deployments that do not depend on any engineer's account.
Task. Propose a design using Databricks tools.
Solution.
- Move the code to a Git repository. Each engineer works in a personal Git folder on a feature branch and opens a pull request on the Git provider; merges to
mainrequire a review. - Convert the job and pipeline to a Declarative Automation Bundle with
stagingandprodtargets, and acatalogvariable overridden per target. - In the CI system, run
databricks bundle validateand unit tests on every pull request. - After merge, deploy to
stagingwithdatabricks bundle deploy -t staging, run an integration job withdatabricks bundle run, then deploy toprodafter approval. - Use a service principal with
run_asin the production target and for the CI credentials, so deployments and production runs do not depend on a person. - Remove write access of engineers to the production folder, so that production changes come only from the pipeline.
Key terms
- Git folder: workspace folder linked to a remote Git repository (formerly Repos).
- Pull request: review and merge request created on the Git provider.
- Declarative Automation Bundle: project of source files and YAML configuration deployed with the CLI (formerly Databricks Asset Bundles).
databricks.yml: root configuration file of a bundle.- Target: named deployment environment (dev, test, prod) with its own overrides.
- Variable: named value referenced as
${var.name}and overridable per target. - Resource: object declared in a bundle, such as a job or a pipeline.
- Development mode / production mode: target settings that change naming, schedules and safety checks.
- Service principal: non-human identity used for automation and production runs.
bundle validate/deploy/run/destroy: CLI commands to check, release, execute and remove a bundle.
Summary checklist
- I can describe the Git folder workflow: branch, commit, push, pull request on the provider.
- I know that production should not run from a personal Git folder.
- I can explain the role of
databricks.yml, resources, targets and variables. - I can explain development mode versus production mode.
- I can order the commands: validate, deploy, run, and say what destroy does.
- I know CI/CD should authenticate with a service principal.
- I can design a dev-to-prod promotion without copying code.
After deployment, use Lakeflow Jobs monitoring and safe repair to inspect the released task graph and its execution results.
Official sources
- Exam guide: https://www.databricks.com/sites/default/files/2026-10/Databricks-Certified-Data-Engineer-Associate-Exam-Guide-August-2026.pdf (opens in a new tab)
- Declarative Automation Bundles configuration: https://docs.databricks.com/aws/en/dev-tools/bundles/settings (opens in a new tab)
- Declarative Automation Bundles deployment modes: https://docs.databricks.com/aws/en/dev-tools/bundles/deployment-modes (opens in a new tab)
- Databricks Git folders: https://docs.databricks.com/aws/en/repos/git-operations-with-repos/ (opens in a new tab)
- Databricks CLI bundle commands: https://docs.databricks.com/aws/en/dev-tools/cli/bundle-commands (opens in a new tab)
- Bundle variables and precedence: https://docs.databricks.com/aws/en/dev-tools/bundles/variables (opens in a new tab)
- Bundle task definitions: https://docs.databricks.com/aws/en/dev-tools/bundles/job-task-types (opens in a new tab)
- Bundle execution identity: https://docs.databricks.com/aws/en/dev-tools/bundles/run-as (opens in a new tab)
- OAuth service-principal authentication: https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m (opens in a new tab)
- GitHub workload identity federation: https://docs.databricks.com/aws/en/dev-tools/auth/provider-github (opens in a new tab)
