Skip to content
FR

CI/CD with Git Folders and Declarative Automation Bundles

Step 21 of 30

CI/CD with Git Folders and Declarative Automation Bundles

Introduction

This lesson covers the Implementing CI/CD section of the official exam outline (about 10% of the scored questions). Until now you have built notebooks, pipelines and jobs, mostly by clicking in a workspace. That works for a prototype, but a production data platform needs the same discipline as any other software: code in version control, reviews before merging, automated checks, and repeatable deployments to development, test and production environments without anyone editing a job by hand.

Databricks gives you two building blocks for this. Git folders (formerly Repos) connect a workspace folder to a Git repository so that you can branch, commit, push and pull from the workspace. Declarative Automation Bundles (formerly Databricks Asset Bundles, often shortened to bundles) describe jobs, pipelines and other resources as YAML files that the Databricks CLI validates, deploys and runs. Together they let a team treat a Lakeflow Job like code.

After this lesson you will be able to explain how Git folders support a branch-and-pull-request workflow, describe the structure of a bundle and its databricks.yml file, use targets and variables to promote the same code from dev to prod, and choose the right CLI command at each stage of a CI/CD pipeline.

Learning objectives

  • Explain what a Databricks Git folder is and which Git operations it supports in the workspace.
  • Describe a branch, commit, push and pull request workflow for notebooks and pipeline code.
  • Describe the structure of a Declarative Automation Bundle and the role of databricks.yml.
  • Use targets, variables and overrides to configure dev, test and prod from one code base.
  • Explain how a bundle deploys Lakeflow Jobs and Lakeflow Spark Declarative Pipelines.
  • Use databricks bundle validate, deploy, run and destroy in a CI/CD workflow.

1. Why CI/CD matters for data engineering

Continuous integration (CI) means every change is merged into a shared branch often, and each merge triggers automatic checks such as linting, unit tests and a validation of the deployment configuration. Continuous delivery or deployment (CD) means a validated change is released to an environment by an automated process rather than by a person clicking in the UI.

Without CI/CD, typical failures look like this: someone edits a production job directly and nobody can tell what changed; a notebook works in a personal folder but fails in production because a parameter differs; two engineers overwrite each other's work in the same shared notebook. The goal of this section of the exam is to know the Databricks tools that remove these problems.

Concern Without CI/CD With Git folders and bundles
Change history Notebook revision history only, no review Git commits, pull requests and reviews
Environments Jobs copied and edited by hand One definition, several targets
Deployment Manual clicks databricks bundle deploy in a pipeline
Rollback Guess the previous settings Redeploy a previous Git commit
Consistency Differences between dev and prod Same code, only variables differ

2. Databricks Git folders

A Git folder is a folder in the workspace that is linked to a remote Git repository (GitHub, GitLab, Azure DevOps, Bitbucket and others are supported). The workspace holds a clone of the repository. Git folders were called Repos in earlier documentation; the exam guide now uses the name Git folders, so recognize both.

What you can do from a Git folder in the workspace UI:

  • Clone a remote repository into the workspace, using a Git credential (a personal access token or a linked account) stored for your user.
  • Create and switch branches.
  • Commit and push changes, and pull changes made by others. Before committing you can review the changed files and write a commit message.
  • Resolve merge conflicts in simple cases, and discard or reset local changes.

What you do not do in the workspace: a pull request is created and reviewed on the Git provider's side (for example GitHub), not inside Databricks. The Git folder is your working copy; the provider is where review, approval and merge happen. After the merge, a pipeline or another engineer pulls the updated branch.

A recommended team workflow:

  1. Each engineer has a personal Git folder (in their user area) cloned from the shared repository.
  2. The engineer creates a feature branch, edits notebooks or Python files, runs them on a development cluster or serverless compute, and commits.
  3. The engineer pushes the branch and opens a pull request on the Git provider.
  4. Automated checks run on the pull request (see section 6). A teammate reviews it.
  5. After approval the branch is merged into main. A CI/CD pipeline then deploys the merged code to the test and production workspaces.

An important rule: production should not run from a user's personal Git folder. A personal folder can sit on any branch with uncommitted changes. Production jobs should run a version-controlled, deployed copy: either a Git source configured on the job (a specific branch, tag or commit) or the files deployed by a bundle.

Concrete example. Contoso Retail has a repository retail-data-platform on GitHub. Amira clones it into her Git folder, creates the branch feature/fix-returns-join, repairs a join in the silver notebook, tests it on a small date range and pushes. A teammate reviews the pull request, a GitHub Action validates the bundle, and the merge to main triggers the deployment to the test workspace.

Git source on a job

A job task can also point to a remote Git repository instead of a workspace path. You configure the repository URL and a branch, tag or commit, and each run uses that version of the notebook. Pinning a tag or commit gives reproducible runs; pointing to a branch gives you "latest of that branch" behavior. This is a lightweight way to version a job even without bundles, but bundles are the recommended approach for managing complete projects.

3. Declarative Automation Bundles: the idea

A bundle is a project that contains:

  • the source files: notebooks, Python files, SQL files, pipeline definitions, tests;
  • the configuration (YAML) that declares the resources to create in a workspace: jobs, pipelines, and other supported objects such as dashboards, models or volumes;
  • one or more targets: the environments where the resources are deployed.

It is declarative: you describe the desired state (a job with these tasks and this schedule) and the CLI makes the workspace match it. Because the description is plain text, it lives in Git, goes through pull requests, and can be validated in CI.

The product was named Databricks Asset Bundles until recently. The exam guide uses the new name, Declarative Automation Bundles, and the Databricks CLI commands keep the bundle word: databricks bundle ....

A bundle is created with databricks bundle init (from a template) and organized like this:

retail-data-platform/
  databricks.yml            # the root configuration file (exactly one)
  resources/
    ingest_job.yml          # job definition
    sales_pipeline.yml      # pipeline definition
  src/
    ingest_orders.py
    transform/
      silver_orders.sql
  tests/
    test_transformations.py

The databricks.yml file

Every bundle has exactly one databricks.yml at the root. It declares the bundle name, includes other YAML files, defines variables and targets, and may declare resources directly.

bundle:
  name: retail-data-platform

include:
  - resources/*.yml

variables:
  catalog:
    description: Unity Catalog catalog for this environment
    default: dev_retail
  warehouse_id:
    description: SQL warehouse used by SQL tasks

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://adb-1111111111111111.1.azuredatabricks.net

  test:
    workspace:
      host: https://adb-2222222222222222.2.azuredatabricks.net
    variables:
      catalog: test_retail

  prod:
    mode: production
    workspace:
      host: https://adb-3333333333333333.3.azuredatabricks.net
      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}
    variables:
      catalog: prod_retail
    run_as:
      service_principal_name: 4f7c0d2e-1111-2222-3333-444455556666

Read it from top to bottom:

  • bundle.name identifies the project. Together with the target it determines where deployed files and state are stored.
  • include pulls in other YAML files so resources stay in small readable files.
  • variables declares named values with optional defaults. They are referenced as ${var.catalog}.
  • targets lists the environments. Each target can set the workspace host, override variables, and change resource settings.

4. Targets, variables and overrides

A target is a named deployment environment such as dev, test or prod. The same resource definitions are deployed to each target, with only the differences declared as overrides.

Common uses of targets and variables:

Need Mechanism
Different workspace per environment workspace.host in each target
Different catalog or schema per environment A variable with a different value per target
Smaller cluster in dev, bigger in prod Override a resource setting under targets.<name>.resources
Pause schedules in dev Override trigger / schedule pause_status for the dev target
Deploy production under a service identity run_as with a service principal

Development and production modes

A target can set mode:

  • mode: development marks deployed resources as development resources: names are prefixed with the user and target (for example [dev amira] ingest_orders), schedules and triggers are paused by default, and deployments are isolated per user. This prevents a developer from accidentally running a copy of a production schedule.
  • mode: production enables safety checks: for example, it expects a Git branch and a deployment identity to be set up properly and does not add the development prefix. Production deployments should run as a service principal, not as a person.

Variables

Variables avoid copying values around. You declare them once, give a default, and override them per target or on the command line:

databricks bundle deploy -t prod --var="catalog=prod_retail"

A variable is referenced in resource files as ${var.catalog}. There are also built-in references such as ${bundle.target} and ${workspace.current_user.userName}.

Precedence in practice: a value given on the command line wins over a target override, which wins over the default declared at the top level.

Resource definitions

A job resource in resources/ingest_job.yml might look like this:

resources:
  jobs:
    ingest_orders_job:
      name: ingest_orders_${bundle.target}
      tasks:
        - task_key: bronze_orders
          notebook_task:
            notebook_path: ../src/ingest_orders.py
            base_parameters:
              catalog: ${var.catalog}
          environment_key: default
        - task_key: silver_orders
          depends_on:
            - task_key: bronze_orders
          sql_task:
            file:
              path: ../src/transform/silver_orders.sql
            warehouse_id: ${var.warehouse_id}
      trigger:
        file_arrival:
          url: /Volumes/${var.catalog}/landing/orders/

A pipeline resource is similar:

resources:
  pipelines:
    sales_pipeline:
      name: sales_pipeline_${bundle.target}
      catalog: ${var.catalog}
      schema: sales
      libraries:
        - file:
            path: ../src/transform/sales_pipeline.sql

The paths are relative to the YAML file. The same files describe the Lakeflow Job (formerly Workflows) and the Lakeflow Spark Declarative Pipeline (formerly Delta Live Tables). When you deploy, the CLI uploads the source files to the workspace and creates or updates the job and the pipeline. If you rename a resource key, the CLI treats it as a new resource, so renaming is a deliberate action.

5. The Databricks CLI workflow

The Databricks CLI is the tool that runs bundle commands. A CI/CD pipeline uses a service principal credential and these commands:

Command What it does When to use it
databricks bundle init Creates a new bundle from a template Starting a project
databricks bundle validate Checks that the configuration is syntactically and structurally valid and prints the resolved settings On every pull request, before deploying
databricks bundle deploy -t <target> Uploads files and creates or updates the declared resources in the workspace Pipeline step after merge, or by a developer to dev
databricks bundle run -t <target> <resource_key> Starts a deployed job or pipeline update and can follow the run Smoke test after deploy, integration test, or manual run
databricks bundle summary Shows deployed resources and links Checking what is deployed
databricks bundle destroy -t <target> Deletes the deployed resources and files of that target Cleaning a temporary or feature environment

If the bundle has a default target (default: true), you can omit -t for that target. For any other target, such as prod, you pass -t prod.

Important behaviors for the exam:

  • validate does not change anything in the workspace. deploy does.
  • deploy is idempotent: running it again with the same configuration updates the workspace to match and does not duplicate resources.
  • run runs a deployed resource; you must deploy first, otherwise the run uses the previously deployed version.
  • destroy removes what the bundle deployed to that target, and it asks for confirmation unless you pass an auto-approve option. Use it carefully, never casually against prod.
  • The CLI authenticates with the workspace through a profile, environment variables or OAuth. In CI use a service principal with OAuth machine-to-machine credentials stored as secrets in the CI system, never a personal access token of a person who may leave the company.

A typical pipeline

# .github/workflows/deploy.yml (simplified)
on:
  pull_request:
  push:
    branches: [main]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: databricks bundle validate -t test
        env:
          DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
          DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
          DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}

  deploy-test:
    if: github.ref == 'refs/heads/main'
    needs: validate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: databricks bundle deploy -t test
      - run: databricks bundle run -t test ingest_orders_job

The flow is: pull request -> validate (and unit tests) -> merge -> deploy to test -> run an integration test -> approval -> deploy to prod.

6. Testing in CI

A bundle fits naturally with automated tests:

  • Unit tests for Python transformation functions (for example with pytest) run in the CI runner without Databricks, using small local data frames.
  • Integration tests deploy to a test target and run a job against test data, then assert on the results.
  • Validation (bundle validate) catches typos, missing variables and wrong structure before any deployment.

Keep transformation logic in functions that take and return data frames. A function like clean_orders(df) is easy to test; logic buried in a notebook cell is not.

Worked example: promoting a pipeline from dev to prod

Scenario. A logistics company has a bundle with a job load_shipments that reads files from a volume and writes to shipments_silver. In dev the table must go to dev_logistics.silver, in production to prod_logistics.silver. The production job must run every hour as a service principal, while in dev nobody wants a schedule running. Today, an engineer changes the catalog name by hand before each release and once forgot to revert it.

Step 1: Remove hard-coded names. Replace the literal catalog in the notebook parameters with ${var.catalog}, and declare catalog in databricks.yml with the default dev_logistics.

Step 2: Declare the targets. Add dev (mode development, default) and prod (mode production) targets. In prod set variables.catalog: prod_logistics and run_as to a service principal.

Step 3: Control the schedule. Define the hourly cron schedule in the job. Development mode pauses it automatically in dev, so no override is needed. In prod, the schedule is active.

Step 4: Automate. In the CI system, run databricks bundle validate -t prod on pull requests, and after merge run databricks bundle deploy -t prod using the service principal credentials stored as secrets.

Result. The same code is deployed everywhere. Only the variable and the mode change, no one edits a catalog by hand, and the production job runs under a non-personal identity.

Why the alternatives are worse. Keeping two copies of the notebook ("dev" and "prod") causes drift. Editing the job in the production UI loses the history and breaks the next deployment. Running production from a personal Git folder ties it to one person's branch state and credentials.

Common pitfalls and exam tips

  • Git folders = Repos. The old name may still appear in options. The pull request itself is created on the Git provider.
  • Bundles = Databricks Asset Bundles. The configuration file is databricks.yml; resources are declared in YAML.
  • validate changes nothing; deploy does. Order: validate, deploy, run.
  • -t selects the target. A default target can be omitted; prod cannot be implied.
  • Variables, not copies. Differences between environments belong in variables and target overrides.
  • Development mode prefixes names and pauses schedules; production mode is meant for service principals and stricter checks.
  • Never use a personal token in CI. Use a service principal with stored secrets.
  • destroy is destructive. It removes the deployed resources of the chosen target.

Exam tip: When a question says "the same job must be deployed to dev, test and prod with different catalogs", the answer is a bundle with targets and variables, not three manually edited jobs and not three copies of the code. When it says "review changes before they reach production", the answer involves Git branches and pull requests on the Git provider.

Mini case study

Situation. A bank's data team has two engineers who edit the same shared notebook folder in the production workspace. Last week an unreviewed change broke the nightly job. The team wants (1) changes reviewed before production, (2) the same job in staging and production with different catalogs, (3) automatic checks, and (4) deployments that do not depend on any engineer's account.

Task. Propose a design using Databricks tools.

Solution.

  1. Move the code to a Git repository. Each engineer works in a personal Git folder on a feature branch and opens a pull request on the Git provider; merges to main require a review.
  2. Convert the job and pipeline to a Declarative Automation Bundle with staging and prod targets, and a catalog variable overridden per target.
  3. In the CI system, run databricks bundle validate and unit tests on every pull request.
  4. After merge, deploy to staging with databricks bundle deploy -t staging, run an integration job with databricks bundle run, then deploy to prod after approval.
  5. Use a service principal with run_as in the production target and for the CI credentials, so deployments and production runs do not depend on a person.
  6. Remove write access of engineers to the production folder, so that production changes come only from the pipeline.

Key terms

  • Git folder: workspace folder linked to a remote Git repository (formerly Repos).
  • Pull request: review and merge request created on the Git provider.
  • Declarative Automation Bundle: project of source files and YAML configuration deployed with the CLI (formerly Databricks Asset Bundles).
  • databricks.yml: root configuration file of a bundle.
  • Target: named deployment environment (dev, test, prod) with its own overrides.
  • Variable: named value referenced as ${var.name} and overridable per target.
  • Resource: object declared in a bundle, such as a job or a pipeline.
  • Development mode / production mode: target settings that change naming, schedules and safety checks.
  • Service principal: non-human identity used for automation and production runs.
  • bundle validate / deploy / run / destroy: CLI commands to check, release, execute and remove a bundle.

Summary checklist

  • I can describe the Git folder workflow: branch, commit, push, pull request on the provider.
  • I know that production should not run from a personal Git folder.
  • I can explain the role of databricks.yml, resources, targets and variables.
  • I can explain development mode versus production mode.
  • I can order the commands: validate, deploy, run, and say what destroy does.
  • I know CI/CD should authenticate with a service principal.
  • I can design a dev-to-prod promotion without copying code.

Official sources