Choosing Databricks Compute for Each Workload

Introduction

This lesson covers the second half of the Databricks Intelligence Platform section of the official exam outline (about 6% of the exam in total, shared with lesson 1). The exam asks you to choose the right compute for a described workload and to understand the limits of each option. These questions are usually short scenarios: an interactive exploration, a nightly production job, a BI dashboard, a streaming pipeline. The best answer balances cost, startup time, isolation and governance.

After this lesson you will be able to compare all-purpose compute and jobs compute, explain what serverless compute changes, pick between serverless, pro and classic SQL warehouses, describe access modes, autoscaling, pools and Photon, and reason about DBU-based costs.

Learning objectives

• Select all-purpose compute, jobs compute, serverless compute or a SQL warehouse for a described workload.
• Explain the three SQL warehouse types and when each is appropriate.
• Describe standard and dedicated access modes and their relationship with Unity Catalog.
• Explain autoscaling, auto termination, pools, Photon and the DBU billing model.
• Recognize the limitations that make a compute option unsuitable.

1. The compute landscape at a glance

Databricks offers several compute types. Every one of them runs Apache Spark (or a Spark-compatible engine such as Photon) and reads the same Unity Catalog tables. They differ in who manages them, how they start, how they are billed and what they are meant for.

2. All-purpose compute versus jobs compute

All-purpose compute

All-purpose compute (often still called an "all-purpose cluster") is created manually, attached to notebooks, and can be shared by several users. It stays running until you terminate it, or until auto termination shuts it down after a configurable idle period. It is meant for development: exploring data, debugging a transformation, running ad-hoc queries.

Because it is interactive and long-lived, it is billed at the higher all-purpose DBU rate. Using it for scheduled production runs is the classic cost mistake.

Jobs compute

Jobs compute is created automatically when a Lakeflow Job run starts and terminated when the run finishes. It is isolated from other workloads (nobody else's notebook can slow your run) and billed at a lower jobs rate. You define it in the task or as a shared job cluster reused by several tasks of the same run.

Rule of thumb for the exam: development and exploration use all-purpose compute; automated, scheduled production workloads use jobs compute (or serverless jobs compute).

Worked example: a nightly ETL on the wrong cluster

Scenario. A team schedules a notebook that runs for 40 minutes each night. They point the job at a long-running all-purpose cluster that also serves 15 analysts during the day. The job slows down in the morning when analysts log in, and the finance team notices the bill is high. What should they change?

Step-by-step reasoning.

1. The workload is automated and scheduled, so it does not need an interactive cluster.
2. Sharing the cluster with analysts causes resource contention, which explains the unpredictable run time.
3. All-purpose DBUs cost more than jobs DBUs, which explains the bill.
4. Solution: configure the task to run on jobs compute (a new cluster per run, with the needed size and autoscaling), or on serverless jobs compute if available in the workspace.

Why the alternatives are weaker. Making the all-purpose cluster larger hides the contention but increases cost. Moving the job to a bigger all-purpose cluster at night only is fragile and still billed at the higher rate. Disabling auto termination makes the cost even worse.

3. Serverless compute

With serverless compute, Databricks provisions, scales and patches the compute in a serverless compute plane. For notebooks, jobs and pipelines, benefits are:

• Fast startup: typically seconds instead of minutes, with no cluster to size.
• Less administration: no node types, Spark versions or init scripts to maintain, and automatic upgrades.
• Elastic scaling: capacity adjusts to the workload.
• Billing: you pay for DBUs consumed; there is no separate bill for the VMs in your cloud account.

Limitations to remember (the exam may describe a situation where serverless is not suitable):

• It supports a defined set of Spark configurations and runtime features; you cannot set arbitrary Spark properties, install init scripts, or choose the instance type.
• Some library or networking requirements (for example a custom container, or reaching private resources) need extra configuration or are not supported; check serverless networking and egress controls in the documentation.
• Workloads that need very specific hardware (GPU types, custom images) or that rely on low-level cluster customization stay on classic compute.
• Cost model differs: for a steady, 24/7 workload with well-tuned classic clusters and reserved cloud capacity, serverless is not automatically cheaper.

Serverless compute requires Unity Catalog, so data access always goes through Unity Catalog permissions.

4. SQL warehouses

A SQL warehouse is compute optimized for SQL queries, BI tools and dashboards. It is the right choice when users connect with Power BI, Tableau, JDBC/ODBC drivers, the SQL editor or Databricks SQL dashboards. There are three types:

Key points:

• Warehouses are sized in T-shirt sizes (2X-Small to 4X-Large). A larger size makes single queries faster; more clusters (min and max) provide concurrency for many simultaneous users through autoscaling.
• Warehouses have auto stop, which stops them after idle time (serverless warehouses often use a very short value).
• SQL warehouses are for SQL (and Python UDFs in SQL). To run arbitrary PySpark notebooks or machine learning libraries, use a notebook compute.

Example. A finance dashboard is opened by 60 managers at 9:00 each Monday, with about 20 concurrent queries for ten minutes. A serverless SQL warehouse starts in seconds when the first manager opens it, scales out clusters for the burst, and stops shortly after. A classic warehouse would either run all week (wasting money) or take minutes to start (frustrating the managers).

5. Access modes

A cluster's access mode controls who can use it and how Unity Catalog enforces security. The current names are standard (formerly shared) and dedicated (formerly single user).

Practical rules:

• Choose standard access mode when several people share compute and you need user isolation.
• Choose dedicated access mode when a single user or service principal needs capabilities restricted in standard mode, such as certain low-level APIs or GPU/ML workloads.
• Legacy "No isolation shared" clusters do not support Unity Catalog and should not be used for governed data.

Exam tip. If a scenario says "many analysts share one cluster and must not see each other's data or processes", the answer is standard access mode. If it says "a data scientist needs a specialized ML library that is not supported on standard mode", think dedicated.

6. Autoscaling, auto termination, pools, and Photon

Autoscaling

With autoscaling enabled, you set a minimum and maximum number of workers and Databricks adds or removes workers according to load. It helps when the workload varies (a join phase needs many workers, a write phase needs few). It does not help much for a job whose load is constant, and a fixed size can be more predictable for benchmarks.

Auto termination

Auto termination stops an all-purpose cluster after N idle minutes, avoiding paying for forgotten clusters. Jobs compute terminates automatically after the run.

Pools

An instance pool keeps a set of idle, ready-to-use VMs. Clusters that draw from the pool start faster (no VM provisioning delay) and you pay only cloud VM costs for the idle instances (no DBU charge while idle). Pools help classic compute where startup time matters and there are many short jobs. Serverless compute removes most of that need.

Photon

Photon is Databricks' native vectorized query engine written in C++. It accelerates SQL and DataFrame operations (scans, joins, aggregations, Delta writes) with no code changes. It is enabled on SQL warehouses and serverless compute by default and can be selected on classic clusters; the DBU rate is higher but the work often finishes faster, so total cost is frequently lower for scan- and join-heavy workloads. Photon does not speed up Python UDFs that run row by row.

7. DBUs and cost reasoning

A Databricks Unit (DBU) is a normalized unit of processing capability per hour. Your cost has two parts on classic compute:

total cost = (DBUs consumed per hour x DBU rate for the product tier) + (cloud VM cost per hour)

On serverless compute, the DBU price already includes the infrastructure. The DBU rate depends on the product (jobs, all-purpose, SQL, pipelines) and the plan.

Worked example: comparing two options

A pipeline runs 2 hours a night. Option A is a classic jobs cluster that needs 6 minutes to start and consumes 12 DBU per hour at a jobs rate of 0.15 USD, plus 3.00 USD per hour for VMs. Option B is serverless jobs compute at an effective 0.37 USD per DBU with 10 DBU per hour of consumption and no VM bill.

• Option A: (12 x 0.15 + 3.00) x 2.1 h (including start) = (1.80 + 3.00) x 2.1 = 10.08 USD
• Option B: 10 x 0.37 x 2.0 h = 7.40 USD

Here serverless is cheaper because it skips startup time and VM overhead. These prices are invented for the exercise: on the exam, you reason about the factors (startup, idle time, rates, VM cost) and do not need real prices.

Choosing compute: decision table

Common pitfalls and exam tips

• Pitfall: running production on all-purpose compute. It is more expensive and shares resources.
• Pitfall: using a SQL warehouse for a PySpark notebook. Warehouses are for SQL.
• Pitfall: leaving auto termination disabled. Idle clusters keep billing.
• Pitfall: assuming autoscaling always saves money. It helps variable loads; constant loads gain little.
• Pitfall: assuming serverless supports everything. Custom Spark configs, init scripts and some libraries are limited.
• Exam tip. The words "BI tool", "dashboard" or "JDBC" point to a SQL warehouse; "scheduled" points to jobs compute; "shared by many users with isolation" points to standard access mode.
• Exam tip. When asked how to reduce startup time on classic clusters, think pools; when asked to remove cluster management altogether, think serverless.

Mini case study

Case. Fabrikam Logistics has three workloads: (1) data engineers develop transformations in notebooks during the day; (2) a pipeline loads delivery events every hour and takes 8 minutes; (3) operations managers refresh a dashboard of delivery delays throughout the day, with brief bursts of activity. The pipeline's startup time on classic jobs compute is 5 minutes, which is almost as long as the run itself.

Tasks. Recommend compute for each workload and justify.

Solution.

1. Development: all-purpose compute in standard access mode with auto termination after 30 minutes (or serverless notebook compute, which starts in seconds and removes sizing decisions).
2. Hourly pipeline: serverless jobs compute, because the five-minute startup dominates the 8-minute run; alternatively classic jobs compute with an instance pool to cut startup time. Do not use the shared all-purpose cluster.
3. Dashboard: a serverless SQL warehouse with a short auto stop, scaling clusters for bursts and starting in seconds.

The result avoids idle cost, improves startup time, and keeps all three workloads under Unity Catalog governance.

Key terms

• All-purpose compute: interactive, manually managed compute.
• Jobs compute: ephemeral compute created per job run at a lower rate.
• Serverless compute: Databricks-managed compute in a serverless plane.
• SQL warehouse: compute for SQL and BI (serverless, pro, classic).
• Access mode: standard or dedicated isolation model.
• Pool: warm idle VMs for fast startup.
• Photon: native vectorized engine.
• DBU: Databricks Unit, the billing unit.

Summary checklist

• I can choose between all-purpose, jobs, serverless and SQL warehouse compute for a scenario.
• I know the difference between serverless, pro and classic SQL warehouses.
• I can explain standard versus dedicated access mode.
• I know what autoscaling, auto termination, pools and Photon do.
• I can reason about cost using DBUs, startup time and idle time.
• I know serverless limitations (custom configs, init scripts, special hardware).

Official sources

• Exam guide: https://www.databricks.com/sites/default/files/2026-05/databricks-certified-data-engineer-associate-exam-guide-may-2026.pdf (opens in a new tab)
• Compute documentation: https://docs.databricks.com/aws/en/compute/ (opens in a new tab)
• SQL warehouse types: https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-types (opens in a new tab)