Databricks Compute Selection: Jobs, Serverless and SQL Warehouses
Introduction
Databricks compute selection starts with the task, its required APIs and its latency target. This lesson covers the second half of the Databricks Intelligence Platform section of the official exam outline (about 6% of the exam in total, shared with lesson 1). The exam asks you to choose the right compute for a described workload and to understand the limits of each option. These questions are usually short scenarios: an interactive exploration, a nightly production job, a BI dashboard, a streaming pipeline. The best answer balances cost, startup time, isolation and governance.
After this lesson you will be able to compare all-purpose compute and jobs compute, explain what serverless compute changes, pick between serverless, pro and classic SQL warehouses, describe access modes, autoscaling, pools and Photon, and reason about DBU-based costs.
Learning objectives
- Select all-purpose compute, jobs compute, serverless compute or a SQL warehouse for a described workload.
- Explain the three SQL warehouse types and when each is appropriate.
- Describe standard and dedicated access modes and their relationship with Unity Catalog.
- Explain autoscaling, auto termination, pools, Photon and the DBU billing model.
- Recognize the limitations that make a compute option unsuitable.
1. The compute landscape at a glance
Databricks offers several compute types. These services can query the same governed tables, but their supported languages and APIs differ. Photon accelerates supported Spark SQL operations; it does not replace every Spark API. They differ in who manages them, how they start, how they are billed and what they are meant for.
| Compute | Managed by | Typical use | Start time | Billing |
|---|---|---|---|---|
| All-purpose compute (classic) | You configure it | Interactive development, notebooks shared by a team | Minutes | DBUs at all-purpose rate plus cloud VM cost |
| Jobs compute (classic) | You configure it, created per run | Automated production jobs and pipelines | Minutes | DBUs at lower jobs rate plus cloud VM cost |
| Serverless compute for notebooks and jobs | Databricks | Interactive and scheduled work without cluster management | Seconds | DBUs only, no separate VM bill |
| Serverless compute for pipelines | Databricks | Lakeflow Spark Declarative Pipelines | Seconds | DBUs only |
| SQL warehouse (serverless, pro, classic) | Databricks (serverless) or customer-configured classic compute | SQL, BI tools, dashboards, JDBC/ODBC | Seconds (serverless) to minutes | DBUs; pro/classic also incur cloud infrastructure charges |
2. All-purpose compute versus jobs compute
All-purpose compute
All-purpose compute (often still called an "all-purpose cluster") is created manually, attached to notebooks, and can be shared by several users. It stays running until you terminate it, or until auto termination shuts it down after a configurable idle period. It is meant for development: exploring data, debugging a transformation, running ad-hoc queries.
Because it is interactive and long-lived, it is billed at the higher all-purpose DBU rate. Using it for scheduled production runs is the classic cost mistake.
Jobs compute
Classic jobs compute is provisioned for the tasks that need it and terminated after its associated work completes. A job cluster can serve one task or be shared by tasks within the same run; it is not a persistent interactive cluster. It avoids sharing workers with analysts and uses jobs pricing. The chosen compute must support the task type. See Lakeflow Jobs task orchestration for task dependencies.
Rule of thumb for the exam: development and exploration use all-purpose compute; automated, scheduled production workloads use jobs compute (or serverless jobs compute).
Worked example: a nightly ETL on the wrong cluster
Scenario. A team schedules a 40-minute notebook run at 05:40. It uses the all-purpose cluster that 15 analysts start querying at 06:00. The overlapping activity makes the last part of the run unpredictable, and the finance team notices the bill is high. What should they change?
Step-by-step reasoning.
- The workload is automated and scheduled, so it does not need an interactive cluster.
- Sharing the cluster with analysts causes resource contention, which explains the unpredictable run time.
- All-purpose DBUs cost more than jobs DBUs, which explains the bill.
- Solution: configure the task to run on jobs compute (a new cluster per run, with the needed size and autoscaling), or on serverless jobs compute if available in the workspace.
Why the alternatives are weaker. Making the all-purpose cluster larger hides the contention but increases cost. Moving the job to a bigger all-purpose cluster at night only is fragile and still billed at the higher rate. Disabling auto termination makes the cost even worse.
3. Serverless compute
With serverless compute, Databricks provisions, scales and patches the compute in a serverless compute plane. For notebooks, jobs and pipelines, benefits are:
- Fast startup: typically seconds instead of minutes, with no cluster to size.
- Less administration: no node types, Spark versions or init scripts to maintain, and automatic upgrades.
- Elastic scaling: capacity adjusts to the workload.
- Billing: you pay for DBUs consumed; there is no separate bill for the VMs in your cloud account.
Limitations to remember (the exam may describe a situation where serverless is not suitable):
- It supports a defined set of Spark configurations and runtime features; you cannot set arbitrary Spark properties, install init scripts, or choose the instance type.
- Some library or networking requirements (for example a custom container, or reaching private resources) need extra configuration or are not supported; check serverless networking and egress controls in the documentation.
- Required OS customization, unsupported APIs or a specific unavailable accelerator can require classic compute. GPU use alone is not sufficient to reject serverless: AI Runtime offers serverless GPU notebooks in Public Preview, subject to workspace and regional availability.
- Cost model differs: for a steady, 24/7 workload with well-tuned classic clusters and reserved cloud capacity, serverless is not automatically cheaper.
For serverless notebooks and jobs, configure external data through Unity Catalog rather than legacy mounts. These products do not support arbitrary cluster customization. Their supported streaming trigger is typically AvailableNow; use an appropriate Lakeflow pipeline for a continuous stream rather than assuming a notebook supports every trigger.
4. SQL warehouses
A SQL warehouse is compute optimized for SQL queries, BI tools and dashboards. It is the right choice when users connect with Power BI, Tableau, JDBC/ODBC drivers, the SQL editor or Databricks SQL dashboards. There are three types:
| Type | Features | Management | When to choose |
|---|---|---|---|
| Serverless | Fast startup, Photon, intelligent workload management, predictive I/O | Databricks | Preferred starting point for supported BI and ad-hoc SQL; measure cost and latency |
| Pro | Photon and predictive I/O; no intelligent workload management; starts in minutes | Customer-configured classic compute | When serverless is unavailable or networking requirements favor this design |
| Classic | Entry-level SQL, Photon but fewer advanced capabilities, runs in your account | You | Simple, cost-sensitive SQL where advanced features are not needed |
Key points:
- Warehouse size sets resources per cluster; cluster count supports concurrent queries. Larger size may help memory pressure, but does not guarantee every query runs faster. Available sizes include a 5X-Large Public Preview for supported pro/serverless warehouses; inspect workspace availability.
- Warehouses have auto stop, which stops them after idle time (serverless warehouses often use a very short value).
- SQL warehouses serve SQL workloads, including supported Python UDFs invoked from SQL. They are not general PySpark notebook or model-training environments. SQL ETL tasks can also use a supported warehouse.
The three general-purpose types above remain the core comparison. Current documentation additionally lists Lakehouse Real-Time as a specialized serverless Beta for low-latency read queries; it is not a general ETL replacement.
Example. A finance dashboard is opened by 60 managers at 9:00 each Monday, with about 20 concurrent queries for ten minutes. A serverless SQL warehouse starts in seconds when the first manager opens it, scales out clusters for the burst, and stops shortly after. A classic warehouse can also auto-stop, but typically starts more slowly. Compare measured startup and queueing behavior against the service target.
5. Access modes
A cluster's access mode controls who can use it and how Unity Catalog enforces security. The current names are standard (formerly shared) and dedicated (formerly single user).
| Access mode | Users | Unity Catalog | Typical use |
|---|---|---|---|
| Standard | Multiple users, isolated from each other | Supported, with enforcement of table, row and column controls | Shared interactive work and jobs |
| Dedicated | Assigned to one user or group (and service principals) | Supported; the compute runs with the permissions of the assigned user or group, and some fine-grained controls (row filters, column masks) depend on the runtime and on serverless filtering | Workloads needing features that standard mode restricts (some languages, libraries, ML runtime and RDD APIs) |
Practical rules:
- Choose standard access mode when several people share compute and you need user isolation.
- Choose dedicated access mode when a single user or service principal needs capabilities restricted in standard mode, such as certain low-level APIs or GPU/ML workloads.
- Legacy "No isolation shared" clusters do not support Unity Catalog and should not be used for governed data.
Exam tip. If a scenario says "many analysts share one cluster and must not see each other's data or processes", the answer is standard access mode. If it says "a data scientist needs a specialized ML library that is not supported on standard mode", think dedicated.
6. Autoscaling, auto termination, pools, and Photon
Autoscaling
With autoscaling enabled, you set a minimum and maximum number of workers and Databricks adds or removes workers according to load. It helps when the workload varies (a join phase needs many workers, a write phase needs few). It does not help much for a job whose load is constant, and a fixed size can be more predictable for benchmarks.
Auto termination
Auto termination stops an all-purpose cluster after N idle minutes, avoiding paying for forgotten clusters. Classic jobs compute is terminated automatically when its associated tasks finish.
Pools
An instance pool keeps a set of idle, ready-to-use VMs. Clusters that draw from the pool start faster (no VM provisioning delay) and you pay only cloud VM costs for the idle instances (no DBU charge while idle). Pools help classic compute where startup time matters and there are many short jobs. Serverless compute removes most of that need.
Photon
Photon is Databricks' native vectorized query engine written in C++. It accelerates SQL and DataFrame operations (scans, joins, aggregations, Delta writes) with no code changes. It is enabled on SQL warehouses and serverless compute by default and can be selected on classic clusters; the DBU rate is higher but the work often finishes faster, so total cost is frequently lower for scan- and join-heavy workloads. Photon does not speed up Python UDFs that run row by row.
7. DBUs and cost reasoning
A Databricks Unit (DBU) is a normalized unit of processing capability per hour. Your cost has two parts on classic compute:
compute cost = (DBU/hour x USD/DBU + VM USD/hour) x billed hoursOn serverless compute, the DBU price already includes the infrastructure. The DBU rate depends on the product and plan. Storage, network transfer and other services may add charges to either design; compare total workload cost.
Worked example: comparing two options
A pipeline does two hours of useful work each night. For this simplified exercise, assume option A is billed at a constant 12 DBU/hour plus 3.00 USD/hour of VM cost for 2.1 hours, including six minutes of startup, with 0.15 USD/DBU. Option B consumes 10 DBU/hour at 0.37 USD/DBU for exactly two hours, with infrastructure included. These are assumed billing intervals, not a rule for charging all real startup phases.
- Option A: (12 x 0.15 + 3.00) x 2.1 h (including start) = (1.80 + 3.00) x 2.1 = 10.08 USD
- Option B: 10 x 0.37 x 2.0 h = 7.40 USD
Under these assumptions, serverless costs less. Real startup, resource consumption and billed duration must be measured. These prices are invented for the exercise: on the exam, you reason about the factors (startup, idle time, rates, VM cost) and do not need real prices.
Choosing compute: decision table
| Situation | Best choice | Why |
|---|---|---|
| Exploring data interactively as a team | All-purpose compute (standard access mode) or serverless notebook compute | Interactive, shareable |
| Nightly scheduled ETL | Jobs compute or serverless jobs | Lower rate, isolated, ends automatically |
| BI dashboard used by many managers | Serverless SQL warehouse | Fast start, concurrency scaling, auto stop |
| Lakeflow Spark Declarative Pipeline | Serverless pipeline compute or pipeline-managed classic compute | Managed scaling and updates |
| Many short jobs, classic compute, slow start | Instance pool | Warm VMs reduce startup |
| Workload requiring an unsupported custom container or accelerator | Compatible classic compute configuration | Verify runtime, access mode and feature support; GPU alone does not decide |
Common pitfalls and exam tips
- Pitfall: running production on all-purpose compute. It is more expensive and shares resources.
- Pitfall: using a SQL warehouse for a PySpark notebook. Warehouses are for SQL.
- Pitfall: leaving auto termination disabled. Idle clusters keep billing.
- Pitfall: assuming autoscaling always saves money. It helps variable loads; constant loads gain little.
- Pitfall: assuming serverless supports everything. Custom Spark configs, init scripts and some libraries are limited.
- Exam tip. Match the operation, not one keyword: JDBC may query a SQL warehouse or ingest from an external database. Scheduled SQL can use a supported warehouse, while Python notebook tasks need compatible jobs compute. Shared classic compute with individual isolation points to standard access mode.
- Exam tip. When asked how to reduce startup time on classic clusters, think pools; when asked to remove cluster management altogether, think serverless.
Mini case study
Case. Fabrikam Logistics has three workloads: (1) data engineers develop transformations in notebooks during the day; (2) a pipeline loads delivery events every hour and takes 8 minutes; (3) operations managers refresh a dashboard of delivery delays throughout the day, with brief bursts of activity. The pipeline's startup time on classic jobs compute is 5 minutes, which is almost as long as the run itself.
Tasks. Recommend compute for each workload and justify.
Solution.
- Development: all-purpose compute in standard access mode with auto termination after 30 minutes (or serverless notebook compute, which starts in seconds and removes sizing decisions).
- Hourly pipeline: serverless jobs compute, because the five-minute startup dominates the 8-minute run; alternatively classic jobs compute with an instance pool to cut startup time. Do not use the shared all-purpose cluster.
- Dashboard: a serverless SQL warehouse with a short auto stop, scaling clusters for bursts and starting in seconds.
The result avoids idle cost, improves startup time, and keeps all three workloads under Unity Catalog governance.
Use job monitoring and Spark troubleshooting to compare run durations and diagnose resource pressure before resizing. Revisit Delta Lake and Unity Catalog architecture when checking where data is stored and governed.
Key terms
- All-purpose compute: interactive, manually managed compute.
- Jobs compute: task-oriented compute provisioned and terminated for automated work.
- Serverless compute: Databricks-managed compute in a serverless plane.
- SQL warehouse: compute for SQL and BI (serverless, pro, classic).
- Access mode: standard or dedicated isolation model.
- Pool: warm idle VMs for fast startup.
- Photon: native vectorized engine.
- DBU: Databricks Unit, the billing unit.
Summary checklist
- I can choose between all-purpose, jobs, serverless and SQL warehouse compute for a scenario.
- I know the difference between serverless, pro and classic SQL warehouses.
- I can explain standard versus dedicated access mode.
- I know what autoscaling, auto termination, pools and Photon do.
- I can reason about cost using DBUs, startup time and idle time.
- I know serverless limitations (custom configs, init scripts, special hardware).
Official sources
- Exam guide: https://www.databricks.com/sites/default/files/2026-10/Databricks-Certified-Data-Engineer-Associate-Exam-Guide-August-2026.pdf (opens in a new tab)
- Job compute selection: https://docs.databricks.com/aws/en/jobs/compute (opens in a new tab)
- Compute configuration: https://docs.databricks.com/aws/en/compute/configure (opens in a new tab)
- Serverless limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations (opens in a new tab)
- AI Runtime GPU notebooks: https://docs.databricks.com/aws/en/compute/serverless/dependencies (opens in a new tab)
- Pools and billing: https://docs.databricks.com/aws/en/compute/pool-index (opens in a new tab)
- Photon: https://docs.databricks.com/aws/en/compute/photon (opens in a new tab)
- Warehouse scaling: https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-behavior (opens in a new tab)
- SQL warehouse types: https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-types (opens in a new tab)
