Skip to content

Databricks Compute Selection: Jobs, Serverless and SQL Warehouses

Step 3 of 30

Databricks Compute Selection: Jobs, Serverless and SQL Warehouses

Introduction

Databricks compute selection starts with the task, its required APIs and its latency target. This lesson covers the second half of the Databricks Intelligence Platform section of the official exam outline (about 6% of the exam in total, shared with lesson 1). The exam asks you to choose the right compute for a described workload and to understand the limits of each option. These questions are usually short scenarios: an interactive exploration, a nightly production job, a BI dashboard, a streaming pipeline. The best answer balances cost, startup time, isolation and governance.

After this lesson you will be able to compare all-purpose compute and jobs compute, explain what serverless compute changes, pick between serverless, pro and classic SQL warehouses, describe access modes, autoscaling, pools and Photon, and reason about DBU-based costs.

Learning objectives

  • Select all-purpose compute, jobs compute, serverless compute or a SQL warehouse for a described workload.
  • Explain the three SQL warehouse types and when each is appropriate.
  • Describe standard and dedicated access modes and their relationship with Unity Catalog.
  • Explain autoscaling, auto termination, pools, Photon and the DBU billing model.
  • Recognize the limitations that make a compute option unsuitable.

1. The compute landscape at a glance

Databricks offers several compute types. These services can query the same governed tables, but their supported languages and APIs differ. Photon accelerates supported Spark SQL operations; it does not replace every Spark API. They differ in who manages them, how they start, how they are billed and what they are meant for.

Compute Managed by Typical use Start time Billing
All-purpose compute (classic) You configure it Interactive development, notebooks shared by a team Minutes DBUs at all-purpose rate plus cloud VM cost
Jobs compute (classic) You configure it, created per run Automated production jobs and pipelines Minutes DBUs at lower jobs rate plus cloud VM cost
Serverless compute for notebooks and jobs Databricks Interactive and scheduled work without cluster management Seconds DBUs only, no separate VM bill
Serverless compute for pipelines Databricks Lakeflow Spark Declarative Pipelines Seconds DBUs only
SQL warehouse (serverless, pro, classic) Databricks (serverless) or customer-configured classic compute SQL, BI tools, dashboards, JDBC/ODBC Seconds (serverless) to minutes DBUs; pro/classic also incur cloud infrastructure charges

2. All-purpose compute versus jobs compute

All-purpose compute

All-purpose compute (often still called an "all-purpose cluster") is created manually, attached to notebooks, and can be shared by several users. It stays running until you terminate it, or until auto termination shuts it down after a configurable idle period. It is meant for development: exploring data, debugging a transformation, running ad-hoc queries.

Because it is interactive and long-lived, it is billed at the higher all-purpose DBU rate. Using it for scheduled production runs is the classic cost mistake.

Jobs compute

Classic jobs compute is provisioned for the tasks that need it and terminated after its associated work completes. A job cluster can serve one task or be shared by tasks within the same run; it is not a persistent interactive cluster. It avoids sharing workers with analysts and uses jobs pricing. The chosen compute must support the task type. See Lakeflow Jobs task orchestration for task dependencies.

Rule of thumb for the exam: development and exploration use all-purpose compute; automated, scheduled production workloads use jobs compute (or serverless jobs compute).

Worked example: a nightly ETL on the wrong cluster

Scenario. A team schedules a 40-minute notebook run at 05:40. It uses the all-purpose cluster that 15 analysts start querying at 06:00. The overlapping activity makes the last part of the run unpredictable, and the finance team notices the bill is high. What should they change?

Step-by-step reasoning.

  1. The workload is automated and scheduled, so it does not need an interactive cluster.
  2. Sharing the cluster with analysts causes resource contention, which explains the unpredictable run time.
  3. All-purpose DBUs cost more than jobs DBUs, which explains the bill.
  4. Solution: configure the task to run on jobs compute (a new cluster per run, with the needed size and autoscaling), or on serverless jobs compute if available in the workspace.

Why the alternatives are weaker. Making the all-purpose cluster larger hides the contention but increases cost. Moving the job to a bigger all-purpose cluster at night only is fragile and still billed at the higher rate. Disabling auto termination makes the cost even worse.

3. Serverless compute

With serverless compute, Databricks provisions, scales and patches the compute in a serverless compute plane. For notebooks, jobs and pipelines, benefits are:

  • Fast startup: typically seconds instead of minutes, with no cluster to size.
  • Less administration: no node types, Spark versions or init scripts to maintain, and automatic upgrades.
  • Elastic scaling: capacity adjusts to the workload.
  • Billing: you pay for DBUs consumed; there is no separate bill for the VMs in your cloud account.

Limitations to remember (the exam may describe a situation where serverless is not suitable):

  • It supports a defined set of Spark configurations and runtime features; you cannot set arbitrary Spark properties, install init scripts, or choose the instance type.
  • Some library or networking requirements (for example a custom container, or reaching private resources) need extra configuration or are not supported; check serverless networking and egress controls in the documentation.
  • Required OS customization, unsupported APIs or a specific unavailable accelerator can require classic compute. GPU use alone is not sufficient to reject serverless: AI Runtime offers serverless GPU notebooks in Public Preview, subject to workspace and regional availability.
  • Cost model differs: for a steady, 24/7 workload with well-tuned classic clusters and reserved cloud capacity, serverless is not automatically cheaper.

For serverless notebooks and jobs, configure external data through Unity Catalog rather than legacy mounts. These products do not support arbitrary cluster customization. Their supported streaming trigger is typically AvailableNow; use an appropriate Lakeflow pipeline for a continuous stream rather than assuming a notebook supports every trigger.

4. SQL warehouses

A SQL warehouse is compute optimized for SQL queries, BI tools and dashboards. It is the right choice when users connect with Power BI, Tableau, JDBC/ODBC drivers, the SQL editor or Databricks SQL dashboards. There are three types:

Type Features Management When to choose
Serverless Fast startup, Photon, intelligent workload management, predictive I/O Databricks Preferred starting point for supported BI and ad-hoc SQL; measure cost and latency
Pro Photon and predictive I/O; no intelligent workload management; starts in minutes Customer-configured classic compute When serverless is unavailable or networking requirements favor this design
Classic Entry-level SQL, Photon but fewer advanced capabilities, runs in your account You Simple, cost-sensitive SQL where advanced features are not needed

Key points:

  • Warehouse size sets resources per cluster; cluster count supports concurrent queries. Larger size may help memory pressure, but does not guarantee every query runs faster. Available sizes include a 5X-Large Public Preview for supported pro/serverless warehouses; inspect workspace availability.
  • Warehouses have auto stop, which stops them after idle time (serverless warehouses often use a very short value).
  • SQL warehouses serve SQL workloads, including supported Python UDFs invoked from SQL. They are not general PySpark notebook or model-training environments. SQL ETL tasks can also use a supported warehouse.

The three general-purpose types above remain the core comparison. Current documentation additionally lists Lakehouse Real-Time as a specialized serverless Beta for low-latency read queries; it is not a general ETL replacement.

Example. A finance dashboard is opened by 60 managers at 9:00 each Monday, with about 20 concurrent queries for ten minutes. A serverless SQL warehouse starts in seconds when the first manager opens it, scales out clusters for the burst, and stops shortly after. A classic warehouse can also auto-stop, but typically starts more slowly. Compare measured startup and queueing behavior against the service target.

5. Access modes

A cluster's access mode controls who can use it and how Unity Catalog enforces security. The current names are standard (formerly shared) and dedicated (formerly single user).

Access mode Users Unity Catalog Typical use
Standard Multiple users, isolated from each other Supported, with enforcement of table, row and column controls Shared interactive work and jobs
Dedicated Assigned to one user or group (and service principals) Supported; the compute runs with the permissions of the assigned user or group, and some fine-grained controls (row filters, column masks) depend on the runtime and on serverless filtering Workloads needing features that standard mode restricts (some languages, libraries, ML runtime and RDD APIs)

Practical rules:

  • Choose standard access mode when several people share compute and you need user isolation.
  • Choose dedicated access mode when a single user or service principal needs capabilities restricted in standard mode, such as certain low-level APIs or GPU/ML workloads.
  • Legacy "No isolation shared" clusters do not support Unity Catalog and should not be used for governed data.

Exam tip. If a scenario says "many analysts share one cluster and must not see each other's data or processes", the answer is standard access mode. If it says "a data scientist needs a specialized ML library that is not supported on standard mode", think dedicated.

6. Autoscaling, auto termination, pools, and Photon

Autoscaling

With autoscaling enabled, you set a minimum and maximum number of workers and Databricks adds or removes workers according to load. It helps when the workload varies (a join phase needs many workers, a write phase needs few). It does not help much for a job whose load is constant, and a fixed size can be more predictable for benchmarks.

Auto termination

Auto termination stops an all-purpose cluster after N idle minutes, avoiding paying for forgotten clusters. Classic jobs compute is terminated automatically when its associated tasks finish.

Pools

An instance pool keeps a set of idle, ready-to-use VMs. Clusters that draw from the pool start faster (no VM provisioning delay) and you pay only cloud VM costs for the idle instances (no DBU charge while idle). Pools help classic compute where startup time matters and there are many short jobs. Serverless compute removes most of that need.

Photon

Photon is Databricks' native vectorized query engine written in C++. It accelerates SQL and DataFrame operations (scans, joins, aggregations, Delta writes) with no code changes. It is enabled on SQL warehouses and serverless compute by default and can be selected on classic clusters; the DBU rate is higher but the work often finishes faster, so total cost is frequently lower for scan- and join-heavy workloads. Photon does not speed up Python UDFs that run row by row.

7. DBUs and cost reasoning

A Databricks Unit (DBU) is a normalized unit of processing capability per hour. Your cost has two parts on classic compute:

Text
compute cost = (DBU/hour x USD/DBU + VM USD/hour) x billed hours

On serverless compute, the DBU price already includes the infrastructure. The DBU rate depends on the product and plan. Storage, network transfer and other services may add charges to either design; compare total workload cost.

Worked example: comparing two options

A pipeline does two hours of useful work each night. For this simplified exercise, assume option A is billed at a constant 12 DBU/hour plus 3.00 USD/hour of VM cost for 2.1 hours, including six minutes of startup, with 0.15 USD/DBU. Option B consumes 10 DBU/hour at 0.37 USD/DBU for exactly two hours, with infrastructure included. These are assumed billing intervals, not a rule for charging all real startup phases.

  • Option A: (12 x 0.15 + 3.00) x 2.1 h (including start) = (1.80 + 3.00) x 2.1 = 10.08 USD
  • Option B: 10 x 0.37 x 2.0 h = 7.40 USD

Under these assumptions, serverless costs less. Real startup, resource consumption and billed duration must be measured. These prices are invented for the exercise: on the exam, you reason about the factors (startup, idle time, rates, VM cost) and do not need real prices.

Choosing compute: decision table

Situation Best choice Why
Exploring data interactively as a team All-purpose compute (standard access mode) or serverless notebook compute Interactive, shareable
Nightly scheduled ETL Jobs compute or serverless jobs Lower rate, isolated, ends automatically
BI dashboard used by many managers Serverless SQL warehouse Fast start, concurrency scaling, auto stop
Lakeflow Spark Declarative Pipeline Serverless pipeline compute or pipeline-managed classic compute Managed scaling and updates
Many short jobs, classic compute, slow start Instance pool Warm VMs reduce startup
Workload requiring an unsupported custom container or accelerator Compatible classic compute configuration Verify runtime, access mode and feature support; GPU alone does not decide

Common pitfalls and exam tips

  • Pitfall: running production on all-purpose compute. It is more expensive and shares resources.
  • Pitfall: using a SQL warehouse for a PySpark notebook. Warehouses are for SQL.
  • Pitfall: leaving auto termination disabled. Idle clusters keep billing.
  • Pitfall: assuming autoscaling always saves money. It helps variable loads; constant loads gain little.
  • Pitfall: assuming serverless supports everything. Custom Spark configs, init scripts and some libraries are limited.
  • Exam tip. Match the operation, not one keyword: JDBC may query a SQL warehouse or ingest from an external database. Scheduled SQL can use a supported warehouse, while Python notebook tasks need compatible jobs compute. Shared classic compute with individual isolation points to standard access mode.
  • Exam tip. When asked how to reduce startup time on classic clusters, think pools; when asked to remove cluster management altogether, think serverless.

Mini case study

Case. Fabrikam Logistics has three workloads: (1) data engineers develop transformations in notebooks during the day; (2) a pipeline loads delivery events every hour and takes 8 minutes; (3) operations managers refresh a dashboard of delivery delays throughout the day, with brief bursts of activity. The pipeline's startup time on classic jobs compute is 5 minutes, which is almost as long as the run itself.

Tasks. Recommend compute for each workload and justify.

Solution.

  1. Development: all-purpose compute in standard access mode with auto termination after 30 minutes (or serverless notebook compute, which starts in seconds and removes sizing decisions).
  2. Hourly pipeline: serverless jobs compute, because the five-minute startup dominates the 8-minute run; alternatively classic jobs compute with an instance pool to cut startup time. Do not use the shared all-purpose cluster.
  3. Dashboard: a serverless SQL warehouse with a short auto stop, scaling clusters for bursts and starting in seconds.

The result avoids idle cost, improves startup time, and keeps all three workloads under Unity Catalog governance.

Use job monitoring and Spark troubleshooting to compare run durations and diagnose resource pressure before resizing. Revisit Delta Lake and Unity Catalog architecture when checking where data is stored and governed.

Key terms

  • All-purpose compute: interactive, manually managed compute.
  • Jobs compute: task-oriented compute provisioned and terminated for automated work.
  • Serverless compute: Databricks-managed compute in a serverless plane.
  • SQL warehouse: compute for SQL and BI (serverless, pro, classic).
  • Access mode: standard or dedicated isolation model.
  • Pool: warm idle VMs for fast startup.
  • Photon: native vectorized engine.
  • DBU: Databricks Unit, the billing unit.

Summary checklist

  • I can choose between all-purpose, jobs, serverless and SQL warehouse compute for a scenario.
  • I know the difference between serverless, pro and classic SQL warehouses.
  • I can explain standard versus dedicated access mode.
  • I know what autoscaling, auto termination, pools and Photon do.
  • I can reason about cost using DBUs, startup time and idle time.
  • I know serverless limitations (custom configs, init scripts, special hardware).

Official sources