Lakeflow Jobs: Task Dependencies, Branching and Retries
Introduction
This lesson opens the Working with Lakeflow Jobs section of the official exam outline (16% of the exam outline; this does not prescribe a fixed number of questions on an individual exam). Lakeflow Jobs orchestration turns ingestion and transformation code into a repeatable task graph for the Databricks Data Engineer Associate certification. Code that runs only when someone clicks "Run" is not a data product. A Lakeflow Job (formerly Databricks Workflows, or simply "Jobs") is the Databricks service that runs your notebooks, SQL, pipelines and other work as a graph of tasks, in the right order, with retries, parameters and alerts.
After this lesson you will be able to pick the right task type for a piece of work, build a task graph with dependencies, control what happens after a failure with retries and run-if conditions, add branching with If/else tasks and loops with For each tasks, and pass information between tasks using job parameters and task values.
Learning objectives
- Name the main task types and choose one for a notebook, a SQL query, a dashboard refresh or a pipeline update.
- Explain how a job is a directed acyclic graph (DAG) of tasks and how dependencies define the order and the parallelism.
- Configure retries, timeouts and run-if conditions and predict which tasks run when a task fails.
- Use an If/else condition task for branching and a For each task for iterating over a list.
- Use job parameters, dynamic value references and task values to pass data between tasks.
1. What a job is made of
A job has a name, one or more tasks, compatible compute for executable tasks (for example jobs/serverless compute, a SQL warehouse or pipeline-managed compute), optional parameters, optional triggers (the next lesson), and notifications. Each execution is a run. Task runs record task execution; retries, repairs and loop iterations can create additional attempts or nested executions. Do not infer that every task executes exactly once.
| Task type | What it runs | Typical use |
|---|---|---|
| Notebook | A Databricks notebook (workspace or Git folder) | Ingestion or transformation logic in Python/SQL |
| Python script / Python wheel / JAR | Packaged code | Reusable libraries, tested code |
| SQL | A saved SQL query, SQL file or supported SQL alert | SQL transformations and KPI refresh |
| Dashboard | Refreshes a published Databricks dashboard | Dashboard results and configured subscriber snapshots |
| Pipeline | An update of a Lakeflow Spark Declarative Pipeline | Streaming tables and materialized views |
| Run Job | Another job | Reuse an existing job as a building block |
| dbt | A dbt project | Teams that already use dbt |
| If/else condition | Evaluates a Boolean expression | Branching |
| For each | Runs a nested task once per input value | Loops over regions, tables, dates |
The exam outline explicitly names notebook, SQL query, dashboard and pipeline tasks, plus branching and looping. Current Dashboard tasks require an accessible published dashboard and a serverless or pro SQL warehouse; configure subscribers in the task. Older SQL dashboard task examples can describe legacy dashboards.
Compute matters here. SQL tasks run on a SQL warehouse. Notebook tasks run on jobs compute or serverless compute. A pipeline task does not run on the job's compute at all: the pipeline uses its own compute and the job simply triggers the update and waits for it. Prefer appropriately configured jobs or serverless compute for production automation. Jobs compute lifecycle is tied to the run, while SQL warehouses and pipeline compute have their own lifecycle. Pricing and total cost depend on the workload and compute product; use the Databricks compute selection lesson to choose and measure a suitable option.
Concrete example. FreshCart, an online grocer, runs a nightly flow: a notebook task ingest_orders loads new files with Auto Loader, a pipeline task build_silver_gold updates the declarative pipeline, a SQL task refresh_kpis runs an INSERT OVERWRITE into a KPI table, and a dashboard task send_daily_dashboard refreshes a dashboard and mails a PDF snapshot to the managers. Four task types, one job.
2. The task graph (DAG) and dependencies
Each task declares which tasks it depends on. Databricks builds a directed acyclic graph: no cycles are allowed. The scheduler starts a task as soon as its dependencies are satisfied. Tasks without dependencies between them can run in parallel, subject to available compute, quotas and concurrency limits.
ingest_orders ──┐
├──> build_silver_gold ──> refresh_kpis ──> send_daily_dashboard
ingest_customers┘Here the two ingestion tasks are eligible to run at the same time, the pipeline task waits for both, and the rest runs in sequence. Two consequences matter on the exam:
- Parallelism is implicit. Removing a dependency permits parallel execution; it does not guarantee simultaneous start times.
- Order is explicit. If task B reads what task A writes, B must depend on A. Without the dependency, B may start first.
A job can also be configured with a maximum number of concurrent runs of the whole job (default 1). Increasing it lets a new run start while an older one is still running, which is useful for event-driven jobs but dangerous for jobs that overwrite the same table.
3. Failure handling: retries, timeouts and run-if
Retries and timeouts
Each task has its own retry policy:
- Retries: the number of additional attempts after a failure. Inspect and set the policy explicitly; defaults and automatic retries differ by task/compute type.
- Minimum interval between retries: the configured minimum interval between the start of the failed attempt and the start of its retry, not necessarily a pause measured from failure completion.
- Retry on timeout: whether a timeout counts as a failure to retry.
- Timeout: the maximum duration of a task run; when exceeded, the run is stopped and marked as timed out.
Retries are for transient problems: a network glitch, an unavailable source, a temporarily unreachable service. They do not fix a bug in your SQL. If a task fails because of a code error, retries repeat the same fault. Make writes idempotent: an interrupted task may have committed data or performed an external side effect before failing.
Run-if conditions
By default, a task runs only when all its upstream tasks succeeded. A run-if condition changes that. It is evaluated on the dependency outcomes:
| Run-if option | The task runs when... | Typical use |
|---|---|---|
| All succeeded (default) | every dependency succeeded | Normal flow |
| At least one succeeded | one or more dependencies succeeded | Several alternative sources, any one is enough |
| None failed | no dependency failed and at least one dependency ran | Continue after optional branches |
| All done | all dependencies finished, whatever the result | Cleanup or completion notification after success/failure |
| At least one failed | one or more dependencies failed | Alert or compensation task |
| All failed | every dependency failed | Fallback when all primary paths are down |
If a success-oriented run-if condition is unmet, the task is marked Upstream failed; a failure-handler whose failure condition is unmet is Excluded. Excluded dependencies count as successful when evaluating a downstream condition. However, if all dependencies are excluded, the downstream task is excluded regardless of its run-if setting, including All done. Upstream failed and upstream canceled outcomes count as failures for this evaluation. An arbitrary “skipped” state is not interchangeable with Excluded.
Concrete example. In the FreshCart job, ingest_customers fails because the CRM export is late. With the default run-if, build_silver_gold is marked Upstream failed and never runs, so that task does not start. This is dependency gating, not a rollback of earlier writes or a transaction across gold tables. A new task notify_oncall depends on ingest_customers with run-if At least one failed and posts to the on-call channel. A task cleanup_staging depends on everything with All done, so it is eligible after success or failure. The all-excluded exception still applies; cleanup code must also preserve the source/checkpoint data needed for repair.
Repair runs
For a failed or canceled multi-task job, a repair run can rerun the unsuccessful subset and dependent tasks instead of the whole job. Successful durable outputs can be reused, but transient notebook variables and cluster memory are not shared state. Repairs use current job/task settings, so check code, configuration and input compatibility before rerunning. For a six-hour job whose final task failed, this can save substantial compute. The job monitoring and repair lesson explains how to diagnose the failure first.
4. Branching with the If/else condition task
An If/else condition task evaluates a comparison and exposes two outcomes: true and false. Downstream tasks attach to one outcome; tasks on the non-selected branch are excluded.
- Operators:
==,!=,>,>=,<,<=. - Operands can be job parameters, task values, or other dynamic value references, for example
{{tasks.validate.values.bad_records}} > 0. - The condition is evaluated by the orchestration service without notebook/SQL compute; scheduling still takes time.
validate ──> check_bad_records ──(true)──> quarantine_and_alert
└─(false)──> publish_goldHere validate is a notebook task that counts invalid rows and stores the count as a task value. The If/else task check_bad_records compares it with zero. If invalid rows exist, the quarantine branch runs; otherwise, the gold publication runs. Remember that a condition task cannot read a value from a task that did not run or that was disabled, and that operator semantics matter: == and != compare strings (12.0 == 12 is false), while >, >=, < and <= compare numerically (12.0 >= 12 is true). Referenced task values must be numeric, string or Boolean values, not arrays or objects.
5. Looping with the For each task
A For each task runs a nested task once per element of an input list. The nested task is one supported task (notebook, SQL, pipeline, Run Job and others; another For each is not allowed), and each iteration receives the current element, available in the nested task as {{input}}.
Settings that matter:
- Inputs: a JSON array typed directly (
["fr","de","es"]), a job parameter reference ({{job.parameters.countries}}), or a task value from an upstream task ({{tasks.list_tables.values.table_names}}). - Concurrency: how many iterations run at the same time (default 1, so sequential; it can be raised to run in parallel, subject to supported limits and available capacity).
- Each iteration is retried and monitored individually. Failed iterations can be repaired individually.
Concrete example. SunPeak Energy loads 40 country tables with the same notebook. Instead of 40 duplicate tasks, a list_countries task writes a task value with the country codes. A For each task reads it, runs the load_country notebook with country={{input}}, and uses concurrency 8. Adding a country requires no change in the job definition.
A common design: a first task computes the list dynamically, a For each iterates, and a final task with run-if All done (or All succeeded) reports a summary.
6. Parameters and task values
Job parameters
Job parameters are key/value pairs defined on the job. They are automatically pushed to tasks accepting key/value parameters. For tasks accepting JSON-array arguments, explicitly map {{job.parameters.<name>}} into the argument list. They have a default value and can be overridden when you start a run manually or through the API. In a notebook, read them with widgets:
run_date = dbutils.widgets.get("run_date") # job parameter "run_date"In task settings, reference them with {{job.parameters.run_date}}. In SQL tasks the parameter is passed as a SQL parameter marker/named parameter.
Dynamic value references
Dynamic value references are {{ ... }} placeholders resolved at run time. Examples: {{job.id}}, {{job.run_id}}, {{task.run_id}}, {{job.start_time.iso_date}}, {{job.parameters.run_date}}, {{tasks.<task_name>.values.<key>}} and {{tasks.<task_name>.result_state}}. Use them to avoid hard-coding dates, to name outputs per run and to wire tasks together.
Task values
Task values let one task pass a small result to downstream tasks in the same run. The set/get Python functions below are supported in Python notebooks, not arbitrary Python script/wheel tasks. Other parameter-supporting tasks can consume values through dynamic references.
# in task "validate"; this demo validates the entire table.
# In a batch workflow, filter to the intended run/date before counting.
bad = spark.table("main.silver.orders").filter("amount IS NULL OR amount <= 0").count()
dbutils.jobs.taskValues.set(key="bad_records", value=bad)
# in a downstream notebook task
n = dbutils.jobs.taskValues.get(taskKey="validate", key="bad_records")Prefer mapping {{tasks.validate.values.bad_records}} to a named downstream task parameter, then reading its widget in a notebook. The direct get example illustrates the API without hiding a missing mandatory count behind default=0. Such a default could incorrectly allow publication. debugValue is an optional interactive-development fixture, not a production validation result. Each task value must have a JSON representation no larger than 48 KiB. Task values are intended for small values (counts, file paths, flags, short lists), not for datasets: write large data to a table and pass its name. They exist only within the run; use a table if you need history.
| Mechanism | Scope | Set by | Typical content |
|---|---|---|---|
| Job parameter | Job run; consumption depends on task parameter support | Job definition or run request | Run date, environment, country |
| Task value | One run, downstream tasks | An upstream task at run time | Row count, list of files, status flag |
| Dynamic value reference | Resolved at run time in task settings | The platform | Run id, start date, parameter or task value |
Worked example: designing a conditional load
Scenario. A bank receives a daily file of transactions. Requirements: (1) load the file into bronze and silver; (2) if more than 100 rows fail validation, do not refresh gold, but send an alert and keep the invalid rows for review; otherwise refresh the gold materialized view; (3) in both cases, release temporary resources at the end while retaining durable input for replay; (4) if the load itself fails because of a temporary storage error, try again up to 2 times, 5 minutes apart.
Step 1: tasks. load (notebook, ingestion with retries 2, interval 5 minutes), validate (notebook, writes the task value bad_records), check (If/else, {{tasks.validate.values.bad_records}} > 100), alert_and_quarantine (notebook, depends on check true), refresh_gold (pipeline task, depends on check false), cleanup (notebook).
Step 2: dependencies. validate depends on load. check depends on validate. alert_and_quarantine is attached to the true outcome, refresh_gold to the false outcome.
Step 3: cleanup. cleanup depends on load, validate and both branches. With one branch excluded, All succeeded could continue after the selected branch succeeds, but would block after its failure. Use All done for success/failure cleanup. The direct load dependency also prevents an all-excluded branch chain from excluding cleanup when validation cannot run. This is orchestration eligibility, not guaranteed execution during infrastructure failure or job cancellation. Release disposable resources only: retain raw inputs, quarantine records, checkpoints and committed outputs needed for safe replay.
Step 4: retries. Configure the stated two retries on load, where the transient storage error occurs, and use idempotent ingestion. Give other executable tasks retry policies only when their own failure modes justify them. A retry of check cannot repair a missing validation result.
Why other choices are worse. Putting retries on every task multiplies cost and does not help with logic errors. Using a single notebook with Python if statements hides the branches from the job UI and prevents separate retries and monitoring. At least one succeeded would block cleanup if every executed dependency failed, whereas All done accepts both success and failure after dependencies finish; it does not override the all-excluded rule or guarantee infrastructure availability.
Common pitfalls and exam tips
- Default run-if is All succeeded. A single failed dependency stops everything downstream unless you choose another condition.
- Retries are for transient failures. They do not solve code bugs and each retry consumes compute.
- No dependency permits parallelism. Resource limits may still delay execution. If order matters, add the dependency.
- If/else compares values; it does not run code. The value must be produced by an earlier task (task value) or come from a parameter.
- For each runs ONE nested task repeatedly; concurrency defaults to sequential and nesting another For each is unsupported.
- Task values are for small data, only for the current run. Job parameters are known before the run starts.
- Pipeline tasks use the pipeline's compute, not the job's.
- Use jobs or serverless compute for production, not all-purpose clusters.
- Names. Lakeflow Jobs is the current name of Databricks Workflows.
Exam tip: Translate the requirement into a run-if word. "Even if something failed" points to All done, subject to the all-excluded exception. "Only when something failed" means At least one failed. "Choose between two paths" means an If/else task. "Repeat the same logic for a list" means For each.
Mini case study
Situation. A media company processes viewing logs for eight regions. A nightly job should: (1) ingest each region's logs with the same notebook; (2) after all regions are loaded, update a pipeline that builds the global audience tables; (3) if ingestion fails for one region, still produce the global tables for the other regions but alert the data team; (4) the region list changes monthly and is maintained in a table.
Task. Design the job.
Solution.
- A
get_regionsnotebook task reads the region table and sets a task valueregions(a JSON list). - A For each task
ingest_regionuses{{tasks.get_regions.values.regions}}as input, runs the ingestion notebook withregion={{input}}, concurrency 4, retries 2 for transient errors. A new region is a table row, not a job change. - A pipeline task
build_audiencedepends on the For each task. By default it would not run if any iteration failed. Requirement 3 says to continue with the other regions, so None failed is wrong (a failure exists) and so is the default; use All done so that it runs once all iterations have finished, and require a durable manifest of successfully committed regions for this run. Filter inputs to those run-id/region pairs, exclude stale prior-run data and label missing-region coverage in the published output. If no region succeeds, skip publication and alert. All done merely allows execution; it does not validate partial results. - A notebook task
alert_teamdepends on the For each task with run-if At least one failed.
Without For each, the team would maintain eight copies of a task. Without All done, one bad region would block the whole audience refresh.
Key terms
- Lakeflow Jobs (formerly Workflows): the service that orchestrates tasks, schedules and triggers.
- Task: one unit of work in a job (notebook, SQL, pipeline, condition, loop...).
- DAG: directed acyclic graph of tasks and dependencies.
- Run-if: the rule that decides whether a task runs, based on upstream outcomes.
- Repair run: reruns an unsuccessful subset and dependent tasks in a multi-task job using current settings.
- If/else condition task: evaluates a comparison and selects a true or false branch.
- For each task: runs a nested task for each element of an input list.
- Job parameter: run-level key/value pair consumed through the supported parameter mechanism of each task.
- Task value: small value set by one task and read by downstream tasks in the same run.
- Dynamic value reference:
{{...}}placeholder resolved when the job or task executes; upstream task values become available after their producer runs.
Summary checklist
- I can pick the task type for a notebook, a SQL query, a dashboard refresh and a pipeline update.
- I can draw a task graph and say which tasks run in parallel.
- I know the six run-if options and the default.
- I can say when retries help and when they do not.
- I can use If/else and For each, and I know what they take as inputs.
- I can pass information with job parameters, dynamic values and task values.
After defining the graph, use the Lakeflow Jobs schedules and triggers lesson to choose when the complete flow should start.
Official sources
- Exam guide: https://www.databricks.com/sites/default/files/2026-10/Databricks-Certified-Data-Engineer-Associate-Exam-Guide-August-2026.pdf (opens in a new tab)
- Lakeflow Jobs: https://docs.databricks.com/aws/en/jobs/ (opens in a new tab)
- Configure task dependencies (run-if): https://docs.databricks.com/aws/en/jobs/run-if (opens in a new tab)
- Configure jobs and concurrency: https://docs.databricks.com/aws/en/jobs/configure-job (opens in a new tab)
- Configure task retry settings: https://docs.databricks.com/aws/en/jobs/configure-task (opens in a new tab)
- Dashboard task: https://docs.databricks.com/aws/en/jobs/tasks/dashboard (opens in a new tab)
- For each task: https://docs.databricks.com/aws/en/jobs/tasks/for-each (opens in a new tab)
- Task values: https://docs.databricks.com/aws/en/jobs/task-values (opens in a new tab)
- Job parameters: https://docs.databricks.com/aws/en/jobs/job-parameters (opens in a new tab)
- Repair runs: https://docs.databricks.com/aws/en/jobs/repair-job-failures (opens in a new tab)
- If/else task: https://docs.databricks.com/aws/en/jobs/tasks/if-else (opens in a new tab)
- Dynamic value references: https://docs.databricks.com/aws/en/jobs/dynamic-value-references (opens in a new tab)
