Scheduling and Triggering Lakeflow Jobs

Introduction

This lesson finishes the Working with Lakeflow Jobs section of the official exam outline (about 16% of the scored questions). In the previous lesson you built a job as a graph of tasks. A job does nothing until something starts it. That something can be a clock (a schedule), an event in your data platform (new files, an updated table), a never-ending loop (continuous mode), or a person or external tool (a manual or API run). Choosing the right start condition is one of the most tested decisions in this section, because it affects freshness, cost and reliability.

After this lesson you will be able to configure a time-based schedule with cron syntax and time zones, describe how file arrival and table update triggers work, explain what a continuous job is, and choose between time-based and data-driven triggers for a given requirement.

Learning objectives

• Configure a scheduled trigger (simple or cron) and explain the role of the time zone and the pause status.
• Explain how a file arrival trigger works and which storage it needs.
• Explain how a table update trigger works and what it watches.
• Describe a continuous job and when it is appropriate.
• Tune a trigger with minimum time between triggers and wait after last change.
• Choose between time-based and data-driven triggers, and combine them for resilience.

1. The trigger types at a glance

A job can have triggers. When an active trigger fires, it starts a run. A job can have several triggers; whichever fires first starts a run.

All of them start the same job definition. The trigger is separate from the tasks.

2. Scheduled (cron) triggers

A scheduled trigger runs the job on a recurring schedule. In the UI you can use a simple form (every day at 02:00, every hour) or enter a cron expression. Databricks uses Quartz cron syntax, with a seconds field.

0 0 2 * * ?          -> every day at 02:00:00
0 30 6 ? * MON-FRI   -> weekdays at 06:30:00
0 0/15 * * * ?       -> every 15 minutes
0 0 8 1 * ?          -> the first day of each month at 08:00

The six (or seven) fields are: seconds, minutes, hours, day of month, month, day of week, optional year. Quartz requires ? in either the day-of-month or the day-of-week field when the other one is specified, which is the most common syntax error.

Important settings:

• Time zone. A schedule is interpreted in the time zone you choose. Always select it explicitly: a "02:00" schedule in UTC is not the same moment as 02:00 in Paris. Daylight saving time can shift the local time of a run when the zone observes it, and a local time that does not exist (the spring clock change) or occurs twice (autumn) is handled by the platform in a way you should not rely on: for strict hourly cadence or heavy jobs, prefer UTC.
• Pause status. A schedule can be paused without deleting it. A paused schedule does not fire; unpausing resumes future runs, and missed runs are not replayed.
• Overlapping runs. If a run is still active when the next schedule fires, behavior depends on the job's maximum concurrent runs (default 1). With the default, the new run is skipped rather than overlapping.

Concrete example. Northwind Logistics must publish a delivery-performance dashboard by 07:00 Paris time on weekdays. The nightly job takes about 90 minutes. The team schedules it with 0 0 5 ? * MON-FRI in the Europe/Paris time zone, leaving a margin of 30 minutes beyond the typical duration.

A time-based schedule is predictable and simple, but it is blind to data. If the upstream file arrives late, the job processes nothing new or, worse, processes yesterday's data and publishes it as today's.

3. File arrival triggers

A file arrival trigger starts the job when new files land in a monitored location. The location must be governed by Unity Catalog: a volume or an external location (the root or a subpath). The trigger checks the location frequently and fires when it sees new files. Your job does not need to know which files caused the run: an ingestion task (for example an Auto Loader task with an availableNow trigger) discovers and processes the new files by itself with its checkpoint. The trigger also exposes information about the event as dynamic value references if a task wants to read it.

Two advanced settings avoid starting too many runs:

Both have a minimum value of 60 seconds in the API. With a wait of 5 minutes, a burst of 200 files over 3 minutes produces a single run that starts 5 minutes after the last file.

Design advice from the documentation: when you monitor a subpath of a large external location, create a volume mapped to the target directory and put the trigger on the volume root. This limits the amount of file metadata that must be evaluated, and avoids an error state in busy locations.

Concrete example. A retailer's partners drop CSV files into /Volumes/main/landing/partner_orders/ at unpredictable times, from 10 to 400 files per day. A file arrival trigger starts the ingestion job with wait after last change of 2 minutes. The job's first task is a notebook running Auto Loader with availableNow, so each run processes only files that have not been seen yet.

4. Table update triggers

A table update trigger starts the job when one or more watched tables are updated. It is the natural way to chain jobs through data: the job that builds silver.orders finishes, and the job that builds the gold views starts without any schedule and without a direct link between the two.

Key facts:

• The watched objects are Unity Catalog managed or Delta tables (up to 10 tables per trigger).
• You choose the condition: start when any of the tables is updated, or only when all of them have been updated since the last run.
• The same two settings exist: minimum time between triggers and wait after last change.
• Dynamic value references let tasks know which tables triggered the run.
• It reacts to the table being updated, not to a particular row.

Concrete example. The gold job build_sales_marts needs both silver.orders and silver.customers to be fresh. A table update trigger with the condition all tables updated on both tables means the job only starts after both have received new data, however late either one is, and no earlier. A cron schedule at 03:00 would start at 03:00 even if silver.customers is still being rebuilt.

5. Continuous jobs

A continuous job keeps exactly one run active. As soon as a run completes (or fails), Databricks starts a new one. If a run fails, the platform restarts the job automatically with an increasing delay (exponential backoff) so that a persistent error does not create a tight failure loop. Practical consequences:

• Only one active run at a time: the maximum number of concurrent runs is effectively 1.
• A continuous job cannot also have a time-based schedule; the whole idea is that the job is always running.
• It is the pattern for streaming workloads that need low latency, such as a long-lived Structured Streaming query or a pipeline in continuous mode.
• It consumes compute all the time. For a stream that has a data arrival gap every night, a triggered job with a data-driven trigger may cost much less.
• You can pause a continuous job to stop new runs, for instance during maintenance.

For pipelines the same distinction exists: a triggered pipeline update runs and stops; a continuous pipeline keeps running. When a continuous job contains a pipeline task, the job's setting determines the execution mode of the pipeline.

6. Choosing between time-based and data-driven triggers

Rules of thumb:

1. If the business promise is a time, use a schedule. If it is freshness after data arrives, use a data-driven trigger.
2. Prefer data-driven triggers over "sleep until the data should be there" schedules: they remove guesswork and idle runs.
3. Prefer triggered runs (schedule, file arrival, table update) over continuous mode unless low latency is a real requirement, because continuous mode costs compute around the clock.
4. Combine triggers when needed. A table update trigger plus a schedule gives "run when data is ready, and in any case at 06:00".

Worked example: choosing the triggers for three jobs

Scenario. A solar-panel manufacturer has three jobs. Job 1 ingests quality-control CSV files that three factories upload irregularly, sometimes 50 files in two minutes and sometimes nothing for a day. Job 2 builds the gold.line_efficiency mart from silver.qc_results and silver.shifts; both must be updated first. Job 3 emails a management summary at 08:00 Berlin time every Monday.

Step 1: Job 1. Irregular file drops in a governed location: a file arrival trigger on the landing volume. Files come in bursts, so set wait after last change to 3 minutes: one run per burst. The ingestion notebook uses Auto Loader with availableNow, so the checkpoint decides which files are new.

Step 2: Job 2. The input is two tables. A table update trigger on both tables with the condition "all tables updated". The job starts only when both have new data. To guard against a day when no update happens at all, add a scheduled trigger at 06:00 as a safety net, and make the job idempotent.

Step 3: Job 3. A promise stated as a time and day: a scheduled trigger with 0 0 8 ? * MON and the Europe/Berlin time zone.

Why the alternatives are worse. A cron schedule for Job 1 either runs when there is nothing to do or waits too long. A continuous job for Job 1 would keep compute running during quiet days. A schedule for Job 2 ignores whether the silver tables are ready. A table update trigger for Job 3 has no relation to the Monday promise.

Common pitfalls and exam tips

• Cron needs a time zone. Without choosing it, the result may not be the local time you expect.
• Quartz syntax. Six fields starting with seconds; ? must appear in day-of-month or day-of-week.
• Pausing is not deleting. A paused trigger does not fire and does not replay missed runs.
• File arrival needs Unity Catalog storage (volume or external location), and the job still needs an ingestion mechanism such as Auto Loader that knows which files are new.
• Minimum time between triggers is a cooldown; wait after last change is a debounce. Do not mix them up.
• Table update triggers watch tables, not rows, up to 10 tables per trigger, with an any/all condition.
• Continuous means always on. It costs compute continuously and cannot be combined with a schedule.
• Combine triggers for resilience: data-driven first, schedule as a backup.

Exam tip: Underline the trigger words in the requirement. "At 6:00 every day" is scheduled. "When files land" is file arrival. "When the upstream table is refreshed" is table update. "Within seconds, always on" is continuous. "Whichever happens first" suggests several triggers on the same job.

Mini case study

Situation. A hospital group receives lab-result files from 12 laboratories at random times. The data engineering team currently runs a job every 10 minutes with a cron schedule. About 80% of the runs find no new file but still start a cluster, cost money and show up as noise in the monitoring dashboard. A downstream job refreshes a clinical-quality dashboard and must run only after the lab results table has changed, and also at least once per day at 05:00 UTC to guarantee that the dashboard shows a refreshed timestamp.

Task. Redesign the triggers.

Solution.

1. Replace the 10-minute cron on the ingestion job with a file arrival trigger on the landing volume, with wait after last change of 2 minutes to group files of the same burst. The ingestion task keeps using Auto Loader with availableNow, so correctness is unchanged and empty runs disappear.
2. For the dashboard job, add a table update trigger on silver.lab_results (condition: any table updated) with minimum time between triggers of 15 minutes so that the dashboard is not rebuilt for every micro-update.
3. Add a scheduled trigger 0 0 5 * * ? in UTC to the same dashboard job as the daily guarantee. A job can carry both triggers; the first one that fires starts a run.
4. Do not use continuous mode: no requirement asks for seconds of latency, and it would cost compute all day.

Key terms

• Trigger: the condition that starts a job run (schedule, file arrival, table update, continuous, or manual).
• Quartz cron: cron dialect used by Databricks with a leading seconds field.
• File arrival trigger: starts a run when new files appear in a Unity Catalog volume or external location.
• Table update trigger: starts a run when watched Unity Catalog tables are updated.
• Minimum time between triggers: cooldown limiting the frequency of runs.
• Wait after last change: debounce delaying the run until activity pauses.
• Continuous job: job that always has one run active and restarts after each end.
• Pause status: setting that disables a trigger without deleting it.

Summary checklist

• I can read and write a simple Quartz cron expression and set the time zone.
• I know file arrival triggers need a Unity Catalog volume or external location.
• I know a table update trigger can watch up to 10 tables with an any/all condition.
• I can tell a cooldown from a debounce.
• I can explain what a continuous job is and what it costs.
• I can choose between time-based and data-driven triggers, and combine them.

Official sources

• Exam guide: https://www.databricks.com/sites/default/files/2026-05/databricks-certified-data-engineer-associate-exam-guide-may-2026.pdf (opens in a new tab)
• Automate jobs with schedules and triggers: https://docs.databricks.com/aws/en/jobs/triggers (opens in a new tab)
• Trigger jobs when new files arrive: https://docs.databricks.com/aws/en/jobs/file-arrival-triggers (opens in a new tab)
• Trigger jobs when source tables are updated: https://docs.databricks.com/aws/en/jobs/trigger-table-update (opens in a new tab)