Skip to content

Lakeflow Jobs Schedules, File Arrival and Table Update Triggers

Step 19 of 30

Lakeflow Jobs Schedules, File Arrival and Table Update Triggers

Introduction

This lesson finishes the Working with Lakeflow Jobs section of the official exam outline (about 16% of the scored questions). The Lakeflow Jobs orchestration lesson built the task graph. Here you choose Lakeflow Jobs schedules and triggers for the Databricks Data Engineer Associate exam. A job does nothing until something starts it. That something can be a clock (a schedule), an event in your data platform (new files, an updated table), a never-ending loop (continuous mode), or a person or external tool (a manual or API run). Choosing the right start condition is one of the most tested decisions in this section, because it affects freshness, cost and reliability.

After this lesson you will be able to configure a time-based schedule with cron syntax and time zones, describe how file arrival and table update triggers work, explain what a continuous job is, and choose between time-based and data-driven triggers for a given requirement.

Learning objectives

  • Configure a scheduled trigger (simple or cron) and explain the role of the time zone and the pause status.
  • Explain how a file arrival trigger works and which storage it needs.
  • Explain how a table update trigger works and what it watches.
  • Describe a continuous job and when it is appropriate.
  • Tune a trigger with minimum time between triggers and wait after last change.
  • Choose between time-based and data-driven triggers, and qualify Multiple Triggers Beta or a dispatcher for resilience.

1. The trigger types at a glance

A job can have triggers. When an active trigger fires, it starts a run. With the Multiple Triggers Beta enabled by a workspace administrator, a job can have up to five triggers. Each active trigger can independently request a run; one firing does not cancel the others or guarantee a single run across simultaneous events. A continuous trigger cannot be combined with another trigger. Without that Beta, use one trigger and an external/API or dispatcher mechanism for additional start conditions. In Jobs API requests supporting the Beta, use the triggers array without the legacy schedule, trigger or continuous fields; updating that array replaces the complete trigger list.

Trigger Starts a run when... Best for
Scheduled the clock matches a schedule (periodic or cron) Reports at fixed times, nightly batches, SLAs expressed in time
File arrival new files appear in a monitored Unity Catalog volume or external location Irregular file drops from partners or upstream systems
Table update one or more watched Unity Catalog tables are updated Downstream jobs that must follow an upstream table
Continuous the previous run completes or fails, with scheduler delay/backoff and at most one active run Near real-time, low-latency streaming jobs
Manual / API a person, a CLI, a CI/CD pipeline or an orchestrator starts it Ad hoc runs, backfills, tests

All of them start the same job definition. The trigger is separate from the tasks.

2. Scheduled (cron) triggers

A scheduled trigger runs the job on a recurring schedule. The simple schedule specifies an interval and unit; its first start time is selected by the scheduler. Use the advanced schedule for an explicit local time, weekday pattern, timezone or cron expression. Databricks uses Quartz cron syntax, with a seconds field.

Text
0 0 2 * * ?          -> every day at 02:00:00
0 30 6 ? * MON-FRI   -> weekdays at 06:30:00
0 0/15 * * * ?       -> every 15 minutes
0 0 8 1 * ?          -> the first day of each month at 08:00

The six (or seven) fields are: seconds, minutes, hours, day of month, month, day of week, optional year. Quartz requires ? in either the day-of-month or the day-of-week field when the other one is specified, which is the most common syntax error.

Important settings:

  • Time zone. A schedule is interpreted in the time zone you choose. Always select it explicitly: a "02:00" schedule in UTC is not the same moment as 02:00 in Paris. Daylight saving time can shift the local time of a run when the zone observes it, and a local time that does not exist (the spring clock change) or occurs twice (autumn) is handled by the platform in a way you should not rely on: for strict hourly cadence or heavy jobs, prefer UTC.
  • Pause status. A schedule can be paused without deleting it. A paused schedule does not fire; unpausing resumes future runs, and missed runs are not replayed.
  • Overlapping runs. If a run is still active when the next schedule fires, behavior depends on the job's maximum concurrent runs (default 1). Without queueing, a request that exceeds the limit can be skipped. With queueing enabled, eligible runs wait for capacity; UI-created jobs enable queueing by default. Inspect both settings and monitor queued/skipped runs rather than assuming either a guaranteed overlap or guaranteed execution.

The scheduler enforces a minimum interval of 10 seconds and is not a low-latency or exact-time guarantee. Cloud or network issues can delay starts by minutes. Set a freshness check and deadline alert in addition to the schedule.

Concrete example. Northwind Logistics must publish a delivery-performance dashboard by 07:00 Paris time on weekdays. The nightly job takes about 90 minutes. The team schedules it with 0 0 5 ? * MON-FRI in the Europe/Paris time zone, leaving a margin of 30 minutes beyond the typical duration.

A time-based schedule is predictable and simple, but it is blind to data. If the upstream file arrives late, the job processes nothing new or, worse, processes yesterday's data and publishes it as today's.

3. File arrival triggers

A file arrival trigger starts the job when new files land in a monitored location. The location must be governed by Unity Catalog: a volume or an external location (the root or a subpath). The trigger checks the location frequently and fires when it sees new files. Your job does not need to know which files caused the run: an ingestion task (for example an Auto Loader task with an availableNow trigger) discovers and processes the new files by itself with its checkpoint. Do not assume the trigger passes a complete list of arriving file paths. The ingestion reader and durable checkpoint establish processing progress; a run trigger is not a file manifest.

Two advanced settings avoid starting too many runs:

Setting Meaning Use when
Minimum time between triggers A cooldown measured after the preceding run completes before another trigger run can start Files arrive constantly and one run per few minutes is enough
Wait after last change A debounce: wait until no new file has appeared for this time before starting Files arrive in bursts and you want the whole batch in one run

The file-arrival API documents a minimum of 60 seconds for these configured intervals. A five-minute quiet period groups an isolated burst, but actual detection is best effort (typically checked about once a minute), and the cooldown, concurrency and queue can delay the start further. Continuous arrivals can keep resetting the quiet timer. A debounce is not proof that a complete business batch has arrived: validate an upstream manifest or completion marker when completeness matters.

Prefer managed file events on the external location. Without file events, the monitored location is limited to 10,000 files and the workspace to 50 jobs with file-arrival triggers on such locations; file-event-enabled locations have different capacity rules. New files are the intended signal, not reliable notifications for every overwrite. Use immutable file names: changes outside retained metadata can behave differently.

Design advice from the documentation: when you monitor a subpath of a large external location, create a volume mapped to the target directory and put the trigger on the volume root. This limits the amount of file metadata that must be evaluated, and avoids an error state in busy locations.

Concrete example. A retailer's partners drop CSV files into /Volumes/main/landing/partner_orders/ at unpredictable times, from 10 to 400 files per day. A file arrival trigger starts the ingestion job with wait after last change of 2 minutes. The job's first task is a notebook running Auto Loader with availableNow, so normal runs with the same intact checkpoint process newly discovered input. Repeated business records in distinct files still require business-key deduplication. See Auto Loader checkpoints and file discovery.

4. Table update triggers

A table update trigger starts the job when one or more watched tables are updated. It is the natural way to chain jobs through data: the job that builds silver.orders finishes, and the job that builds the gold views starts without any schedule and without a direct link between the two.

Key facts:

  • Supported objects include Unity Catalog managed Delta/Iceberg tables, external Delta tables, materialized views and streaming tables, plus eligible views over supported tables. A trigger can monitor up to 25 underlying tables if file events are enabled on every monitored table; otherwise the limit is 10. Source tables of views count toward that limit, separately for each monitored view.
  • You choose the condition: start when any of the tables is updated, or only when all watched inputs have registered an update. This is a change-detection condition, not an atomic multi-table transaction or proof of a matching business batch.
  • The same two settings exist: minimum time between triggers and wait after last change.
  • Dynamic value references let tasks know which tables triggered the run.
  • It reacts to supported table changes, including updates, merges and deletes, not to a particular row. The Run as identity needs SELECT and the relevant USE CATALOG/USE SCHEMA privileges. If evaluation lacks access, the trigger cannot start the job.

Concrete example. The gold job build_sales_marts needs both silver.orders and silver.customers to be fresh. A table update trigger with the condition all tables updated on both tables means the job only starts after both have received new data, without guessing the arrival times. It does not prove that both updates belong to the same reporting date or that an entire upstream pipeline succeeded. Validate the batch identifiers and freshness before publishing. A cron schedule at 03:00 would start at 03:00 even if silver.customers is still being rebuilt.

5. Continuous jobs

A continuous job keeps restarting work with at most one active run. After a run ends, scheduling the next takes time; the documentation describes a gap normally under 60 seconds, with longer failure backoff. If a run fails, the platform restarts the job automatically with an increasing delay (exponential backoff) so that a persistent error does not create a tight failure loop. Practical consequences:

  • Only one active run at a time: the maximum number of concurrent runs is effectively 1.
  • A continuous job cannot also have a time-based schedule; the whole idea is that the job is always running.
  • It suits always-on streaming workloads, but is distinct from a Spark streaming trigger. Serverless notebook/jobs use bounded triggers such as AvailableNow; time-based ProcessingTime and Spark Continuous triggers are unsupported there. Use a continuous pipeline where supported for low-latency serverless processing.
  • Repeated runs or long-lived processing can keep compute busy for much of the day; startup gaps and compute lifecycle mean this is not a literal guarantee of one continuously occupied cluster. For a stream that has a data arrival gap every night, a triggered job with a data-driven trigger may cost much less.
  • Pausing prevents new trigger runs; active runs continue unless explicitly stopped. Continuous jobs do not support task dependencies or the usual finite retry policy. Their task-retry mode and job restart/backoff govern recovery.

A restart or checkpoint alone does not make arbitrary external side effects idempotent. For pipelines the same distinction exists: a triggered pipeline update runs and stops; a continuous pipeline keeps running. Current Databricks guidance recommends orchestrating a continuous pipeline with a continuous job; the job then manages the pipeline execution lifecycle. That is separate from a notebook's Structured Streaming trigger.

6. Choosing between time-based and data-driven triggers

Requirement Best choice Why
"Report must be ready every weekday by 07:00" Scheduled The requirement is stated in time
"Process partner files as soon as they arrive; arrival is irregular" File arrival Runs follow data, no wasted runs
"Rebuild marts when silver tables change" Table update Chains jobs without guessing durations
"Always-on streaming with measured low-latency needs" Continuous job or supported continuous pipeline Choose the compatible compute and streaming mode, then measure freshness
"Re-run last week's data after a bug fix" Manual / API with parameters One-off backfill
"Normally follows data, but must run at least daily" Data trigger plus scheduled backup with Multiple Triggers Beta, or a scheduled dispatcher Additional execution opportunity, with monitoring and idempotency

Rules of thumb:

  1. If the business promise is a time, use a schedule. If it is freshness after data arrives, use a data-driven trigger.
  2. Prefer data-driven triggers over "sleep until the data should be there" schedules: they remove guesswork and idle runs.
  3. Prefer triggered runs (schedule, file arrival, table update) over continuous mode unless low latency is a real requirement, because continuous mode costs compute around the clock.
  4. When Multiple Triggers Beta is enabled, combine an event trigger with a scheduled backup. Otherwise, let a separate scheduled dispatcher invoke the same worker job through Run Job/API. Keep transformation logic in one place, make writes idempotent and monitor missed or duplicate requests. A scheduled attempt at 06:00 is not a guarantee that complete data will be published then.

Worked example: choosing the triggers for three jobs

Scenario. A solar-panel manufacturer has three jobs. Job 1 ingests quality-control CSV files that three factories upload irregularly, sometimes 50 files in two minutes and sometimes nothing for a day. Job 2 builds the gold.line_efficiency mart from silver.qc_results and silver.shifts; both must be updated first. Job 3 emails a management summary at 08:00 Berlin time every Monday.

Step 1: Job 1. Irregular file drops in a governed location: a file arrival trigger on the landing volume. Files come in bursts, so set wait after last change to 3 minutes: one run per burst. The ingestion notebook uses Auto Loader with availableNow, so the checkpoint decides which files are new.

Step 2: Job 2. The input is two tables. A table update trigger on both tables with the condition "all tables updated". The trigger waits for changes to both inputs; the job also checks that their batch identifiers match. If Multiple Triggers Beta is enabled, add a 06:00 scheduled backup. Otherwise use a scheduled dispatcher invoking the same worker. A backup without fresh data must alert or report stale input, not manufacture a successful refresh.

Step 3: Job 3. A promise stated as a time and day: a scheduled trigger with 0 0 8 ? * MON and the Europe/Berlin time zone.

Why the alternatives are worse. A cron schedule for Job 1 either runs when there is nothing to do or waits too long. A continuous job for Job 1 would keep compute running during quiet days. A schedule for Job 2 ignores whether the silver tables are ready. A table update trigger for Job 3 has no relation to the Monday promise.

Common pitfalls and exam tips

  • Cron needs a time zone. Without choosing it, the result may not be the local time you expect.
  • Quartz syntax. Six fields starting with seconds; ? must appear in day-of-month or day-of-week.
  • Pausing is not deleting. A paused trigger does not fire and does not replay missed runs.
  • File arrival needs Unity Catalog storage (volume or external location), and the job still needs an ingestion mechanism such as Auto Loader that knows which files are new.
  • Minimum time between triggers is a cooldown; wait after last change is a debounce. Do not mix them up.
  • Table update triggers watch tables, not rows, with an any/all condition; 10 underlying tables normally, up to 25 when every monitored table has file events enabled.
  • Continuous means always on. It costs compute continuously and cannot be combined with a schedule.
  • Multiple triggers are Beta. Enable the feature before combining up to five triggers; a continuous trigger remains exclusive.

Exam tip: Underline the trigger words in the requirement. "At 6:00 every day" is scheduled. "When files land" is file arrival. "When the upstream table is refreshed" is table update. "Within seconds, always on" is continuous. A requirement for both event-driven and daily execution needs qualified Multiple Triggers support or an external dispatcher. Each trigger can request a run; they do not form an exclusive first-wins race.

Mini case study

Situation. A hospital group receives lab-result files from 12 laboratories at random times. The data engineering team currently runs a job every 10 minutes with a cron schedule. About 80% of the runs find no new file but still start a cluster, cost money and show up as noise in the monitoring dashboard. A downstream job refreshes a clinical-quality dashboard and normally follows changes to the lab results table, with a daily 05:00 UTC freshness audit even when no new data has arrived. A timestamp must reflect the last successful data update rather than hide a stale clinical dataset.

Task. Redesign the triggers.

Solution.

  1. Replace the 10-minute cron on the ingestion job with a file arrival trigger on the landing volume, with wait after last change of 2 minutes to group files of the same burst. The ingestion task keeps using Auto Loader with availableNow, with a durable checkpoint. This reduces scheduled empty runs; preserve validation and monitor trigger evaluation failures.
  2. For the dashboard job, add a table update trigger on silver.lab_results (condition: any table updated) with minimum time between triggers of 15 minutes so that the dashboard is not rebuilt for every micro-update.
  3. With Multiple Triggers Beta enabled, add a scheduled trigger 0 0 5 * * ? in UTC as the daily audit. Otherwise schedule a separate dispatcher that invokes the same worker. Check freshness in the worker, and alert when no new results are available. Monitor capacity and failures; neither a cron nor a second trigger guarantees execution or fresh source data.
  4. Do not use continuous mode: no requirement asks for seconds of latency, and it would cost compute all day.

Key terms

  • Trigger: the condition that starts a job run (schedule, file arrival, table update, continuous, or manual).
  • Quartz cron: cron dialect used by Databricks with a leading seconds field.
  • File arrival trigger: starts a run when new files appear in a Unity Catalog volume or external location.
  • Table update trigger: starts a run when watched Unity Catalog tables are updated.
  • Minimum time between triggers: cooldown limiting the frequency of runs.
  • Wait after last change: debounce delaying the run until activity pauses.
  • Continuous job: job restarted after completion/failure with at most one active run, scheduler gaps and failure backoff.
  • Pause status: setting that disables a trigger without deleting it.

Summary checklist

  • I can read and write a simple Quartz cron expression and set the time zone.
  • I know file arrival triggers need a Unity Catalog volume or external location.
  • I know the table-count limit is 10, or 25 with file events on all underlying monitored tables, and understand any/all conditions.
  • I can tell a cooldown from a debounce.
  • I can explain what a continuous job is and what it costs.
  • I can choose triggers, qualify Multiple Triggers Beta and monitor freshness and missed executions.

For late or missing runs, use job monitoring and troubleshooting to distinguish source delays, trigger evaluation errors and queued compute.

Official sources