How to Build Production Grade Data Pipelines on Databricks
Technology Posts

How to Build Production Grade Data Pipelines on Databricks

Krunal Kanojiya|July 1, 2026|15 Minute read|Listen
TL;DR
  • Make reruns safe first. Retries, backfills and manual reruns are normal in production, so a load that duplicates rows on a second run is a defect, not an edge case.
  • Separate who changes what. Development, staging and production need separate environments, and production changes should arrive through reviewed, version-controlled deployments run by a service principal.
  • Alert on the data, not only the job. A job can finish green and load nothing. Monitor volume, duration and expectation results as well as failure.
  • Write the recovery steps before go-live. A runbook that any engineer on the team can follow matters more than a clever pipeline nobody else understands.

A pipeline that works in development has proven one thing: the logic is correct for the data you tested. Production adds conditions you did not test. A source sends no rows overnight, a column appears without notice, a retry fires after half the writes landed, or an engineer fixes something in the UI at 11pm and forgets to tell anyone.

This guide covers the operating layer around the transformation code on Databricks: replay-safe loads, environment separation, deployment, monitoring, testing and recovery. It assumes you already build pipelines with Lakeflow pipelines (built on Apache Spark Declarative Pipelines) and Lakeflow Jobs.

The code samples are patterns to adapt, with illustrative table and column names. We have not run them against your workspace, so test them there first.

For the basics behind this article, see Databricks for Data Engineering: Architecture, Components, and Best Practices, Lakeflow Pipelines for Data Engineering and Designing Scalable ETL Pipelines on Databricks.

What separates a production pipeline from a prototype

A prototype succeeds when conditions are normal. A production pipeline defines what happens when they are not, and makes that visible to the team. In practice this comes down to five properties:

PropertyWhat it means in practice
Replay-safe loadsRunning the same input twice leaves the same result
Environment separationChanges pass through dev and staging before production data
Meaningful monitoringAlerts cover volume, duration and data quality, not only job failure
Reviewed deploymentProduction configuration comes from Git, not from UI edits
A recovery runbookWritten steps that someone other than the author can follow

Replay-safe loads: making a second run harmless

A replay-safe (idempotent) load produces the same output whether it runs once or several times on the same input. This matters because Databricks itself retries work. The Lakeflow pipelines documentation says updates started from Jobs or the Pipelines API use automatic retry and restart behavior. Backfills and manual reruns add more repeats.

What breaks replay safety

Three patterns commonly cause duplicate-data incidents:

  • Append-only writes without a key. Rerunning an append over the same source batch writes every row again.
  • Inserts that never check for an existing row. A second run creates a second copy.
  • Windows defined by the clock. A filter such as "the last 24 hours" returns different rows depending on when the job runs.

How to build it

Use MERGE for upserts, but know its limit. The Delta merge documentation states that only a single row from the source can match a given row in the target. If your source batch holds several versions of the same key, the merge can fail. Repeated keys that match nothing in the target can also insert duplicates.

So prepare the source first: keep one row per key, with an explicit ordering and a tie-breaker, and only overwrite a target row with newer data.

-- Pattern to adapt. Table and column names are illustrative.
MERGE INTO silver.customers AS target
USING (
  SELECT * EXCEPT (rn)
  FROM (
    SELECT *,
           ROW_NUMBER() OVER (
             PARTITION BY customer_id
             ORDER BY updated_at DESC, ingest_id DESC
           ) AS rn
    FROM bronze.customers_raw
  )
  WHERE rn = 1
) AS source
ON target.customer_id = source.customer_id
WHEN MATCHED AND source.updated_at >= target.updated_at THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *

MERGE does not make a pipeline replay-safe by itself. The deduplication step, the ordering rule and the condition on the update are what do that. This is our engineering analysis, not a statement from the documentation.

Define windows from the data or from a run parameter, not from the current time:

-- Same result on any day the run is repeated for this date
SELECT * FROM bronze.events WHERE event_date = :run_date

For streaming tables in Lakeflow pipelines, Databricks manages checkpoints for you, so a retry resumes from tracked progress. End-to-end guarantees still depend on the source and the sink. Apache Spark's Structured Streaming guide describes exactly-once semantics as requiring replayable sources and idempotent sinks.

Note also that a full refresh or checkpoint reset changes what gets reprocessed (see the update types), so treat it as a deliberate recovery decision.

If your pipeline loads incremental changes, Incremental Loads, CDC, and Change Data Feed in Delta Lake covers the patterns in more depth.

Environment separation and identity

Running experiments next to production data is the quickest way to corrupt a production table. Consider a scenario (illustrative, not a Lucent project): an engineer tests a transformation, believes the target is a scratch schema, and writes into a production Gold table. Dashboards now mix test rows with real ones, and nobody can say which rows are which.

A common model uses three environments:

EnvironmentPurposeWho deploysTypical data
DevelopmentIndividual testingAny engineerSample or anonymized data, scratch tables
StagingIntegration and validationCI/CD onlyProduction-like volume and schemas, anonymized where it holds personal data
ProductionLive consumersCI/CD onlyGoverned, monitored, service-principal-owned pipelines

Whether you separate these as workspaces, as catalogs, or both depends on your security and compliance needs, so decide that with your platform and security teams. The rule that matters is that no person deploys by hand to staging or production.

Identity is the other half. Databricks documentation recommends service principals instead of users for CI/CD and describes OAuth token federation as a way to avoid storing Databricks secrets in the CI system.

A Lakeflow pipeline also runs updates with its owner's credentials, so setting a service principal as the production owner keeps pipelines from depending on a person's account. Keep tokens and passwords out of repositories, bundle files and logs, and use fictional or anonymized data in non-production environments.

You can check for drift with the audit log system table, system.access.audit. Its reference page marks it as Public Preview as of November 2026 and notes that most audit logs are available only in the workspace's region. Filter it to your production workspace and group the actions by identity.

If you see personal accounts instead of service principals changing production resources, you have a process gap. Check the schema reference for the exact column names and covered events before building a report on it.

Testing and deployment

Test before promotion

Production-grade means a change is checked before it reaches production data. A practical minimum:

  1. Unit-test transformation logic on small, fixed inputs, including duplicates and nulls.
  2. Define expectations on the datasets that matter. Lakeflow pipelines support warn, drop and fail policies for rows that break a rule.
  3. Run the pipeline twice on the same window in staging and compare row counts and key metrics. This is the replay test for the section above.
  4. Check against a baseline, such as duration and output volume from earlier healthy runs.

Deploy through Declarative Automation Bundles

Declarative Automation Bundles (formerly known as Databricks Asset Bundles) define jobs, pipelines and their settings as source files you keep in Git. The Databricks CI/CD page says that "Declarative Automation Bundles are the recommended approach to CI/CD on Databricks." A bundle gives each production configuration a history, a review and a way back to the previous version.

# Pattern to adapt. Run these from CI under a service principal.
databricks bundle validate
databricks bundle deploy --target staging
# run integration checks against staging, then promote
databricks bundle deploy --target prod

databricks bundle validate catches configuration errors before anything is deployed. The bundles overview describes validate, deploy and run as the core CLI workflow.

Do not edit production resources in the UI and update the YAML later. The next deployment can overwrite the manual change, and then the bundle and the workspace disagree about the truth.

Rollback is two different problems

Reverting a bundle to the previous commit restores the code and configuration. It does not undo rows the bad run already wrote. For the data, decide ahead of time how you will recover: reprocess from Bronze, restore a table version, or repair a downstream table. Confirm that your Delta table history and retention settings actually allow what you plan to do, because they are not an automatic backup.

Monitoring that catches problems before analysts do

Most teams alert on job failure. That is the last signal to fire, and some failures never trigger it. Three layers catch more, and the thresholds below are examples to tune, not universal rules.

Layer 1: volume and freshness with SQL alerts

A load can finish successfully and write nothing. Databricks SQL alerts run a query on a schedule, evaluate a condition and notify recipients; the latest version became generally available on 14 May 2026. A volume alert for a Silver table:

-- Alert condition: rows_loaded equals 0. Match the window to your expected load cadence.
SELECT COUNT(*) AS rows_loaded
FROM silver.events
WHERE ingestion_timestamp >= current_timestamp() - INTERVAL 1 HOUR

Apply this to the Silver tables that feed dashboards or models. Set the window to the cadence the source actually delivers, since a table fed once a day should not alert hourly.

Layer 2: duration warnings on jobs

Databricks job notifications can warn when a run exceeds a duration threshold, and you must set the expected duration to use it. A starting point is to warn at about twice a healthy run's duration, then adjust after you have a few weeks of run history. The warning arrives while the job is still running, which leaves time to investigate.

Layer 3: expectation results from the event log

Job status and duration show operational trouble. Expectation results show data-quality trouble. Expectation metrics are stored in the pipeline event log, in details:flow_progress.data_quality.expectations, with passed_records and failed_records per expectation (event log reference). Note that if an expectation is set to fail the update when it finds an invalid record, metrics are not recorded.

The query below follows the pattern in the Databricks documentation: it parses the expectations array, explodes it and totals the results for the latest update. Replace YOUR_PIPELINE_ID with your pipeline's ID. By default only the pipeline's run-as user can query the event log, and Databricks recommends creating a view before sharing access.

-- Adapted from the Databricks event log documentation. Not run against a live pipeline.
CREATE VIEW event_log_raw AS SELECT * FROM event_log('YOUR_PIPELINE_ID');

WITH latest_update AS (
  SELECT origin.pipeline_id, origin.update_id AS latest_update_id
  FROM event_log_raw AS origin
  WHERE origin.event_type = 'create_update'
  ORDER BY timestamp DESC
  LIMIT 1
)
SELECT
  row_expectations.dataset AS dataset,
  row_expectations.name AS expectation,
  SUM(row_expectations.passed_records) AS passing_records,
  SUM(row_expectations.failed_records) AS failing_records
FROM (
  SELECT explode(
    from_json(
      details:flow_progress:data_quality:expectations,
      "array<struct<name: string, dataset: string, passed_records: int, failed_records: int>>"
    )
  ) AS row_expectations
  FROM event_log_raw, latest_update
  WHERE event_type = 'flow_progress'
    AND origin.update_id = latest_update.latest_update_id
)
GROUP BY row_expectations.dataset, row_expectations.name

To see a trend, store each update's results in a table and chart the failure rate over time. A pass rate that falls from week to week usually points to a change upstream, so investigate it before it becomes an incident. Databricks also notes that the pipeline_events system table, currently in Beta, collects events across pipelines in a region, which may suit fleet-wide monitoring.

For deeper design of the expectations themselves, read Data Quality and Reliability Patterns for Databricks Pipelines.

The runbook every production pipeline needs

A runbook is a written procedure that tells anyone on the team what to do when a specific pipeline fails. The Azure Well-Architected guidance for Databricks says operational runbooks should include diagnostic commands, log locations, escalation contacts and recovery procedures (Azure-specific guidance, but the contents apply on any cloud).

Cover four things for each production pipeline:

  1. Purpose and dependents. Which sources it reads, which tables it writes, which dashboards or models depend on it, and who the business owner is.
  2. How to find the failure. Where the job lives, what a normal run looks like, and which event log query shows the failing step.
  3. Recovery steps for the common failure modes. The table below is a starting point.
  4. Escalation contacts. The source system owner, the data engineering on-call and the stakeholder who owns the service level.
Failure modeFirst checkRecovery action
No rows or late source dataSource delivery, Auto Loader checkpoint, storage permissionsConfirm the source with its owner, then rerun the affected window
Schema change or mismatchNew or changed columns in the source and the Delta tableDecide whether to accept the change (for example by enabling schema merge), update downstream views, rerun
Partial write or duplicatesRow counts per key and run, the failed task in the jobDeduplicate or rebuild the affected window from Bronze, then rerun with the replay-safe logic
Expectation failures above thresholdEvent log: which expectation, which dataset, which rowsQuarantine the bad batch, notify the source owner, replay once corrected

Keep the runbook in the same Git repository as the pipeline code and update it when the pipeline changes. A page in a wiki that nobody revisits after the first month becomes documentation debt.

Production readiness checklist

Before a pipeline goes live in production, confirm each item:

  • Replay safety: the source batch is deduplicated before MERGE, windows come from data or parameters, and a double run in staging produced matching counts.
  • Isolation and identity: separate dev, staging and production environments exist, and a service principal owns production jobs and pipelines.
  • Deployment: a bundle defines all job and pipeline configuration, databricks bundle validate runs in CI, and the production workspace has no manual UI changes in the workflow.
  • Monitoring: a volume alert covers each Silver table with a production consumer, a duration warning is set on each production job, and expectation results are reviewed on a schedule.
  • Recovery: the runbook is in Git, covers the failure modes above, and a second engineer has reviewed it.

Build it in-house or bring in help?

The checklist is achievable for most teams, but it takes engineering time away from feature work. If you are weighing that trade-off, When Should You Hire Data Engineers Instead of Building Slowly In-House lays out the decision.

Lucent Innovation provides Databricks data engineering services, so we have a commercial interest in this topic. If you want a second opinion on a pipeline, bring the source systems, the consumers that depend on it and the failure you are trying to prevent, using anonymized details. You can read about our data engineering with Databricks services to see how we work.

Where to go next in this series

SHARE

Krunal Kanojiya
Krunal K.
Technical Content Writer

Facing a Challenge? Let's Talk.

Whether it's AI, data engineering, or commerce tell us what's not working yet. Our team will respond within 1 business day.

Start the Conversation

Frequently Asked Questions

Still have Questions?

Let’s Talk

Share your requirements

(+1)

Our Global Footprint

With offices in the USA and India, we partner with businesses around the world.

Lucent Innovation

Engineering Partners. Not Vendors.

Certified Databricks Partner & Shopify Plus Agency delivering production-grade data, AI, and Commerce solutions since 2013.

Follow Us

Get in Touch
Databricks PartnerShopify Plus PartnerISO Certified

Lucent Innovation, © 2026. All rights reserved.