Data Quality and Reliability Patterns for Databricks Pipelines

It was a normal Tuesday morning.

A data analyst opened the weekly revenue dashboard before a 9 am meeting with the sales team. The numbers were supposed to be ready. They were. But something looked wrong.

Revenue was down 7% from the week before. No promotion had ended. No outage had been reported. The analyst refreshed the page. Same number. She flagged it with the data engineering team.

The engineer on call pulled up the Databricks workspace. The pipeline had run at 2am like it always did. Green checkmark. No errors. No failed tasks. The output table had exactly the right number of rows. Everything the platform was telling them said the job was fine.

One quick query changed that.

Seven percent of the customer_id values in the Silver table were null. Every revenue calculation tied to those rows returned zero. The pipeline had not crashed. It had not raised a single alert. It had quietly ingested broken data, transformed it correctly, and served it upstream. The job was healthy. The data was not.

That 9 am meeting did not go the way anyone planned.

This is how most data quality problems actually look. Not a pipeline crash with a red alert at 3am. Just a job that runs cleanly on bad data, and a team that finds out about it when a business user asks why the numbers do not add up.

Expectations are what prevent this. They are quality rules written in SQL, applied to every row at the exact moment it gets written to a Delta table. No separate validation job to schedule. No post-load checking to maintain. The rule runs at write time, inside the pipeline itself, before broken data has a chance to reach anyone downstream.

In this guide, we covers how to write expectations, when to use each of the three action modes, how to build a quarantine pattern so bad data is captured not just deleted, and what changed in 2026 that lets you store quality rules in Unity Catalog instead of rewriting them inside every pipeline.

This article builds on Lakeflow Pipelines for Data Engineering and How to Build Production-Grade Data Pipelines on Databricks. The full series starts at Modern Data Engineering: The Complete Guide.

Why Pipelines Succeed While Data Quality Fails

Most engineers monitor jobs. They set up alerts for job failures, duration thresholds, and infrastructure errors. Those alerts tell you the pipeline broke.

They tell you nothing about whether the data is correct.

As the Databricks Community blog on data quality puts it directly: pipelines often run successfully even when data quality is poor. A dataset may have missing values, duplicate records, incorrect formats, or unexpected spikes. The pipeline completes, but the output becomes unreliable. Downstream users trust the system because it is running, but the insights generated may be incorrect.

The window between bad data arriving and someone noticing it is the most expensive gap in a data platform. As DZone's analysis of Delta expectations explains: the traditional approach treats validation as a post-processing step. In high-throughput lakehouses processing terabytes daily, that window can represent millions of corrupted records propagating downstream before anyone notices.

Expectations collapse that window to zero. Validation happens during the write transaction itself. Bad data either gets flagged, removed, or stopped before it reaches the next layer.

How Expectations Work in Lakeflow Declarative Pipelines

An expectation is a SQL Boolean condition attached to a streaming table or materialized view. Every row passing through the pipeline gets checked against it. If the condition returns false, the expectation fires.

Basic syntax in SQL:

CREATE OR REFRESH STREAMING TABLE silver.orders
CONSTRAINT valid_order_id EXPECT (order_id IS NOT NULL)
CONSTRAINT valid_amount EXPECT (amount > 0)
AS SELECT * FROM STREAM(bronze.orders_raw);

As the Databricks expectations documentation confirms: expectations use standard SQL Boolean statements to specify constraints. You can combine multiple expectations for a single dataset and set expectations across all dataset declarations in a pipeline.

The Three Actions: WARN, DROP and FAIL

Choosing the right action for each expectation is the design decision that matters most. Use the wrong one and you either lose data you needed or let bad data through that you should have stopped.

Action	What Happens to the Row	Pipeline Continues?	When to Use
WARN (default)	Kept in the output table, violation logged	Yes	Monitoring and trend tracking. You want to know something is wrong without stopping the pipeline.
DROP	Removed from the output, never written	Yes	Rows that are genuinely invalid and would corrupt downstream aggregations.
FAIL	Pipeline run stops immediately	No	Critical fields where any failure means the entire batch is untrustworthy.

WARN: Monitor Without Stopping

WARN keeps the bad row in the output but logs the failure in the event log with pass/fail counts. Use WARN when:

• The rule is new and you are not yet sure how often it fires in production
• The violation is worth monitoring but does not break downstream use
• You want to track trend data (pass rate dropping over time signals a source system problem)

CONSTRAINT valid_country_code EXPECT (country_code IN ('US', 'UK', 'IN', 'AU'))
-- No ON VIOLATION clause means WARN by default

DROP: Remove Without Stopping

DROP silently removes rows that fail the expectation. The output table receives only valid rows. Use DROP when:

• The invalid rows would produce meaningless or misleading aggregations if they stayed
• The volume of invalid rows is expected to be small and losing them is acceptable
• The source regularly sends noise rows (test events, system pings) that should never reach Silver

CONSTRAINT non_null_customer_id EXPECT (customer_id IS NOT NULL) ON VIOLATION DROP ROW

The danger with DROP: if you drop more rows than expected, you never know. The pipeline completes, the output table looks normal, and the row count is lower than it should be without any alert. Combine DROP with a WARN expectation on the same field, or monitor row counts via SQL Alerts, to catch unexpected drop volumes.

FAIL: Stop and Alert

FAIL stops the entire pipeline run when any row violates the expectation. Use FAIL when:

• A field is so central to the pipeline's purpose that bad values make the entire output meaningless
• A compliance requirement says certain data must never reach the output layer
• You would rather have a failed pipeline incident than have incorrect data reach dashboards

CONSTRAINT valid_transaction_id EXPECT (transaction_id IS NOT NULL AND LENGTH(transaction_id) = 36)
ON VIOLATION FAIL UPDATE

FAIL is powerful but easy to overuse. One bad row should not always stop a pipeline that processes a billion rows. For most production tables, FAIL belongs on the most critical fields only, not every expectation.

The Batch Threshold Pattern: Fail on Percentage, Not Single Rows

Most engineers write expectations that fire on individual rows. A single null triggers DROP. A single bad value triggers FAIL. This works for critical fields but creates a problem for noisy sources.

A real-world example: your Bronze table receives events from a mobile app. On average, 0.3% of events have a null device_id because of an SDK bug. That is expected and acceptable. You write a DROP expectation and move on.

Then one day a deployment bug pushes 40% null device_id events. Your DROP expectation removes 40% of your rows silently. The Silver table shows lower than normal row counts. Nobody gets alerted because the expectation is doing exactly what you told it to do.

The batch threshold pattern fixes this. You set a FAIL expectation that triggers when the failure rate for a batch exceeds a threshold. One bad row is noise. 40% bad rows is a source system incident.

In Lakeflow Declarative Pipelines, implement this with a secondary expectation on the same field using aggregation logic:

-- Individual row check: drop single null rows
CONSTRAINT non_null_device EXPECT (device_id IS NOT NULL) ON VIOLATION DROP ROW

-- Batch-level check: fail if more than 5% of rows are null
CONSTRAINT device_null_rate EXPECT ( (SELECT COUNT() FROM stream WHERE device_id IS NULL) / (SELECT COUNT() FROM stream) < 0.05 ) ON VIOLATION FAIL UPDATE

As the Databricks expectation recommendations documentation confirms: multiple expectations can be combined on a single dataset. Using both row-level and batch-level checks on the same field gives you noise tolerance at the row level and incident detection at the batch level.

This pattern is not documented in any competitor article on Databricks data quality. It is the difference between a quality framework that catches problems and one that generates false confidence.

The Quarantine Pattern: Route Bad Data Instead of Dropping It

DROP removes bad rows. That is clean and fast. It is also a problem in two situations:

1. You need to investigate why rows are failing and fix the source
2. A compliance audit asks where certain records went and why

The quarantine pattern solves both. Instead of dropping bad rows, you route them to a separate quarantine table. The main Silver table receives only valid rows. The quarantine table receives only invalid rows. Both tables are queryable, governed by Unity Catalog, and auditable.

-- Main Silver table: valid rows only
CREATE OR REFRESH STREAMING TABLE silver.customers
CONSTRAINT valid_id EXPECT (customer_id IS NOT NULL) ON VIOLATION DROP ROW
AS SELECT * FROM STREAM(bronze.customers_raw)
WHERE customer_id IS NOT NULL;

-- Quarantine table: invalid rows only
CREATE OR REFRESH STREAMING TABLE silver.customers_quarantine
AS SELECT *, current_timestamp() AS quarantined_at, 'null_customer_id' AS failure_reason
FROM STREAM(bronze.customers_raw)
WHERE customer_id IS NULL;

The quarantine table gives your team three things:

• Root cause investigation: you can query exactly which rows failed and why
• Source system feedback: when the quarantine table fills up faster than usual, the source team gets a specific query result showing what their system sent
• Audit trail: for regulated industries, you can demonstrate that every invalid record was captured, retained, and traceable

As the Databricks reliability best practices documentation confirms: to quarantine identified invalid records, expectation rules can be defined so that invalid records are stored in another table. This is the recommended pattern for production environments where data loss is not acceptable.

Unity Catalog Expectation Storage: The 2026 Change That Matters

Before 2026, expectations lived in pipeline source code. If you used the same quality rule across 15 pipelines, you wrote it 15 times. When the business rule changed, you updated it 15 times. When an auditor asked "what are your data quality rules for customer tables?", you searched through pipeline code files.

In 2026, Databricks added centralized expectation storage in Unity Catalog. As confirmed in the Databricks 2026 SDP release notes: you can now store and manage data quality expectations directly in Unity Catalog tables, centralizing data quality rules with your data governance framework. This enables version-controlled, auditable quality rules that can be shared across multiple pipelines.

What this means practically:

• Write a quality rule once in a Unity Catalog table
• Reference it from any pipeline that validates the same field
• Update the rule in one place and it applies everywhere
• Query Unity Catalog to get a complete inventory of every data quality rule in your organization
• Review the audit log to see when a rule was changed and by whom

This is what turns data quality from a pipeline feature into a governance capability.

Monitoring Quality Over Time: The Event Log Dashboard

Single-run pass/fail counts are not enough. You need trends.

An expectation that passes 99.8% of rows this week and 91% next week is a source system problem in progress. It will not show up as a job failure. It will not trigger a FAIL expectation (which only fires at 0%). It will only surface if you track pass rates over time.

Query the event log to build this view:

SELECT
  date(timestamp) AS run_date,
  details:flow_progress:data_quality:expectations[0]:name AS rule_name,
  details:flow_progress:data_quality:expectations[0]:pass_count AS passed,
  details:flow_progress:data_quality:expectations[0]:fail_count AS failed,
  ROUND(passed / (passed + failed) * 100, 2) AS pass_rate_pct
FROM event_log('<pipeline-id>')
WHERE event_type = 'flow_progress'
ORDER BY timestamp DESC

Set this query as a SQL Alert scheduled to run daily. Alert when pass_rate_pct drops below your defined threshold for any rule. A 2% drop in pass rate on a revenue-critical field is worth a Slack notification. Finding out about it from an analyst three weeks later is not.

The key data quality dimensions to monitor across every Silver table:

• Completeness: what percentage of required fields have non-null values
• Validity: what percentage of values fall within expected formats or ranges
• Uniqueness: what percentage of rows have a unique value on the primary key
• Volume: is the row count for each batch within the expected range for that time window

What This Series Covers Next

• Workflow Orchestration with Lakeflow Jobs covers how to add quality gate tasks between pipeline runs, how to trigger alerts and downstream actions when quality thresholds are breached, and how to design multi-task workflows that handle quality failures gracefully.
• Data Governance with Unity Catalog in Databricks covers how Unity Catalog manages the centralized expectation store, data lineage, column-level security, and audit logging that together make a data platform trustworthy for regulated industries.