# Databricks Data Quality

> Native Databricks data quality: expectations in Lakeflow pipelines warn on, drop, or fail invalid records, and data profiling under Unity Catalog data quality monitoring tracks table metrics and drift over time.

Source: https://toolsfordata.com/tools/databricks-data-quality/ · Updated 2026-11-17

[Docs](https://docs.databricks.com/aws/en/ldp/expectations)

Best for: Databricks teams that want data quality rules inside their Lakeflow pipelines and table monitoring in Unity Catalog, without adding a vendor.

## Overview

- Core Strength: Databricks-native — pipeline expectations plus Unity Catalog freshness, volume and drift monitoring
- Test Types: Partial — SQL, table compare, failed rows
- Detection: Full — profiling, anomaly detection, pipeline monitoring
- Monitor Coverage: 10 of 11 types — as-code + UI
- Incident Workflow: Partial — triage, alert feedback, resolution, on-call
- Quality Analytics: Partial — dashboards
- Lineage Depth: Full — table, column, end-to-end, impact analysis
- Governance Overlap: Partial — catalog, catalog sync, classification
- FinOps & AI Observability: Full — query performance, cost monitoring, AI observability
- AI Agent Features: Partial — chat, auto test gen
- Developer Surface: Full — API, Python SDK, MCP, dbt…
- Deployment Model: Partial — SaaS, hybrid, cloud-native app
- Security & Access: Full — SSO, SCIM, RBAC, audit log…
- Pricing: Usage — Databricks DBUs
- Free Option: None
- Notable Caveat: Databricks only; anomaly detection covers freshness and row counts only

## What it's used for

Databricks Data Quality is used to check data inside Databricks without adding a vendor: expectations in Lakeflow pipelines handle invalid records on each update, and Unity Catalog data quality monitoring tracks table metrics and drift.

## Pricing

Databricks Data Quality has no separate license: pipelines and monitoring run on Databricks compute and are billed in DBUs.

## Pricing

- Model: platform usage
- Source: https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/ (checked 2026-11-10)

## Features

### Data Testing

- SQL Table Tests: Yes — Pipeline expectations are SQL constraints (nulls, ranges, accepted values, any boolean rule) that warn, drop or fail on invalid records. ([source](https://docs.databricks.com/aws/en/ldp/expectations))
- DataFrame Tests: No — Expectations apply to datasets in Lakeflow pipelines; validating standalone Pandas or Spark DataFrames is not documented (that is DQX). ([source](https://docs.databricks.com/aws/en/ldp/expectations))
- Unstructured Data Tests: No — Expectations are SQL constraints; there is no LLM or JSON Schema check. ([source](https://docs.databricks.com/aws/en/ldp/expectations))
- Table Comparison Tests: Yes — The documented row-count pattern compares counts between two tables inside a pipeline and fails the update when they differ. ([source](https://docs.databricks.com/aws/en/ldp/expectation-patterns))
- Data Diff: No — No row-by-row comparison of two datasets is documented. ([source](https://docs.databricks.com/aws/en/ldp/expectation-patterns))
- Data Contracts: No — Quality rules are constraints on pipeline datasets; there is no producer-consumer contract. ([source](https://docs.databricks.com/aws/en/ldp/expectation-patterns))
- Inspect Failed Rows: Yes — Records that fail a warn expectation stay in the table to query, and the documented quarantine pattern splits invalid rows out; there is no built-in viewer. ([source](https://docs.databricks.com/aws/en/ldp/expectation-patterns))

### Data Observability

- Data Profiling: Yes — Data profiling computes null fractions, distinct counts and distributions per column and stores them over time in metric tables. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/))
- Anomaly Detection: Yes — Anomaly detection learns each table's update cadence and row-count range from history and flags late or incomplete data. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Freshness Monitors: Yes — Anomaly detection predicts each table's next commit and marks it stale when it is late. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Volume Monitors: Yes — The completeness check compares rows written in the last 24 hours with a learned expected range. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Schema Monitors: No — Anomaly detection covers freshness and completeness only; schema changes are not detected. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/results))
- Field Monitors: Yes — Data profiling stores per-column statistics as time series and computes drift, and SQL alerts can watch them. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/))
- Distribution Monitors: Yes — Drift metrics compare a column's distribution between time windows or against a baseline table. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/))
- Segment Monitors: Yes — Profiles accept slicing expressions, and completeness checks can be sliced by up to three columns (beta). ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/create-monitor-api))
- Custom SQL Monitors: Yes — Custom metrics are SQL expressions computed on every profile refresh and stored over time. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/custom-metrics))
- Partition-Based Monitors: Yes — Time series profiles compute metrics only over the last 30 days of data instead of the full table. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/))
- Window-Based Monitors: Yes — Time series profiles aggregate metrics into windows from 5 minutes to 1 year. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/create-monitor-api))
- UI-Based Monitors: Yes — Anomaly detection and data profiles are turned on and configured in Catalog Explorer without code. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Monitors-as-Code: Yes — Expectations are code in pipeline source files, and quality monitors are a resource in Databricks Asset Bundles. ([source](https://docs.databricks.com/aws/en/dev-tools/bundles/resources))
- Pipeline Monitoring: Yes — The pipeline event log records each update's status, progress, data quality metrics and errors. ([source](https://docs.databricks.com/aws/en/ldp/monitor-event-logs))

### Incident Management

- Alert Notifications: Yes — Anomaly alerts go to email, Slack, Microsoft Teams or webhooks; profile alerts are SQL alerts. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/alerts))
- Incident Triage: Yes — Unhealthy tables are listed as incidents with High, Medium or Low impact, and users can assign one to themselves. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Alert Feedback: Yes — Users can mark an unhealthy table as Not an issue, which dismisses it for 7 days. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Incident Resolution: Yes — Incidents resolve automatically or are dismissed, and a Recently Resolved list records each as Auto-resolved or Not an issue. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Ticketing Integrations: No — Notification destinations are email, Slack, Teams, PagerDuty and webhooks; there is no Jira or ServiceNow destination. ([source](https://docs.databricks.com/aws/en/admin/workspace-settings/notification-destinations))
- On-Call Integrations: Yes — PagerDuty is a native notification destination for SQL alerts on profile metrics. ([source](https://docs.databricks.com/aws/en/admin/workspace-settings/notification-destinations))

### Data Quality Analytics

- Data Quality Dashboards: Yes — A Data Quality Monitoring dashboard shows the share of healthy tables by catalog, and each profile gets its own dashboard. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Data Health Scores: No — Each table gets a status (Healthy, Unhealthy and so on), not a numeric quality score. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Incident Resolution Metrics: No — Incidents show detected and resolved times, but no MTTD or MTTR metrics. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))

### Data Lineage

- Table-Level Lineage: Yes — Unity Catalog captures table lineage automatically for queries run on Databricks. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage))
- Column-Level Lineage: Yes — Unity Catalog captures column lineage for queries run on Databricks. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage))
- End-to-End Lineage: Yes — Lineage can include external sources such as Salesforce and BI tools such as Tableau or Power BI once they are registered as external assets. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage))
- Impact Analysis: Yes — Lineage shows downstream tables, jobs and dashboards, and anomaly detection uses it to find the upstream cause. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage))

### Data Governance

- Data Catalog: Yes — Catalog Explorer is Unity Catalog's searchable inventory, with health indicators on each table. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Data Catalog Integration: Yes — Health indicators and results show in Unity Catalog's Catalog Explorer; no push to third-party catalogs is documented. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Data Classification: Yes — Data classification scans catalogs and tags sensitive columns such as PII automatically (preview). ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-classification))
- Data Products: No — Results roll up by catalog and schema, not by data product. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))

### FinOps

- Query Performance: Yes — The query history system table and query profiles show duration and performance of queries. ([source](https://docs.databricks.com/aws/en/sql/user/alerts/))
- Cost Monitoring: Yes — Data quality monitoring is billed as its own product in the system.billing.usage table. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/))

### LLMOps

- AI Observability: Yes — Inference profiles track model inputs, predictions and quality metrics for model serving endpoints over time. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/))

### Workflows

- Scheduling: Yes — Anomaly detection runs automatically per table, and data profiles refresh on a cron schedule. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/create-monitor-api))
- CI/CD Integrations: Yes — An expect_or_fail violation fails the pipeline update, so downstream job tasks don't run; checks on pull requests are not documented. ([source](https://docs.databricks.com/aws/en/ldp/expectation-patterns))
- Circuit Breakers: Yes — expect_or_fail (ON VIOLATION FAIL UPDATE) stops the pipeline update and rolls back the table, so job tasks chained after it don't run. ([source](https://docs.databricks.com/aws/en/ldp/expectations))
- Webhooks: Yes — Anomaly alerts and SQL alerts can call webhook destinations. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/alerts))
- REST / GraphQL API: Yes — A REST API creates, refreshes and deletes data profiles and anomaly monitors. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/create-monitor-api))
- MCP Server: Yes — Databricks managed MCP servers (Databricks SQL in preview) let AI agents query tables, including data quality result tables; there are no data quality tools of their own. ([source](https://docs.databricks.com/aws/en/generative-ai/mcp/managed-mcp))
- Python SDK: Yes — The databricks-sdk Python package creates, refreshes and reads data profiles. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/create-monitor-api))
- dbt Integration: Yes — dbt tasks in Lakeflow Jobs run dbt projects, including dbt tests, from the workspace or Git. ([source](https://docs.databricks.com/aws/en/jobs/dbt))

### AI Agents

- AI Chat Assistant: Yes — Genie Code, the in-product assistant, answers questions about lineage and tables in natural language. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage))
- Natural Language Test Generation: No — Checks are configured in the UI, SDK, API or bundles; no plain-English check builder is documented. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/data-profiling/create-monitor-api))
- Automated Test Generation: Yes — Turning on anomaly detection for a schema creates freshness and completeness checks for every table, with thresholds learned from history. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- AI Issue Triaging: No — Anomaly detection names a likely upstream cause from Unity Catalog lineage, not from AI. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- AI Issue Resolution: No — No AI-proposed or automated fix is documented. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))

### Security

- SSO: Yes — Workspace and account login use SSO through the identity provider. ([source](https://docs.databricks.com/aws/en/admin/users-groups/scim/))
- SCIM Provisioning: Yes — SCIM provisioning syncs users and groups from Okta or Microsoft Entra ID. ([source](https://docs.databricks.com/aws/en/admin/users-groups/scim/))
- RBAC: Yes — Unity Catalog privileges control who can turn on monitors, see health indicators and read results. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/))
- Audit Log: Yes — The system.access.audit table (preview) records user actions in the workspace. ([source](https://docs.databricks.com/aws/en/admin/system-tables/))
- Compliance Certifications: Yes — Databricks provides a SOC 2 Type II report and ISO certifications. ([source](https://www.databricks.com/trust/compliance))

### Deployment

- SaaS Deployment: Yes — Databricks runs the control plane, and anomaly detection and profiling run on serverless compute. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/))
- Hybrid Deployment: Yes — Pipelines with expectations can run on classic compute in the customer's own cloud account while Databricks runs the control plane. ([source](https://docs.databricks.com/aws/en/getting-started/overview))
- Self-Hosted Deployment: No — Databricks is not available as self-hosted software. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/))
- Cloud-Native App: Yes — Monitors and expectations run inside the Databricks workspace and write results to Unity Catalog tables. ([source](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/))

### Open Source

- Open Source: No — The monitoring and expectations are Databricks features; open-source Spark pipelines do not include expectations. ([source](https://spark.apache.org/docs/latest/declarative-pipelines-programming-guide.html))

## Integrations

8 integrations, each sourced from the vendor docs.

- Data Governance: Databricks Unity Catalog
- Data Storage: Databricks Delta Lake
- Data Warehouse: Databricks SQL
- Notifications: Email, Slack, Microsoft Teams, Webhook
- Open Table Formats: Apache Iceberg

## Alternatives

- [DQX by Databricks](https://toolsfordata.com/tools/dqx/): Databricks Labs' open-source option: PySpark checks generated from profiling, without formal support SLAs.
- [Snowflake Data Quality](https://toolsfordata.com/tools/snowflake-data-quality/): The Snowflake equivalent, with no extra vendor if your warehouse is Snowflake.
- [BigQuery Data Quality](https://toolsfordata.com/tools/bigquery-data-quality/): The Google Cloud equivalent for BigQuery tables.
- [AWS Glue Data Quality](https://toolsfordata.com/tools/aws-glue-data-quality/): The AWS equivalent, paid per use inside Glue.
- [Telmai](https://toolsfordata.com/tools/telmai/): Consider Telmai for lakehouse monitoring with native Delta Lake, Iceberg and Hudi support.
- [Monte Carlo](https://toolsfordata.com/tools/monte-carlo/): Consider Monte Carlo for ML monitors and lineage across Databricks and the rest of the stack.
