Best 21 Data Quality Tools for Apache Spark (2026)

Open-source Databricks Labs data quality framework that defines and runs row- and column-level data quality checks on PySpark DataFrames in pipelines and notebooks, for batch and Structured Streaming.

Open-source Spark data quality library from AWS Labs that turns VerificationSuite checks into Spark jobs that compute metrics over a DataFrame, then evaluates assertions on those metrics, with PyDeequ as the Python API.

Open-source PySpark data quality framework from Nike that applies row, aggregate, and query rules inside PySpark pipelines through a decorator, writing clean rows to a target table and failures to an error table.

Open-source, declarative PySpark data quality framework with 30+ built-in row and aggregate checks, defined in Python or as dicts, YAML, or JSON, and run in a single pass in your Spark pipelines.

Open-source Python data quality framework that adds a Spark data source and validates in-memory Spark DataFrames passed in at runtime as batches.

Open-source DataFrame validation library that validates pyspark.sql DataFrames with DataFrameModel schemas, returning a Spark DataFrame with the errors collected instead of failing the job.

Open-source DataFrame comparison library from Capital One that joins two PySpark DataFrames, classic or Spark Connect, on join columns with null-safe equality and reports unequal values with tolerances.

Open-source data validation library by Posit that validates PySpark tables through its Ibis backend, installed with the pyspark extra.

Open-source Python profiling library (formerly ydata-profiling) that profiles PySpark DataFrames for large datasets when installed with the spark extra.

Data tests built into open-source dbt Core that run as SQL in Apache Spark through the dbt-spark adapter: generic tests such as unique and not_null, custom SQL tests, and unit tests.

Managed AWS data quality service built on Deequ, with DQDL rulesets, rule recommendations, and anomaly detection for data lakes and Glue pipelines.

Open-source data contracts engine that scans Spark DataFrames registered as temporary views, passing your Spark session to programmatic Soda scans with the soda-spark-df package.

AI-native data quality platform built on data contracts that scans Spark DataFrames registered as temporary views, passing your Spark session to programmatic Soda scans with the soda-spark-df package.

Open-source command-line tool for ODCS data contracts that tests in-memory Spark DataFrames against a data contract programmatically, from Python.

Open-source data quality operations center that connects over JDBC to a running Spark Thrift Server and runs profiling and monitoring checks on the tables it imports.

Data + AI observability platform that connects to a Spark Thrift Server (binary or HTTP mode, or a Databricks cluster) through the Monte Carlo CLI and runs health queries.

Autonomous data quality monitoring platform that monitors Databricks tables across SQL warehouses, Hive Metastore, and Spark, and pushes data quality results into Unity Catalog.

Autonomous data and AI platform that captures Spark lineage with the OpenLineage listener, sending events to its control plane and mapping each Spark application to a pipeline.

AI-native data quality platform that monitors Spark and Hadoop workloads on EMR clusters, adding anomaly detection and validation to EMR jobs.

Agentic data trust platform that runs Ataccama ONE Spark jobs on Databricks clusters, where its processing engine copies the required libraries to the cluster and runs the job there.

Data observability, quality, and lineage platform that extracts Spark job, run, and task metadata and maps lineage to upstream and downstream tables, with a job failure monitor.