AWS Glue Data Quality

AWS Glue data testing

AWS Glue Data Quality runs rulesets written in DQDL on tables in the Glue Data Catalog or on DataFrames in Glue jobs. It compares tables and lets you inspect the records that fail.

Compare: data testing tools

• SQL Table Tests: DQDL rules cover completeness, uniqueness, column values, lengths, data types, referential integrity, freshness and custom SQL (docs).

• DataFrame Tests: The Evaluate Data Quality transform validates Spark DataFrames inside AWS Glue ETL jobs and notebooks (docs).

• Table Comparison Tests: AggregateMatch, RowCountMatch, SchemaMatch and ReferentialIntegrity compare a dataset with reference datasets (docs).

• Inspect Failed Rows: Row-level results in ETL jobs show which records passed and failed each rule (docs).

• Unstructured Data Tests: No LLM-based checks on free text are offered; rules are statistical or SQL-based (docs).

• Data Contracts: No data contract feature; DQDL rulesets define checks but not producer-consumer agreements (docs).

• Data Diff: DatasetMatch compares a primary and reference dataset on key columns and reports a match percentage; returning individual changed records is not documented (docs).

AWS Glue data observability

AWS Glue Data Quality profiles data with DQDL analyzers and learns from statistics over time to flag anomalies. Rules are set up in the console or as rulesets deployed with CloudFormation.

Compare: data profiling tools · data observability tools

• Data Profiling: Analyzers compute column statistics (row count, completeness, distinct values, mean and more) without pass/fail rules (docs).

• Anomaly Detection: ML-based anomaly detection learns from statistics over time and flags anomalies, with the DetectAnomalies rule (docs).

• Freshness Monitors: DataFreshness and FileFreshness rules check how recent the data is (docs).

• Volume Monitors: RowCount rules plus anomaly detection on row counts over time (docs).

• Schema Monitors: ColumnCount, ColumnExists, ColumnDataType and SchemaMatch rules detect schema changes (docs).

• Field Monitors: Analyzers track column statistics over time, and anomaly detection flags deviations (docs).

• Distribution Monitors: Distribution, Entropy, Mean and StandardDeviation rules, plus anomaly detection on these statistics (docs).

• Custom SQL Monitors: CustomSql rule runs a SQL query and evaluates its result (docs).

• Partition-Based Monitors: Filters and partition predicates limit evaluation to selected partitions for incremental validation (docs).

• UI-Based Monitors: Rulesets can be created in the AWS Glue console and Glue Studio (docs).

• Monitors-as-Code: DQDL rulesets can be deployed with AWS CloudFormation (docs).

• Pipeline Monitoring: Not part of Glue Data Quality; job monitoring sits in AWS Glue and CloudWatch (docs).

• Segment Monitors: Not documented (docs).

• Window-Based Monitors: Not documented (docs).

AWS Glue incident management

AWS Glue Data Quality publishes results and alerts through AWS services such as EventBridge. It has no incident workflow.

• Alert Notifications: Publishes Amazon EventBridge events on each run, which can trigger SNS email or other notifications (docs).

• Incident Triage: No incident workflow; results are events and run history (docs).

• Alert Feedback: Not offered (docs).

• Incident Resolution: No incident resolution workflow (docs).

• Ticketing Integrations: No ticketing integration (docs).

• On-Call Integrations: No direct PagerDuty or Opsgenie integration; alerts go through EventBridge (docs).

AWS Glue data quality analytics

AWS Glue Data Quality scores each ruleset.

• Data Health Scores: Each evaluation run produces a data quality score (percentage of rules passed) (docs).

• Incident Resolution Metrics: Not offered (docs).

• Data Quality Dashboards: Run history and scores show in the console; trend dashboards require querying results with Athena or building them in another tool (docs).

AWS Glue data governance

AWS Glue Data Quality runs on tables in the AWS Glue Data Catalog and shows results next to them.

• Data Catalog: Works on top of the AWS Glue Data Catalog, which inventories tables and schemas (docs).

• Data Catalog Integration: Built into the AWS Glue Data Catalog; scores show on catalog tables (docs).

• Data Products: Not offered (docs).

• Data Classification: AWS Glue offers sensitive data detection in ETL jobs, but it is separate from Glue Data Quality (docs).

AWS Glue workflows

AWS Glue Data Quality runs on a schedule or inside Glue jobs, with an AWS API and the Python SDK (boto3). It has no dbt integration.

• Scheduling: Data quality rulesets on Data Catalog tables can be scheduled (docs).

• REST / GraphQL API: AWS Glue API for rulesets, evaluation runs and results (docs).

• Python SDK: Available through the AWS SDK for Python (boto3) and in Glue PySpark jobs (docs).

• CI/CD Integrations: No CI/CD or pull request integration; rulesets deploy through CloudFormation (docs).

• dbt Integration: No dbt integration (docs).

• Webhooks: No direct webhooks; EventBridge can forward events to HTTP endpoints (docs).

• MCP Server: No Glue Data Quality-specific MCP server is documented (docs).

AWS Glue AI agents

AWS Glue Data Quality recommends rules from data statistics. It has no AI chat or triage.

• Automated Test Generation: Rule recommendations analyze data statistics and suggest a DQDL ruleset; anomaly detection also recommends rules (docs).

• AI Chat Assistant: No chat assistant for data quality (docs).

• Natural Language Test Generation: Not offered (docs).

• AI Issue Triaging: Not offered (docs).

• AI Issue Resolution: Not offered (docs).