AWS Glue Data Quality
Rule-based and ML-assisted data quality checks for data lakes and pipelines: DQDL rules, rule recommendations, checks on Data Catalog tables and in ETL jobs, quality scores, failed records, anomaly detection, alerts, and cost.
3 min read · Last reviewed · MLA-C02
Sign in to bookmark tutorials and track what you have read.
On this page
Already know this topic?
Try the AWS Glue Data Quality quiz: 5 questions on AWS Glue Data Quality.
What it is
AWS Glue Data Quality measures and monitors the quality of data in Data lakeA central store for large amounts of structured and unstructured data kept in its original form, usually in Amazon S3, which many tools can analyze. AWS Lake Formation helps build and secure one.Glossary entry and pipelines. You write rules in the Data Quality Definition Language (DQDL), such as "order_id is complete and unique" or "row count is greater than the average of the last 10 runs", and Glue evaluates them and gives a data quality score: the percentage of rules that pass. It is serverless, built on the open-source Deequ framework, and can also recommend rules and detect anomalies with ML (Machine learning)Software that learns patterns from example data to make predictions or decisions, instead of following rules written by hand. A model is first trained on data, then used to make predictions on new data, which is called inference.Glossary entry.
Choose it when a question asks to check data quality automatically, stop bad data before it reaches a data lake or a training job, or watch data quality over time (MLA-C02 tasks 1.3.1 and 4.1.2).
Two places to run it
| Entry point | How it works | Choose it for |
|---|---|---|
| Data Catalog tables | Evaluate rulesets on tables (S3, Amazon Redshift, Iceberg, Hudi, Delta Lake, and others); get rule recommendations; schedule runs | Ongoing checks of Encryption at restEncrypting data where it is stored, such as on disks, in databases, or in backups, so that it cannot be read without the key. On AWS it usually uses keys in AWS KMS.Glossary entry, by data stewards, with little code |
| AWS Glue ETL (Extract, transform, and load)Copying data from sources, cleaning and reshaping it, and loading it into a target such as a data warehouse. AWS Glue is the AWS serverless ETL service.Glossary entry jobs | Add a data quality step in Glue Studio or in a script | Proactive checks in transit: find the failed records, quarantine them, and stop the job before bad data loads |
Concepts and features
- DQDL rules: completeness, uniqueness, value ranges, allowed values, data types, freshness, referential integrity between datasets, custom SQL (Structured Query Language)The standard language for querying and changing data in relational databases. Analytics services such as Amazon Athena and Amazon Redshift also use SQL.Glossary entry, and more. Dynamic rules compare with past runs (for example
RowCount > avg(last(10))). - Rule recommendations: Glue analyzes a table and proposes a starting ruleset.
- Anomaly detection: analyzers collect statistics over time, and ML flags unusual values or volumes that no rule describes.
- Results: written to Amazon S3, sent to Amazon CloudWatch and Amazon EventBridge, so a rule on a low score can alert a team or stop a pipeline.
Security, resilience, performance, and cost
Security
Runs use IAM roles with access only to the tables and buckets they check; results in S3 are encrypted like other Glue outputs; Lake Formation permissions apply to governed tables.
Resilience
Serverless and managed; checks in ETL jobs can stop a run so that bad data never lands, and results history supports audits.
Performance
Checks in ETL jobs scale with the job (auto scaling and Flex are supported there); incremental checks use job bookmarks or pushdown predicates so only new data is evaluated.
Cost
You pay for the compute (DPU hours) that evaluations and recommendations use, with no license. Check new partitions incrementally instead of whole tables.
Monitoring
Watch quality scores in CloudWatch, route results through EventBridge to SNS or Step Functions, and record API (Application programming interface)A defined way for one program to ask another for data or actions, such as a set of HTTPS requests and responses. Every AWS service is controlled through its API, which the console, the CLI, and the SDKs call for you.Glossary entry calls in CloudTrail.
Comparisons
| Need | Choose |
|---|---|
| Rule-based checks on Data Catalog tables or inside Glue ETL, with failed-record handling | AWS Glue Data Quality |
| Visual profiling and cleaning without code | AWS Glue DataBrew |
| Data insights and target leakage before ML training | Data Wrangler in SageMaker Canvas |
| Drift of model inputs in production | Model monitoring with data capture and a baseline (see SageMaker AI deployment) |
Reverse mindmap: when the question points here
- Points to Glue Data Quality
- Stop the ETL run when rules fail leads to Data quality step in the Glue job
- Quarantine only the bad rows leads to Failed records in ETL checks
- Suggest rules for a new table leads to Rule recommendations
- Volume suddenly drops, no rule for it leads to Anomaly detection
- Points elsewhere
- Clean data with clicks leads to DataBrew
- Production model inputs drift leads to Model monitoring
Key takeaways
- Write rules in DQDL; the score is the share of rules that pass.
- Run on Data Catalog tables (with recommendations) or inside Glue ETL jobs (with failed-record handling and stopping bad loads).
- Dynamic rules and ML anomaly detection catch changes over time.
- Results go to S3, CloudWatch, and EventBridge for alerts and automation.
Sources
- AWS Glue Developer Guide: AWS Glue Data Quality
- AWS Glue Developer Guide: Data Quality Definition Language (DQDL) reference