Skip to main content
LetMeMaster
Search Loading account Sign in

AWS Glue

Serverless data integration: the Data Catalog, crawlers, ETL jobs, job bookmarks, DataBrew, and Data Quality, and when to choose Amazon EMR instead.

8 min read · Last reviewed · SAA-C03

Loading your progress

Sign in to bookmark tutorials and track what you have read.

On this page

What it is

AWS Glue is a serverless data integration service. It helps you discover data, keep a central catalog of it, and move and transform it with ETL (Extract, transform, and load)Copying data from sources, cleaning and reshaping it, and loading it into a target such as a data warehouse. AWS Glue is the AWS serverless ETL service.Glossary entry (ETL) jobs. There are no servers or clusters to manage: you pay for the processing capacity a job or crawler uses while it runs.

The Data Catalog

The AWS Glue Data Catalog is a central metadata store: databases and tables that describe where data lives, its columns and types, and its partitions. Each AWS account has one Data Catalog per AWS RegionA separate geographic area where AWS runs its services, such as us-east-1 in Northern Virginia, made up of several Availability Zones. Most resources live in one Region; you choose Regions for latency, data residency, and the services you need.Glossary entry.

A table in the Data Catalog holds metadata only. The data stays where it is, in Amazon S3, a relational database, or another store. Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum all read their table definitions from the Data Catalog, so one catalog serves every engine.

  • Partition indexes speed up queries on tables with many thousands of partitions: instead of loading every partition and filtering, the catalog fetches only the matching ones.
  • Sharing with other accounts works through a Data Catalog resource policy or, for fine-grained control, through AWS Lake Formation.

Crawlers

A crawler connects to a data store, works through a list of classifiers to infer the schema (CSV, JSON (JavaScript Object Notation)A text format for structured data made of name-and-value pairs and lists, such as {"bucket": "logs"}. IAM policies, most AWS API responses, and many CloudFormation templates are written in JSON.Glossary entry, Avro, XML, JDBC databases, or your own patterns), and creates or updates tables in the Data Catalog. Run crawlers on a schedule to pick up new partitions and schema changes automatically.

One catalog for many engines. A crawler infers the schema of data in Amazon S3 and writes table definitions to the Glue Data Catalog. Athena, EMR, and Redshift Spectrum all read their table definitions from the same catalog, while the data stays in S3.

ETL jobs

  • A Spark job runs a transformation script on a serverless Apache Spark engine. AWS Glue can generate the script in Python (PySpark) or Scala, or you build the job visually in AWS Glue Studio. A Python shell job runs a plain Python script for small tasks.
  • A streaming ETL job runs continuously, reading from Amazon Kinesis Data Streams, Apache Kafka, or Amazon MSK, and loading results into Amazon S3 or a JDBC data store. It tracks its position with checkpoints.
  • Triggers start jobs on a schedule, on demand, or when an event occurs; workflows chain crawlers, jobs, and triggers.
  • Job bookmarks record which data a batch job has already processed, so each scheduled run handles only new data. For Amazon S3 sources they track which objects were processed; for JDBC sources, such as an Amazon RDS table, they use a bookmark key whose values only increase (or only decrease), such as an order_id. By default Glue uses the primary key, if its values increase or decrease sequentially.
  • Capacity is measured in Data Processing Units (DPUs): one DPU is 4 vCPUs and 16 GB of memory. You choose a worker type, such as G.1X (1 DPU) or G.2X (2 DPUs) for most jobs, larger G types for demanding jobs, or memory-optimized R types, and the number of workers.
  • AWS Glue DataBrew: a visual tool to clean and normalize data without writing code, with more than 250 ready-made transformations.
  • AWS Glue Data Quality: rules that measure and monitor data quality, so bad data can be caught before it reaches the Data lakeA central store for large amounts of structured and unstructured data kept in its original form, usually in Amazon S3, which many tools can analyze. AWS Lake Formation helps build and secure one.Glossary entry.
  • Sensitive data detection finds personal and other sensitive data in a pipeline, and the FindMatches ML (Machine learning)Software that learns patterns from example data to make predictions or decisions, instead of following rules written by hand. A model is first trained on data, then used to make predictions on new data, which is called inference.Glossary entry transform finds duplicate records that do not match exactly.
  • AWS Glue Schema Registry: a free, serverless registry of schemas (Avro, JSON Schema, or Protobuf) for streaming data in Apache Kafka, Amazon MSK, and Kinesis Data Streams. Compatibility rules stop producers from publishing changes that would break consumers.

Security, resilience, performance, and cost

Security

  • IAM role: every job and crawler runs with an IAM role. Grant it only the S3 buckets, databases, and catalog tables it needs.
  • Encryption of job output: a security configuration encrypts what jobs write to Amazon S3 (SSE (Server-side encryption)Encryption that a service applies when it stores your data and removes when you read it back. Amazon S3 encrypts every new object with SSE-S3 by default; SSE-KMS uses keys in AWS KMS instead.Glossary entry-S3 or SSE-KMS), their CloudWatch logs, and their job bookmarks.
  • Encryption of the catalog: Data Catalog encryption protects databases, tables, and partitions with an AWS KMS key, either the AWS managed key or a Customer managed keyA KMS key that you create, own, and manage: you control its key policy, rotation, and deletion. AWS managed keys, in contrast, are created and managed for you by an AWS service.Glossary entry.
  • Private network access: a job that only moves data between S3 buckets needs no network setup. To reach a database in a SubnetA range of IP addresses in a VPC, inside one Availability Zone. A subnet is public when its route table sends internet traffic to an internet gateway, and private when it does not.Glossary entry, create a Glue connection in that VPC. Glue places ENI (Elastic network interface)A virtual network card in a VPC subnet, with private IP addresses, security groups, and a MAC address. EC2 instances, Lambda functions connected to a VPC, and interface endpoints all connect through network interfaces.Glossary entry with private IP (Internet Protocol)The protocol that addresses and routes packets between networks. Each device has an IP address: IPv4 addresses look like 192.0.2.1, and IPv6 addresses are longer, like 2001:db8::1.Glossary entry addresses in the subnet, and the Security groupA stateful virtual firewall for network interfaces, such as an EC2 instance's, whose rules can only allow traffic. Return traffic for allowed connections is allowed automatically, and a rule can refer to another security group instead of IP addresses.Glossary entry needs a self-referencing inbound rule for all TCP (Transmission Control Protocol)The transport protocol that delivers data reliably and in order: it sets up a connection first and resends lost packets. HTTP, SSH, and most database connections run over TCP.Glossary entry ports. From inside the VPC, a job needs an S3 VPC endpointA private connection from a VPC to an AWS service or another endpoint service without crossing the internet. Gateway endpoints, for Amazon S3 and DynamoDB, are route table targets; interface endpoints are network interfaces that use AWS PrivateLink.Glossary entry to reach Amazon S3, and a NAT gatewayA managed VPC component that lets instances in private subnets start connections to the internet or other networks, while blocking connections started from outside. It is billed per hour and per gigabyte processed.Glossary entry only if it must reach the internet.

Resilience

  • A job can retry automatically up to 10 times when it fails; a job that reaches its timeout is not retried.
  • Job run queuing makes runs wait for capacity instead of failing when service quotas are reached.
  • Job bookmarks, and checkpoints for streaming jobs, let a job resume where it left off without duplicating data.

Performance

  • Choose larger or memory-optimized worker types for heavy joins and aggregations, and more workers for more parallelism.
  • Partition the data in S3, filter on partitions, and add partition indexes to tables with many partitions.

Cost

  • Jobs and crawlers are billed per DPU-hour, by the second. The Data Catalog is free for the first million stored objects and the first million requests each month.
  • The Flex execution class runs non-urgent jobs, such as tests, pre-production jobs, and one-time loads, on spare capacity at a lower rate. AWS's pricing example shows $0.29 per DPU-hour with Flex against $0.44 standard.
  • Set a sensible job timeout: jobs are billed for their run time, so a stuck job costs money until it times out.
  • Use job bookmarks so scheduled jobs do not reprocess old data, and stop streaming jobs that are not needed, since they bill while they run.

Monitoring

  • Amazon CloudWatch: job metrics, observability metrics, and continuous logging to CloudWatch Logs while a job runs. The Apache Spark UI shows how a job spent its time.
  • Amazon EventBridge receives job and crawler state changes, such as a job that FAILED or hit its TIMEOUT. A rule can notify an Amazon SNS topic or start an AWS Lambda function. A delay notification threshold sends an event when a run takes longer than expected.
  • AWS CloudTrail records Glue API (Application programming interface)A defined way for one program to ask another for data or actions, such as a set of HTTPS requests and responses. Every AWS service is controlled through its API, which the console, the CLI, and the SDKs call for you.Glossary entry calls.

AWS Glue or Amazon EMR?

If the requirement is Choose
Serverless ETL with no clusters to manage AWS Glue
Discover and catalog data for Athena and other engines AWS Glue crawlers and Data Catalog
Clean data visually with no code AWS Glue DataBrew
Control over the cluster, frameworks such as Hadoop or HBase, or Spot Instances Amazon EMR

Reverse mindmap: when the question points here

Is it AWS Glue?
  • What to use
    • Discover the schema of data in S3 automatically leads to Glue crawler
    • A central metadata catalog for Athena, EMR, and Spectrum leads to Glue Data Catalog
    • Serverless ETL with no infrastructure leads to Glue ETL job
    • Transform streaming data continuously leads to Glue streaming ETL
    • Process only new data on each scheduled run leads to Job bookmarks
    • Clean and normalize data without writing code leads to Glue DataBrew
    • Check data quality before loading it leads to Glue Data Quality
    • Enforce schemas for Kafka or Kinesis producers leads to Glue Schema Registry
  • Run it securely
    • Read a database in a private subnet leads to Glue connection in the VPC
    • Encrypt job output, logs, and bookmarks with KMS leads to Security configuration
    • Encrypt table definitions leads to Data Catalog encryption
    • Reach S3 from a job inside the VPC leads to S3 VPC endpoint
  • Run it cheaply and reliably
    • Non-urgent or test jobs at a lower rate leads to Flex execution class
    • Alert when a job fails leads to EventBridge rule to SNS
    • Faster queries on tables with many partitions leads to Partition indexes
  • Not Glue
    • Hadoop, HBase, or control over the cluster leads to Amazon EMR
    • Column or row permissions on data lake tables leads to AWS Lake Formation
    • Query the data with SQL leads to Amazon Athena

Key takeaways

  • Glue is serverless ETL: you pay per DPU-hour, by the second, only while jobs and crawlers run.
  • The Data Catalog stores metadata; the data stays in its original store.
  • Secure it with a Least privilegeGiving each person or system only the permissions it needs for its task, and no more. In IAM, this means starting with no access and allowing specific actions on specific resources.Glossary entry role, security configurations, catalog encryption, and a VPC connection for private databases.
  • Flex jobs, timeouts, and bookmarks keep costs down; EventBridge rules alert on failed jobs.

Sources

Check your understanding

Loading your quiz progress

Sign in to check your understanding with 14 questions on AWS Glue.

Glossary Install the app

LetMeMaster is an independent educational project and is not affiliated with or endorsed by Amazon Web Services. Practice results do not predict actual exam outcomes.

© 2026 LetMeMaster. All rights reserved. Content may not be copied or redistributed without permission.

Proudly made in Canada

Add LetMeMaster to your Home Screen

  1. Tap the Share button in your browser's toolbar. In Safari it is at the bottom of the screen, or at the top on iPad.
  2. Scroll down and tap Add to Home Screen.
  3. Tap Add. Open LetMeMaster from its icon on your Home Screen.