Amazon EMR

What it is

Amazon EMR (previously Amazon Elastic MapReduce) is a managed cluster platform for big data frameworks such as Apache Spark, Apache Hadoop, Apache Hive, Apache HBase, and Presto. You choose the frameworks and the cluster size, and EMR provisions and configures the Amazon EC2 instances. It suits large-scale processing where you want control over the frameworks, their configuration, and the instances that run them.

Ways to run EMR

EMR on EC2: a cluster of EC2 instances that you size and configure. Most exam scenarios mean this.
EMR on EKS: run EMR workloads on an existing Amazon EKS cluster, alongside other containerized applications.
EMR Serverless: run Spark and Hive jobs with no clusters to configure or operate. EMR Serverless works out the resources each job needs and releases them when the job finishes.

Nodes in a cluster

Node type	What it does
Primary node	Manages the cluster and coordinates tasks. Every cluster has one.
Core nodes	Run tasks and store data in HDFS.
Task nodes	Run tasks only; they store no HDFS data. Optional.

Where to keep the data

HDFS on the cluster is fast but ephemeral: it is reclaimed when the cluster ends. Keep persistent input and output in Amazon S3, and use HDFS for intermediate results.

Because the data lives in S3, a cluster can be transient: it starts, runs its steps, and terminates automatically, so you pay only while there is work to do. A long-running cluster stays up for continuous or interactive work.

An Amazon EMR cluster. The primary node coordinates the core nodes, which run tasks and store HDFS data, and the task nodes, which run tasks only. Core and task nodes read input from and write output to Amazon S3, where the lasting data is kept.

Security, resilience, performance, and cost

Security

IAM roles: every cluster has a service role, which EMR uses to create resources, and an EC2 instance profile, which the applications on the nodes use to reach Amazon S3 and other services. Grant the instance profile only the buckets and data the jobs need.
Security configurations turn on encryption and are reused across clusters:
- At rest: data in Amazon S3 with SSE (Server-side encryption)Encryption that a service applies when it stores your data and removes when you read it back. Amazon S3 encrypts every new object with SSE-S3 by default; SSE-KMS uses keys in AWS KMS instead.Glossary entry-S3, SSE-KMS, or client-side encryption, and local disks (EBS volumes and instance store) with KMS-based encryption.
- In transit: TLS (Transport Layer Security)The protocol that encrypts traffic between a client and a server and lets the server prove its identity with a certificate. It replaced SSL, though the older name is still common; HTTPS is HTTP over TLS.Glossary entry for the applications' network traffic. Traffic between the cluster and Amazon S3 always uses TLS.
- Authentication: Kerberos for users and applications.
Fine-grained access: EMR can enforce AWS Lake Formation permissions on Data lakeA central store for large amounts of structured and unstructured data kept in its original form, usually in Amazon S3, which many tools can analyze. AWS Lake Formation helps build and secure one.Glossary entry tables.
Network: run clusters in SubnetA range of IP addresses in a VPC, inside one Availability Zone. A subnet is public when its route table sends internet traffic to an internet gateway, and private when it does not.Glossary entry, with an S3 VPC endpointA private connection from a VPC to an AWS service or another endpoint service without crossing the internet. Gateway endpoints, for Amazon S3 and DynamoDB, are route table targets; interface endpoints are network interfaces that use AWS PrivateLink.Glossary entry for data access. Block public access is on by default: EMR refuses to launch a cluster in a public subnet whose Security groupA stateful virtual firewall for network interfaces, such as an EC2 instance's, whose rules can only allow traffic. Return traffic for allowed connections is allowed automatically, and a rule can refer to another security group instead of IP addresses.Glossary entry allow inbound traffic from anywhere, except on allowed ports (port 22 by default).

Resilience

A cluster can have three primary nodes, so it keeps running if one fails. EMR turns on termination protection for these clusters.
Persistent data belongs in Amazon S3, not HDFS, so a lost cluster can be recreated and pick up where it left off.
Instance fleets let you list several instance types (and AZ (Availability Zone)An isolated location within an AWS Region, made up of one or more data centers with their own power, cooling, and networking. The AZs in a Region are linked by fast, low-latency connections, so spreading resources across them protects against the failure of one site.Glossary entry), so EMR can find capacity, including Spot capacity, when one type is scarce.

Performance

Choose instance types for the work: more CPU for compute-heavy jobs, more memory for caching, and enough core nodes for the HDFS data a job needs.
Managed scaling adds and removes core and task nodes as the workload changes.

Cost

Spot Instances cost much less than On-Demand but can be taken back by EC2. Each node type reacts differently to losing an instance:

Task nodes hold no HDFS data, so losing one does not lose data. They are the natural place for Spot Instances.
Core nodes store HDFS data; losing them risks data loss.
The primary node ends the cluster if it is lost.

AWS's guidance for long-running and data-critical clusters: run the primary and core nodes On-Demand and add task nodes on Spot. For cost-driven work where losing partial work is acceptable, or for testing, the whole cluster can run on Spot. Managed scaling adds and removes core and task capacity automatically based on the workload.

An auto-termination policy shuts a cluster down after it has been idle for a set time (one hour by default), so forgotten clusters stop billing.
EMR Serverless charges only for the resources each job uses, with nothing to pay between jobs.

Monitoring

Amazon EventBridge receives EMR events: cluster state changes (for example a cluster that terminated with errors), step state changes, and scaling or capacity problems.
Amazon CloudWatch metrics show cluster activity, such as whether a cluster is idle.
Logs can be archived to Amazon S3, so they survive after the cluster ends; AWS CloudTrail records EMR API (Application programming interface)A defined way for one program to ask another for data or actions, such as a set of HTTPS requests and responses. Every AWS service is controlled through its API, which the console, the CLI, and the SDKs call for you.Glossary entry calls.

Amazon EMR or AWS Glue?

If the requirement is	Choose
Hadoop, HBase, or Presto, or control over the cluster and its configuration	Amazon EMR
Large processing jobs at the lowest cost using Spot Instances	Amazon EMR
Serverless ETL (Extract, transform, and load)Copying data from sources, cleaning and reshaping it, and loading it into a target such as a data warehouse. AWS Glue is the AWS serverless ETL service.Glossary entry with a central catalog and no clusters	AWS Glue

Reverse mindmap: when the question points here

Is it Amazon EMR?

Yes, when the question says
- Run Spark, Hadoop, Hive, or HBase with cluster control leads to Amazon EMR
- Migrate an on-premises Hadoop cluster leads to Amazon EMR
- Spark jobs with no clusters to manage leads to EMR Serverless
Run it cheaply
- Cut cost without risking HDFS data leads to Spot Instances for task nodes
- Cluster only for the duration of a job leads to Transient cluster
- Clusters left running after the work is done leads to Auto-termination after idle time
- Workload that varies through the day leads to Managed scaling
Run it securely and reliably
- Encrypt S3 data, local disks, and traffic leads to Security configuration
- Stop clusters being opened to the internet leads to Block public access
- The primary node must not be a single point of failure leads to Three primary nodes
- Keep data after the cluster terminates leads to Store it in Amazon S3
No, when the question says
- Serverless ETL with a data catalog leads to AWS Glue
- Ad hoc SQL on S3 leads to Amazon Athena
- A data warehouse for reporting leads to Amazon Redshift

Key takeaways

EMR runs open-source big data frameworks on clusters you control.
Core nodes store HDFS data; task nodes do not, so they are safe on Spot.
HDFS is ephemeral; keep persistent data in Amazon S3 and use transient clusters.
Security configurations encrypt Encryption at restEncrypting data where it is stored, such as on disks, in databases, or in backups, so that it cannot be read without the key. On AWS it usually uses keys in AWS KMS.Glossary entry and in transit; block public access is on by default.
Auto-termination, managed scaling, and Spot task nodes keep clusters cheap.