Amazon MSK (Managed Streaming for Apache Kafka)

What it is

Amazon Managed Streaming for Apache Kafka (Amazon MSK) runs Apache Kafka clusters for you. MSK handles the control plane, such as creating, updating, and deleting clusters, and detects and replaces failed brokers. Your applications use the normal Apache Kafka API (Application programming interface)A defined way for one program to ask another for data or actions, such as a set of HTTPS requests and responses. Every AWS service is controlled through its API, which the console, the CLI, and the SDKs call for you.Glossary entry to create topics and to produce and consume data.

MSK runs open-source versions of Apache Kafka, so existing applications, tools, and plugins work without changes to application code. That is the main reason to choose it: the company already uses Kafka, or wants to keep the Kafka ecosystem.

Cluster types

MSK Provisioned: you choose the number and size of brokers, at least one per AZ (Availability Zone)An isolated location within an AWS Region, made up of one or more data centers with their own power, cooling, and networking. The AZs in a Region are linked by fast, low-latency connections, so spreading resources across them protects against the failure of one site.Glossary entry, each in its own SubnetA range of IP addresses in a VPC, inside one Availability Zone. A subnet is public when its route table sends internet traffic to an internet gateway, and private when it does not.Glossary entry. Two broker types are available:
- Standard brokers: you size the storage.
- Express brokers: storage scales automatically with nothing to provision, and AWS states they offer up to three times more throughput per broker and scale up to 20 times faster than Standard brokers. They run only across three Availability Zones.
MSK Serverless: MSK provisions and scales capacity for you, and you pay for the throughput you use. Serverless clusters require IAM access control; Kafka ACL (Access control list)A list attached to a resource that names who may access it and what they may do. VPC subnets have network ACLs; Amazon S3 buckets also support ACLs, but new buckets disable them in favor of policies.Glossary entry are not supported.

Moving data in and out

MSK Connect runs Kafka Connect connectors for you. Source connectors bring data into topics, for example Debezium streaming database changes; sink connectors write topics to systems such as Amazon S3 or Amazon OpenSearch Service. Connectors scale automatically.
MSK Replicator replicates data between MSK Provisioned clusters in the same or different AWS RegionA separate geographic area where AWS runs its services, such as us-east-1 in Northern Virginia, made up of several Availability Zones. Most resources live in one Region; you choose Regions for latency, data residency, and the services you need.Glossary entry.

Security, resilience, performance, and cost

Security

Encryption at restEncrypting data where it is stored, such as on disks, in databases, or in backups, so that it cannot be read without the key. On AWS it usually uses keys in AWS KMS.Glossary entry is always on, with an AWS KMS key: the AWS managed key by default, or a Customer managed keyA KMS key that you create, own, and manage: you control its key policy, rotation, and deletion. AWS managed keys, in contrast, are created and managed for you by an AWS service.Glossary entry that you choose when you create the cluster.
Encryption in transitEncrypting data while it moves across a network, usually with TLS, so that it cannot be read or changed on the way.Glossary entry uses TLS (Transport Layer Security)The protocol that encrypts traffic between a client and a server and lets the server prove its identity with a certificate. It replaced SSL, though the older name is still common; HTTPS is HTTP over TLS.Glossary entry 1.2. Traffic between brokers is encrypted by default. For traffic between clients and brokers you choose TLS only (the default), TLS and plaintext, or plaintext only.
Client authentication and authorization:
- IAM access control: clients sign in with IAM, and IAM policies allow or deny Kafka actions such as writing to a topic. MSK Serverless always uses it.
- Mutual TLS: clients present certificates issued by AWS Private CA (Certificate authority)A trusted organization or service that issues digital certificates, vouching that a public key belongs to a named domain or organization. Browsers trust certificates signed by well-known CAs; AWS Private CA issues certificates for internal use.Glossary entry.
- SASL/SCRAM: user names and passwords stored in AWS Secrets Manager.
- With mutual TLS or SASL/SCRAM, Apache Kafka ACLs decide what each client may do.
Network: brokers run in subnets of your VPC, protected by Security groupA stateful virtual firewall for network interfaces, such as an EC2 instance's, whose rules can only allow traffic. Return traffic for allowed connections is allowed automatically, and a rule can refer to another security group instead of IP addresses.Glossary entry. Multi-VPC private connectivity (AWS PrivateLink) lets clients in other VPCs and AWS accounts in the same Region connect privately, with a cluster policy controlling which accounts may connect.

Resilience

Run a three-Availability Zone cluster with a replication factor of 3 and min.insync.replicas of 2, so topics stay writable while MSK replaces or patches one broker at a time.
Give clients a connection string with brokers from every Availability Zone.
MSK detects failed brokers and replaces them, reusing their storage where possible. For another Region, use MSK Replicator.

Performance

AWS recommends keeping broker CPU (user plus system) under 60%. To scale, move to a larger broker size or add brokers and partitions.
Alarm on disk usage (KafkaDataLogsDiskUsed) at 85%, and turn on storage auto scaling for Standard brokers. Express brokers need no storage management.

Cost

Provisioned: broker instance hours plus storage per GB-month; Express brokers also charge per GB written. Tiered storage moves older data on Standard brokers to a low-cost tier, so you can keep data longer without adding brokers.
Serverless: an hourly cluster charge, an hourly charge per partition, and per GB written and read.
MSK Connect: per worker capacity unit per hour.
Replication traffic between brokers in the cluster is not charged; normal data transfer charges apply to traffic in and out.

Monitoring

Amazon CloudWatch receives Kafka metrics from the brokers, including consumer lag. You can also scrape metrics with Prometheus (open monitoring).
MSK sends storage capacity alerts to the console, the AWS Health Dashboard, and Amazon EventBridge before Standard brokers run out of disk.
AWS CloudTrail records MSK API calls such as creating or updating clusters, and, when the cluster uses IAM access control, Kafka actions such as topic changes.
Broker logs can be delivered to CloudWatch Logs, Amazon S3, or Amazon Data Firehose.

Amazon MSK or Kinesis Data Streams?

Both store ordered streams of records that several consumers can read and replay. The difference is mostly about Kafka.

If the requirement is	Choose
Existing Kafka producers, consumers, or connectors must keep working	Amazon MSK
Migrate a self-managed Kafka cluster to a managed service	Amazon MSK
A managed stream with no Kafka knowledge needed	Kinesis Data Streams
Simply deliver streaming data to S3 or Redshift	Amazon Data Firehose

Reverse mindmap: when the question points here

Is it Amazon MSK?

Yes, when the question says
- The company already runs Apache Kafka leads to Amazon MSK
- Kafka applications must work without code changes leads to Amazon MSK
- Kafka with no brokers to size or scale leads to MSK Serverless
- Kafka Connect connectors, such as Debezium leads to MSK Connect
- Replicate Kafka data to another Region leads to MSK Replicator
Run it securely
- Control topic access with IAM policies leads to IAM access control
- Clients authenticate with certificates leads to Mutual TLS with AWS Private CA
- Client passwords kept out of code leads to SASL/SCRAM with Secrets Manager
- Clients in other VPCs or accounts leads to Multi-VPC private connectivity
Run it reliably and cheaply
- Survive the loss of one broker or AZ leads to Three AZs, replication factor 3
- Keep Kafka data longer at lower cost leads to Tiered storage
- Brokers running out of disk leads to Storage auto scaling
No, when the question says
- A managed stream and no mention of Kafka leads to Kinesis Data Streams
- Deliver streaming data to S3 with no code leads to Data Firehose
- A message queue between two components leads to Amazon SQS

Key takeaways

The clue for MSK is almost always "Apache Kafka".
MSK runs open-source Kafka, so existing Kafka code and tools work unchanged.
Data is always encrypted at rest; clients authenticate with IAM, mutual TLS, or SASL/SCRAM.
Three Availability Zones with a replication factor of 3 keep topics available during broker failures and patching.
MSK Serverless removes capacity planning; tiered storage keeps old data cheaply.
MSK Connect runs Kafka Connect connectors; MSK Replicator copies data between clusters and Regions.