Amazon Kinesis Data Streams

What it is

Amazon Kinesis Data Streams collects and stores large streams of data records in real time, such as application logs, clickstreams, and market data feeds. Applications called consumers read the records, usually within a second of their being written, and process them: they update dashboards, raise alerts, or pass results to other services.

The key idea is that the stream keeps the data for a retention period. Several applications can read the same records independently, and an application can go back and read records again.

Core concepts

Shard: a unit of capacity and an ordered sequence of records. Each shard accepts up to 1,000 records or 1 MB per second of writes, and supports up to 2 MB per second of reads.
Partition keyThe attribute that DynamoDB hashes to decide which partition stores an item. A primary key is a partition key alone, or a partition key with a sort key; keys with many distinct values spread the load evenly.Glossary entry: chosen by the producer for each record. Kinesis hashes it to decide which shard stores the record, so all records with the same key go to the same shard.
Data record: a sequence number, a partition key, and a data blob of up to 1 MB.
Retention period: 24 hours by default, and up to 365 days. Longer retention costs extra.

Because records with the same partition key land in the same shard, consumers read the records for one key, such as one customer or one device, in the order they were written.

Kinesis Data Streams. Producers write records to a data stream; records with partition key A go to shard 1 and records with key B to shard 2. Three consumers read the same stream: a Lambda function, a Kinesis Client Library application, and Amazon Data Firehose.

Capacity modes

On-demand: Kinesis manages the shards for you and you pay per GB written and read. A stream accommodates up to double its peak write throughput of the previous 30 days; traffic that more than doubles within 15 minutes can be ThrottlingLimiting how many requests a client can make in a period. A throttled request is rejected, often with HTTP status 429 or a throttling error, and should be retried later with backoff.Glossary entry briefly, so producers should retry. Choose it when traffic is unpredictable. An account-level On-demand Advantage option lowers the per-GB rates in exchange for a minimum commitment.
Provisioned: you choose the number of shards and pay for each shard per hour. Change the count with UpdateShardCount, or split a busy ("hot") shard and merge quiet ones. Choose it when traffic is predictable or you need control over how data is spread across shards.

You can switch a stream between the modes twice in 24 hours without disrupting producers or consumers.

Consumers

Many kinds of applications can read a stream: applications built with the Kinesis Client Library, AWS Lambda functions, Amazon Managed Service for Apache Flink, and Amazon Data Firehose.

By default, consumers share each shard's 2 MB per second of read throughput.
With enhanced Fan-outSending one message or event to many receivers at once so that each can process it in parallel, such as an Amazon SNS topic delivering to several SQS queues.Glossary entry, each registered consumer gets its own 2 MB per second per shard, so adding consumers does not slow the others down.

If a question describes several applications reading the same stream and falling behind, enhanced fan-out is the likely fix.

For real-time analytics on the stream itself, such as counts or averages over a sliding 5-minute window, use Amazon Managed Service for Apache Flink (formerly Amazon Kinesis Data Analytics). It runs Apache Flink applications with no servers to manage, reads Kinesis Data Streams or Amazon MSK, and computes windowed aggregations continuously as records arrive.

Security, resilience, performance, and cost

Security

IAM controls who can write to and read from each stream. A Resource-based policyA permissions policy attached to a resource, such as an S3 bucket policy or an SQS queue policy, that says which principals can use it, including principals in other accounts. Identity-based policies, in contrast, attach to users, groups, and roles.Glossary entry on a stream or consumer can grant access to another AWS account, for example an AWS Lambda function there.
Encryption at restEncrypting data where it is stored, such as on disks, in databases, or in backups, so that it cannot be read without the key. On AWS it usually uses keys in AWS KMS.Glossary entry: SSE (Server-side encryption)Encryption that a service applies when it stores your data and removes when you read it back. Amazon S3 encrypts every new object with SSE-S3 by default; SSE-KMS uses keys in AWS KMS instead.Glossary entry with an AWS KMS key, either the AWS managed key or a Customer managed keyA KMS key that you create, own, and manage: you control its key policy, rotation, and deletion. AWS managed keys, in contrast, are created and managed for you by an AWS service.Glossary entry. It applies only to data written after you turn it on. To share an encrypted stream with another account, use a customer managed key and grant that account access to it.
Private access: an interface VPC endpointA private connection from a VPC to an AWS service or another endpoint service without crossing the internet. Gateway endpoints, for Amazon S3 and DynamoDB, are route table targets; interface endpoints are network interfaces that use AWS PrivateLink.Glossary entry (AWS PrivateLink) lets producers and consumers in a VPC reach Kinesis without an internet gateway or NAT (Network address translation)Rewriting the IP addresses in packets as they pass through a device, most often so that many private addresses can share one public address to reach the internet.Glossary entry device. Endpoint policies can limit which streams are reachable through it.

Resilience

Kinesis synchronously replicates data across three AZ (Availability Zone)An isolated location within an AWS Region, made up of one or more data centers with their own power, cooling, and networking. The AZs in a Region are linked by fast, low-latency connections, so spreading resources across them protects against the failure of one site.Glossary entry.
The retention period is the safety net: if a consumer fails, it can resume from where it stopped as long as the records have not expired. Alarm on consumer lag well before it reaches the retention period.

Performance

Throughput errors (ProvisionedThroughputExceededException) mean a shard's limits were hit. Add shards, split hot shards, or choose partition keys that spread records evenly. A single partition key can never exceed one shard's limits.
Use enhanced fan-out when several consumers need full read throughput.

Cost

On-demand: per GB written and read (plus a per-stream hourly charge in On-demand Standard).
Provisioned: per shard-hour plus PUT payload units; merge shards you no longer need.
Retention beyond 24 hours, enhanced fan-out, and enhanced shard-level metrics all cost extra.

Monitoring

Amazon CloudWatch receives stream-level metrics every minute at no charge; enhanced shard-level metrics cost extra and help find hot shards.
Watch GetRecords.IteratorAgeMilliseconds (how far behind consumers are), WriteProvisionedThroughputExceeded and ReadProvisionedThroughputExceeded (throttling), and IncomingBytes compared with your shard capacity.
AWS CloudTrail records API (Application programming interface)A defined way for one program to ask another for data or actions, such as a set of HTTPS requests and responses. Every AWS service is controlled through its API, which the console, the CLI, and the SDKs call for you.Glossary entry calls such as creating streams or changing shard counts.

Kinesis Data Streams or Data Firehose?

	Kinesis Data Streams	Amazon Data Firehose
Main job	Store a stream for real-time processing	Deliver a stream to a destination
Consumers	Your own applications, Lambda, Flink, and others	None to write; Firehose delivers for you
Data kept for replay	Yes, for the retention period	No
Delivery timing	Records readable within about a second	Buffered by size or time before delivery
Capacity	On-demand or provisioned shards	Fully managed

The two also work together: a Firehose stream can read from a Kinesis data stream and deliver the records to Amazon S3, while other applications process the same stream in real time.

Kinesis Data Streams can now also deliver records directly to Amazon S3 and to Apache Iceberg tables. In scenario questions, a requirement to "deliver streaming data to a destination with no code" still points first to Data Firehose.

Reverse mindmap: when the question points here

Is it Kinesis Data Streams?

Yes, when the question says
- Process data in real time with custom logic leads to Kinesis Data Streams
- Several applications read the same data leads to Kinesis Data Streams
- Consumers compete for read throughput leads to Enhanced fan-out
- Replay or reprocess recent records leads to Longer retention period
- Windowed aggregations on the stream in real time leads to Managed Service for Apache Flink as the consumer
- Keep order per device or customer leads to That ID as the partition key
Run it securely
- Encrypt records at rest with your own key leads to SSE with a customer managed KMS key
- Producers in a VPC without internet access leads to Interface VPC endpoint
- A consumer in another account leads to Resource-based policy
Run it well
- Unpredictable traffic, no capacity planning leads to On-demand mode
- Write throttling on one shard leads to Split the hot shard or change partition keys
- Consumers falling behind leads to Alarm on iterator age
No, when the question says
- Load streaming data into S3 or Redshift with no code leads to Amazon Data Firehose
- Existing Apache Kafka applications leads to Amazon MSK
- Decouple two components with a queue leads to Amazon SQS

Key takeaways

A shard is the unit of capacity: 1 MB or 1,000 records per second in, 2 MB per second out.
Records with the same partition key go to the same shard and stay in order.
Retention defaults to 24 hours and can be extended to 365 days, which makes replay possible.
Encrypt with KMS, reach it privately through an interface VPC endpoint, and share it with resource-based policies.
On-demand for unpredictable traffic, provisioned for predictable traffic; watch iterator age and throttling metrics.
Choose Data Streams to process; choose Data Firehose to deliver.