Amazon Data Firehose

What it is

Amazon Data Firehose delivers streaming data to a destination for you. Producers send records to a Firehose stream, and Firehose buffers them and writes them to Amazon S3, Amazon Redshift, Amazon OpenSearch Service, Splunk, Apache Iceberg tables, or an HTTP (Hypertext Transfer Protocol)The protocol that web browsers and APIs use to send requests, such as GET and POST, and receive responses. On its own it is not encrypted; HTTPS adds TLS.Glossary entry endpoint. There are no consumer applications to write and no capacity to manage.

Firehose was previously called Amazon Kinesis Data Firehose.

Where the data comes from

Direct PUT: applications send records with the AWS SDK (Software development kit)A set of libraries for a programming language that wraps a service's API and handles details such as signing requests and retrying. AWS SDKs exist for Python (Boto3), JavaScript, Java, Go, and other languages.Glossary entry or the Kinesis agent.
A Kinesis data stream: Firehose reads an existing stream and delivers its records, while other applications process the same stream.
An Amazon MSK topic: Firehose reads a Kafka topic and delivers it, with no connector to run.
AWS services: Amazon CloudWatch Logs, Amazon EventBridge (CloudWatch Events), and AWS IoT can send data to a Firehose stream. Some of them can only send to a Firehose stream in the same AWS RegionA separate geographic area where AWS runs its services, such as us-east-1 in Northern Virginia, made up of several Availability Zones. Most resources live in one Region; you choose Regions for latency, data residency, and the services you need.Glossary entry.

Buffering: near real time, not real time

Firehose collects incoming records into a buffer and delivers it when the buffer reaches a size or a time limit, whichever comes first. For Amazon S3, the defaults are 5 MB and 300 seconds, and the interval can be set up to 900 seconds. If delivery falls behind, Firehose raises the buffer size to catch up.

Buffering produces fewer, larger files, which suits analytics. It also means data typically arrives seconds to minutes after it was sent. If a question needs each record processed within a second, look at Kinesis Data Streams instead.

For an Amazon Redshift destination, Firehose first writes the data to Amazon S3 and then runs a Redshift COPY command to load it.

Changing data on the way

Transform with AWS Lambda: Firehose invokes your Lambda function on each buffered batch, for example to filter records or convert them to JSON (JavaScript Object Notation)A text format for structured data made of name-and-value pairs and lists, such as {"bucket": "logs"}. IAM policies, most AWS API responses, and many CloudFormation templates are written in JSON.Glossary entry.
Convert to Apache ParquetAn open-source columnar file format for analytics that stores compressed columns along with the schema. Querying Parquet files with Amazon Athena scans less data than CSV or JSON, so it is faster and cheaper.Glossary entry or ORC: Firehose converts JSON records to a Columnar storageStoring data column by column instead of row by row, so that analytical queries read only the columns they need, and similar values compress well. Amazon Redshift and the Parquet and ORC file formats store data this way.Glossary entry before storing them in Amazon S3, using a schema from the AWS Glue Data Catalog. Columnar files save space and make queries faster. Other input formats, such as CSV, need a Lambda transformation to JSON first.
Compress: GZIP, Snappy, or Zip for Amazon S3.
Dynamic partitioning: Firehose groups records by keys in the data, such as customer_id, and delivers each group to its own S3 prefix. Queries in Amazon Athena then scan only the partitions they need, which is faster and cheaper.
Back up source records: keep a copy of the original data, or only the records that failed, in an S3 bucket.

Data Firehose delivery. Records arrive by Direct PUT, from a Kinesis data stream, or from logs, into a Firehose stream that buffers them until a size or time limit. Each batch can go to an optional Lambda function for transformation and back. Firehose delivers the batches to Amazon S3, and a COPY command loads them from S3 into Redshift.

Security, resilience, performance, and cost

Security

IAM role: Firehose assumes a role you provide to write to the destination and to the backup bucket. Grant it only those resources.
Encryption in transitEncrypting data while it moves across a network, usually with TLS, so that it cannot be read or changed on the way.Glossary entry: Firehose encrypts all data in transit with TLS (Transport Layer Security)The protocol that encrypts traffic between a client and a server and lets the server prove its identity with a certificate. It replaced SSL, though the older name is still common; HTTPS is HTTP over TLS.Glossary entry.
Encryption at restEncrypting data where it is stored, such as on disks, in databases, or in backups, so that it cannot be read without the key. On AWS it usually uses keys in AWS KMS.Glossary entry: for Direct PUT and AWS service sources, turn on SSE (Server-side encryption)Encryption that a service applies when it stores your data and removes when you read it back. Amazon S3 encrypts every new object with SSE-S3 by default; SSE-KMS uses keys in AWS KMS instead.Glossary entry with an AWS KMS key, which can be a Customer managed keyA KMS key that you create, own, and manage: you control its key policy, rotation, and deletion. AWS managed keys, in contrast, are created and managed for you by an AWS service.Glossary entry. When the source is a Kinesis data stream, the stream's own encryption protects the data, and Firehose does not store it unencrypted. At the destination, objects written to Amazon S3 can use SSE-KMS.
Private access: producers in a VPC can reach Firehose through an interface VPC endpointA private connection from a VPC to an AWS service or another endpoint service without crossing the internet. Gateway endpoints, for Amazon S3 and DynamoDB, are route table targets; interface endpoints are network interfaces that use AWS PrivateLink.Glossary entry (AWS PrivateLink). Firehose can also deliver into a VPC, for example to an OpenSearch Service domain in SubnetA range of IP addresses in a VPC, inside one Availability Zone. A subnet is public when its route table sends internet traffic to an internet gateway, and private when it does not.Glossary entry.

Resilience

If Amazon S3 is unavailable, Firehose keeps retrying. With Direct PUT it stores records for up to 24 hours, after which undelivered data is lost; with a Kinesis or MSK source, it retries for as long as the stream keeps the records.
For Redshift and OpenSearch Service you set a retry duration (0 to 7,200 seconds). For OpenSearch Service, Splunk, and HTTP endpoints, records that still fail are written to the S3 backup bucket so you can load them later. For Redshift, Firehose skips the failed batch and writes a manifest of the skipped S3 objects to the errors/ folder, which you use to backfill manually.

Performance and limits

A record can be up to 1,000 KiB, and PutRecordBatch takes up to 500 records or 4 MiB per call.
With Direct PUT, each Firehose stream has a throughput quota (for example 5 MiB per second in US East (N. Virginia), US West (Oregon), and Europe (Ireland), and 1 MiB per second in most other Regions). Firehose raises it automatically when ThrottlingLimiting how many requests a client can make in a period. A throttled request is rejected, often with HTTP status 429 or a throttling error, and should be retried later with backoff.Glossary entry occurs, and you can request a higher quota for expected bursts. Reading from a Kinesis data stream avoids this quota.
Dynamic partitioning supports 500 active partitions per stream by default.

Cost

You pay per GB ingested. For Direct PUT and Kinesis sources, each record is rounded up to the next 5 KB, so sending many tiny records costs more than batching them into fewer, larger ones.
Format conversion, dynamic partitioning, and delivery into a VPC each add their own charges.
There are no set-up fees or minimum commitments.

Monitoring

Amazon CloudWatch metrics are free. Watch IncomingBytes and ThrottledRecords on the way in, and DeliveryToS3.Success and DeliveryToS3.DataFreshness (the age of the oldest record not yet delivered) on the way out. Separate metrics report KMS key errors.
Turn on error logging to CloudWatch Logs to see why deliveries or transformations fail, and use AWS CloudTrail for API (Application programming interface)A defined way for one program to ask another for data or actions, such as a set of HTTPS requests and responses. Every AWS service is controlled through its API, which the console, the CLI, and the SDKs call for you.Glossary entry calls.

Firehose or Kinesis Data Streams?

If the requirement is	Choose
Deliver streaming data to S3, Redshift, or OpenSearch with the least effort	Data Firehose
Process each record within about a second with custom code	Kinesis Data Streams
Several applications read the same data, or records can be replayed	Kinesis Data Streams
Both: process in real time and archive to S3	Kinesis Data Streams, with Firehose reading the stream

Reverse mindmap: when the question points here

Is it Data Firehose?

Yes, when the question says
- Load streaming data into S3 or Redshift with no code leads to Data Firehose
- Deliver logs to OpenSearch Service or Splunk leads to Data Firehose
- Convert streaming JSON to Parquet or ORC leads to Record format conversion
- Store streaming data by customer or date for cheaper queries leads to Dynamic partitioning
- Transform records before delivery leads to Lambda transformation
Run it securely and reliably
- Encrypt Direct PUT data with your own key leads to SSE with a customer managed KMS key
- Producers in a VPC without internet access leads to Interface VPC endpoint
- Keep records that failed to load leads to S3 backup bucket
- Data must survive a long destination outage leads to Read from a Kinesis data stream
Run it cheaply
- Many tiny records leads to Batch them; each record is billed as at least 5 KB
No, when the question says
- Each record processed within a second leads to Kinesis Data Streams
- Several consumers or replaying records leads to Kinesis Data Streams
- Existing Apache Kafka applications leads to Amazon MSK

Key takeaways

Firehose delivers; it does not store data for consumers to read later.
Buffering by size or time makes delivery near real time, not real time.
Lambda transformation, Parquet or ORC conversion, compression, and dynamic partitioning all happen on the way to the destination.
Failed records go to an S3 backup bucket (for Redshift, a manifest of skipped objects in errors/); with Direct PUT, undelivered data is kept for up to 24 hours.
Batch small records: each one is billed as at least 5 KB.
Firehose can read from a Kinesis data stream or an MSK topic.