What Is Amazon Kinesis? Data Streams vs Firehose, Pricing With Examples, Python and Lambda Tutorial, and ProvisionedThroughputExceededException Fix

Amazon Kinesis is AWS's family of services for streaming data: records that never stop arriving, such as website clicks, orders, application logs, sensor readings and GPS pings. Kinesis Data Streams stores that flow for 24 hours up to 365 days so several applications can read it in real time; Amazon Data Firehose (formerly Kinesis Data Firehose) delivers it into S3, Redshift, OpenSearch, Snowflake, Splunk or Iceberg tables with no code; Managed Service for Apache Flink (formerly Kinesis Data Analytics) runs real-time analytics on it; and Kinesis Video Streams handles camera video. What you pay depends on the capacity mode you pick. On-demand Standard costs $0.04 per stream-hour (about $28.80 a month per stream) plus $0.08 per GB written and $0.04 per GB read. Provisioned mode costs $0.015 per shard-hour (about $10.80 a month per shard, and each shard takes 1 MB or 1,000 records a second) plus $0.014 per million PUT payload units. The newer On-demand Advantage mode cuts the data rates to $0.032 and $0.016 per GB with no stream fee, but commits your account to 25 MiB/s each way, about $3,037 a month at minimum. Firehose costs $0.029 per GB, with every record rounded up to 5 KB, and Kinesis Data Streams is not part of the AWS Free Tier. Below: what each Kinesis service does, Data Streams vs Firehose, every price with worked examples (including why a steady stream can cost almost six times more on-demand), a Python producer and consumer, Lambda triggers, Firehose to S3, the new streaming tables delivery, Kinesis vs Kafka and SQS, and how to fix ProvisionedThroughputExceededException for good.

The furniture store's owner wanted a screen in the back office that showed what was happening right now: which sofas people were looking at on the website, what the in-store tablets were selling, and where the three delivery vans were. The nightly sales report arrived the next morning, which was fine for accounting and useless for deciding whether to move staff to the showroom during a Saturday rush. Jake, who had dropped by to fix the showroom Wi-Fi, looked at the plan on the whiteboard and asked why the website could not just email the owner every time someone clicked. Ethan laughed, wrote "12,000 clicks an hour" next to the question, and opened the Kinesis console.

New to event-driven AWS? The cards below cover Lambda, SQS, EventBridge, Athena and SNS, the services you will most often see next to Kinesis.

What is Amazon Kinesis?

Most data used to move in batches: a file every night, a report every morning. Streaming data is different. It arrives continuously, one small record at a time, and it is most valuable in the first few seconds: a fraud signal, a server error, a customer abandoning a cart, a van stuck in traffic. Amazon Kinesis is the set of AWS services built to collect, store, move and analyze that kind of data as it arrives, with no servers for you to run.

"Kinesis" is a family name, and the four members do quite different jobs:

• Amazon Kinesis Data Streams (KDS): a durable, ordered log of records. Producers write records in; the stream keeps them for 24 hours by default (up to 365 days); any number of consumer applications read them, each at its own pace, and can replay them. This is the core service, and most of this page is about it.
• Amazon Data Firehose: a delivery service. It takes records from your producers or from a data stream, optionally transforms them, batches them, and writes them to a destination such as S3, Redshift, OpenSearch, Snowflake, Splunk, Apache Iceberg tables or an HTTP endpoint. AWS renamed it from Kinesis Data Firehose in 2024, which is why you will see both names.
• Amazon Managed Service for Apache Flink: runs Apache Flink applications (Java, Python or SQL in Studio notebooks) that analyze streams in real time, such as rolling five-minute sales totals or anomaly detection. It used to be called Kinesis Data Analytics; the old Kinesis Data Analytics for SQL applications product stopped accepting new applications on October 15, 2025, and existing SQL applications were scheduled for deletion from January 27, 2026, so Flink is now the way to run streaming SQL on AWS.
• Amazon Kinesis Video Streams (KVS): ingests, stores and plays back video from cameras and phones, including live two-way WebRTC streams. It shares the Kinesis name but works differently from the other three, so it has its own section below.

What Kinesis is used for: clickstream and app analytics, real-time dashboards, log and metrics collection, IoT and vehicle telemetry, fraud and anomaly detection, change data capture from databases, feeding machine learning features, and building a data lake from events. If your data shows up as a continuous flow and someone needs it in seconds rather than tomorrow, Kinesis is a candidate.

Jake: "So it's a conveyor belt. Everything gets dropped on it, and anyone who cares can watch it go by."

Ethan: "A conveyor belt that remembers the last day of everything, so the person who was on a break can catch up."

How Kinesis Data Streams works: shards, partition keys and consumers

1. A producer writes a record with PutRecord or PutRecords (up to 500 records per call). A record is a blob of data up to 10 MiB, usually a small JSON event of a few hundred bytes, plus a partition key you choose, such as an order ID, user ID or device ID.
2. Kinesis hashes the partition key (MD5, to a 128-bit number) and uses it to pick a shard. Every shard owns a range of hash values, so the same partition key always goes to the same shard, and records for one key stay in order.
3. The shard stores the record with a unique, increasing sequence number, replicated across three Availability Zones, for the stream's retention period.
4. Consumers read the records shard by shard, in order. A standard consumer polls with GetRecords and shares the shard's 2 MB per second of read capacity with the other standard consumers; an enhanced fan-out consumer gets its own dedicated 2 MB per second per shard, pushed over HTTP/2.
5. Each consumer remembers its position (a checkpoint) so it can stop, restart and continue where it left off, or rewind to reprocess old records.
6. Records expire when they are older than the retention period; nothing is deleted when it is read, which is the key difference from a queue.

Each shard in provisioned mode gives you a fixed amount of capacity: 1 MB per second or 1,000 records per second of writes, and 2 MB per second of reads shared by standard consumers, with at most 5 GetRecords calls per second. In on-demand mode, Kinesis manages the shards for you, but the same per-shard limits still exist underneath, which is why a single very busy partition key can still be throttled.

Who writes to a stream (producers): your code with the AWS SDK; the Kinesis Producer Library (KPL), which batches and aggregates many small records into fewer, larger ones; the Kinesis Agent, which tails log files on Linux servers; Kinesis Agent for Microsoft Windows, which ships Windows event logs, IIS logs and performance counters; CloudWatch Logs subscription filters; AWS IoT rules; DynamoDB (change data capture into a stream); and AWS DMS for database changes.

Who reads from a stream (consumers): AWS Lambda functions with an event source mapping; applications built with the Kinesis Client Library (KCL); Firehose for delivery; Managed Service for Apache Flink; AWS Glue streaming jobs; and Spark or other tools that speak the Kinesis API.

Kinesis architecture in one sentence: producers on the left push records into a stream made of shards; the stream keeps them in order for hours or days; consumers on the right read them independently, usually Lambda for reactions, Firehose for archiving, and Flink for analytics.

Kinesis Data Streams vs Data Firehose

This is the comparison most people search for first, because the two overlap. The short version: Data Streams is storage your applications read; Firehose is a pipe that delivers to a destination.

	Kinesis Data Streams	Amazon Data Firehose
Main job	Store an ordered stream for your own consumers	Deliver data to S3, Redshift, OpenSearch, Snowflake, Splunk, Iceberg, HTTP
Who reads it	Your apps: Lambda, KCL, Flink, Firehose, many at once	Only its destination
Latency	About 70 to 200 milliseconds to consumers	Seconds to minutes, set by the buffer (up to 900 seconds)
Replay old data	Yes, within retention (24 hours to 365 days)	No; data is delivered, then gone from Firehose
Ordering	Per shard, by partition key	Not guaranteed at the destination
Scaling	On-demand (automatic) or shards you manage	Fully automatic
Transformations	None; consumers do the work	Lambda transforms, JSON to Parquet or ORC, dynamic partitioning, compression
Price model	Per stream-hour or shard-hour plus data	Per GB ingested ($0.029, records rounded up to 5 KB)
Code to write	Producers and consumers	Producers only, or none if the source is another AWS service

Use Data Streams when more than one application needs the same events, when you need sub-second reactions, ordering per key, or the ability to replay yesterday's data after fixing a bug. Use Firehose when the goal is "get these events into S3 (or Redshift, or OpenSearch) in files I can query", with no consumer code. Use both, which is very common: producers write to a data stream, Lambda reacts in real time, and a Firehose stream reads the same data stream and archives everything to S3 for querying with Athena. That is exactly what the furniture store built.

Capacity modes: On-demand Standard, On-demand Advantage and Provisioned

Every data stream runs in one of three modes, and the mode decides both how the stream scales and how you are billed. Picking the right one is the biggest cost decision you make with Kinesis.

• On-demand Standard: no capacity planning. A new stream starts with 4 MB/s of write and 8 MB/s of read capacity and scales automatically to double the highest write rate it has seen in the past 30 days, up to 10 GB/s of writes in US East (N. Virginia), US West (Oregon) and Europe (Ireland), or 200 MB/s by default elsewhere. You pay a fixed $0.04 per stream-hour plus per-GB charges. Throttling can still happen if your traffic more than doubles its previous peak within about 15 minutes.
• On-demand Advantage: an account-level setting that switches every on-demand stream in a Region to cheaper per-GB rates with no per-stream fee, free enhanced fan-out premium, up to 50 enhanced fan-out consumers per stream, and warm throughput, which lets you pre-scale a stream before a known spike (a sale, a product launch) at no extra charge. The catch: turning it on commits the account to at least 25 MiB/s of writes and 25 MiB/s of reads across all on-demand streams in the Region, billed as a shortfall if you use less, and you must keep it on for at least 24 hours.
• Provisioned: you choose the number of shards and pay per shard-hour. Cheapest by far for steady, predictable traffic, and it gives you fine control (split one hot shard, merge cold ones), but you must watch the metrics and scale before you hit the limits.

	On-demand Standard	On-demand Advantage	Provisioned
Fixed charge	$0.04 per stream-hour (about $28.80 a month)	None, but a 25 MiB/s minimum each way	$0.015 per shard-hour (about $10.80 a month)
Data written	$0.08 per GB	$0.032 per GB	$0.014 per million 25 KB PUT units
Data read	$0.04 per GB ($0.05 with enhanced fan-out)	$0.016 per GB, fan-out included	Included for standard consumers
Scaling	Automatic, to 2x previous peak	Automatic, plus warm throughput	You change the shard count
Fan-out consumers	Up to 20 per stream	Up to 50 per stream	Up to 20 per stream
S3 and streaming tables delivery	Yes	Yes	No
Best for	Spiky or unknown traffic, new projects	Large accounts above about 10 MB/s, many consumers or streams	Steady, predictable traffic

You can switch a stream between on-demand and provisioned twice in any 24 hours, with no downtime: the stream shows Updating and keeps accepting reads and writes. A good pattern is to start a new stream in on-demand mode, watch its traffic for a few weeks, and move it to provisioned once you know its steady rate.

Amazon Kinesis Data Streams pricing

US East (N. Virginia) list prices in October 2026. Kinesis Data Streams is not included in the AWS Free Tier, so even a test stream is billed from its first hour.

What you pay for	On-demand Standard	On-demand Advantage	Provisioned
Stream or shard	$0.04 per stream-hour	None	$0.015 per shard-hour
Writes	$0.08 per GB (records rounded up to 1 KB)	$0.032 per GB	$0.014 per million PUT payload units (25 KB each)
Reads, standard consumers	$0.04 per GB	$0.016 per GB	Included
Reads, enhanced fan-out	$0.05 per GB	$0.016 per GB	$0.015 per consumer-shard-hour + $0.013 per GB
Retention up to 7 days	$0.10 per GB-month	$0.023 per GB-month	$0.020 per shard-hour
Retention beyond 7 days (to 365)	$0.023 per GB-month	$0.023 per GB-month	$0.023 per GB-month, + $0.021 per GB read back
Delivery to S3 (raw)	$0.0275 per GB	$0.011 per GB	Not available
Delivery to streaming tables (Iceberg)	$0.035 per GB	$0.014 per GB	Not available

How the provisioned PUT payload unit works: every record is counted in 25 KB chunks, so a 2 KB record is one unit, a 30 KB record is two units, and 1,000 small records a second for a month is 2.59 billion units, about $36. How on-demand rounding works: each record is rounded up to the nearest 1 KB, so a 200-byte record is billed as 1 KB; packing several small events into one record cuts the bill.

What else can appear on your bill: CloudWatch charges for enhanced shard-level metrics, KMS requests if you turn on server-side encryption (see our KMS costs breakdown), the DynamoDB lease table that the KCL creates for each consumer application, Lambda invocations for Lambda consumers, and data transfer for producers or consumers in another Region. Producers and consumers in the same Region pay no data transfer to or from the stream.

Amazon Data Firehose pricing

• Ingestion from Direct PUT or a Kinesis data stream: $0.029 per GB for the first 500 TB a month, with lower rates above that. Each record is rounded up to the next 5 KB, so a 1 KB record is billed as 5 KB.
• From Amazon MSK: $0.055 per GB, with no 5 KB rounding.
• From vended logs (such as CloudWatch Logs or VPC flow logs sent straight to Firehose): $0.13 per GB, no rounding.
• Delivery to Apache Iceberg tables: $0.045 per GB from a Kinesis data stream; to Snowflake: $0.071 per GB.
• Format conversion (JSON to Parquet or ORC): $0.018 per GB converted.
• Dynamic partitioning: $0.020 per GB processed, $0.005 per 1,000 S3 objects delivered, and $0.07 per hour of JQ processing if you use JQ expressions.
• Delivery into a VPC (for example a private OpenSearch domain): $0.01 per GB plus $0.01 per hour per Availability Zone.
• CloudWatch Logs decompression: $0.00325 per GB.

The 5 KB rounding is the line that surprises people. If your events are tiny, Firehose bills you as if they were 5 KB each, which can multiply the cost by five or more. The fix is to send fewer, bigger records: batch several events into one record (newline-separated JSON works well) up to about 5 KB, and the rounding disappears.

Worked examples: what Kinesis really costs

All examples use a 30-day month (720 hours, 2,592,000 seconds) and 1 GB = 1,024 MB, the same way AWS's own calculator counts.

Example 1: a small app sending 50 events a second. Each event is 2 KB, so the stream takes 100 KB a second, about 247 GB a month, read by one consumer.

• On-demand Standard: $28.80 for the stream + 247.19 GB x $0.08 ($19.78) + 247.19 GB x $0.04 read ($9.89) = $58.47 a month.
• Provisioned with one shard: $10.80 + 129.6 million PUT units x $0.014 per million ($1.81) = $12.61 a month.

Example 2: the furniture store. Website clicks, tablet orders and van GPS pings averaged 500 events a second at about 1 KB each, roughly 1,236 GB a month, read by two consumers: a Lambda function for the live dashboard and Firehose for the S3 archive.

• On-demand Standard: $28.80 + 1,235.96 GB x $0.08 ($98.88) + two consumers reading 2 x 1,235.96 GB x $0.04 ($98.88) = $226.56 a month.
• Provisioned with two shards (one would do, the second is headroom for Saturday rushes): $21.60 + 1,296 million PUT units x $0.014 per million ($18.14) = $39.74 a month, almost six times cheaper for the same data.

Ethan: "Same stream, same data. One costs $226, the other $40."

Jake: "Then why would anyone use on-demand?"

Ethan: "Because on the day you launch, you have no idea whether it's 50 events a second or 5,000. On-demand finds out for you. Then you switch."

Example 3: the Firehose archive, done wrong and done right. Firehose reading those 500 one-KB records a second bills each one as 5 KB: 6,179.8 GB x $0.029 = $179.21 a month. When the store's producers packed five events into each record (100 records a second of about 5 KB), the same data cost 1,235.96 GB x $0.029 = $35.84 a month, and the stream's PUT units fell from $18.14 to $3.63 as a bonus. Adding Parquet conversion for cheaper Athena queries costs another 1,235.96 GB x $0.018 = $22.25.

Example 4: when On-demand Advantage pays off. The minimum commitment is 25 MiB/s in and out: 63,281.25 GB x $0.032 ($2,025.00) + 63,281.25 GB x $0.016 ($1,012.50) = $3,037.50 a month, with no stream fees. The same traffic on On-demand Standard with one consumer costs $5,062.50 + $2,531.25 = $7,593.75 plus stream-hours. With one consumer, Advantage becomes cheaper than Standard once your account averages about 10 MiB/s of writes (about 25,300 GB a month) across its on-demand streams, sooner if you run many streams or several consumers.

Example 5: keeping data for a week, or a month. Extending the store's two provisioned shards from 24 hours to 7 days costs 2 x 720 x $0.020 = $28.80 a month. Keeping 30 days for replay adds long-term storage for the 23 days beyond the first week: about 948 GB x $0.023 = $21.79 a month, plus $0.021 per GB if you ever read that old data back.

Example 6: a dedicated enhanced fan-out consumer. On the two provisioned shards: 2 x 720 x $0.015 ($21.60) + 1,235.96 GB x $0.013 ($16.07) = $37.67 a month per fan-out consumer.

Example 7: forgotten test streams. Ten on-demand streams left over from a hackathon, with no traffic at all, cost 10 x $28.80 = $288 a month. Ten one-shard provisioned streams would cost $108. Either way, delete what you are not using, and set a budget alert.

Example 8: Managed Service for Apache Flink. Flink is billed at $0.11 per Kinesis Processing Unit (KPU) hour, and AWS adds one extra KPU per application for orchestration, so the smallest always-on application with one KPU costs about 2 x 720 x $0.11 = $158.40 a month, plus $0.10 per GB-month of running application storage.

Kinesis Data Streams tutorial: create a stream and send data with Python

1. Open the Kinesis console, choose Data streams, then Create data stream.
2. Name it, for example store-events, and choose On-demand while you are learning your traffic, or Provisioned with one or two shards for a small, steady workload.
3. Create the stream. It moves from Creating to Active in under a minute.
4. Turn on encryption on the stream's configuration tab with the AWS managed key (aws/kinesis) or your own KMS key, and adjust the retention period if 24 hours is not enough.
5. Give producers and consumers IAM permissions: producers need kinesis:PutRecord and kinesis:PutRecords; consumers need kinesis:GetRecords, kinesis:GetShardIterator, kinesis:DescribeStreamSummary and kinesis:ListShards (plus the KMS key permissions if encryption uses your own key).
6. Send a test record with the CLI, then run the Python producer below.
7. Read it back with the simple consumer below, the console's Data viewer tab, or a Lambda trigger.

# On-demand stream
aws kinesis create-stream --stream-name store-events --stream-mode-details StreamMode=ON_DEMAND

# Or a provisioned stream with two shards
aws kinesis create-stream --stream-name store-events --shard-count 2

# Send one record (AWS CLI v2 needs the raw-in-base64-out flag for plain text)
aws kinesis put-record --stream-name store-events --partition-key order-1001 \
  --data '{"type":"order","sku":"SOFA-3S-GRY","amount":899}' --cli-binary-format raw-in-base64-out

# Check the stream's mode, shard count and status
aws kinesis describe-stream-summary --stream-name store-events

A Python producer that handles throttling properly. PutRecords can partly succeed: some records in a batch are stored while others fail with ProvisionedThroughputExceededException or an internal error, and the call itself still returns normally. You must check FailedRecordCount and resend only the failed records, or you silently lose data:

import json, random, time
import boto3

kinesis = boto3.client("kinesis", region_name="us-east-1")
to_json = json.JSONEncoder().encode

def send_events(events, stream="store-events"):
    records = [{"Data": to_json(e).encode(), "PartitionKey": e["order_id"]} for e in events]
    for attempt in range(8):
        resp = kinesis.put_records(StreamName=stream, Records=records)
        if resp["FailedRecordCount"] == 0:
            return
        # keep only the records that failed, then back off with jitter and retry them
        records = [r for r, res in zip(records, resp["Records"]) if "ErrorCode" in res]
        time.sleep(random.uniform(0, min(5, 0.1 * 2 ** attempt)))
    raise RuntimeError(f"{len(records)} records still failing after retries")

send_events([{"order_id": f"order-{n}", "type": "click", "sku": "SOFA-3S-GRY"} for n in range(400)])

A simple consumer for learning. This reads one shard from the newest record onward. Real applications should use Lambda or the KCL, which handle every shard, checkpoints, resharding and failures for you:

import time
import boto3

kinesis = boto3.client("kinesis", region_name="us-east-1")
shard_id = kinesis.list_shards(StreamName="store-events")["Shards"][0]["ShardId"]
iterator = kinesis.get_shard_iterator(StreamName="store-events", ShardId=shard_id,
                                      ShardIteratorType="LATEST")["ShardIterator"]
while iterator:
    resp = kinesis.get_records(ShardIterator=iterator, Limit=1000)
    for record in resp["Records"]:
        print(record["SequenceNumber"], record["PartitionKey"], record["Data"].decode())
    iterator = resp["NextShardIterator"]
    time.sleep(1)  # stays well under the 5 GetRecords calls per second per shard

Use TRIM_HORIZON instead of LATEST to start from the oldest record still in the stream, or AT_TIMESTAMP to replay from a point in time, which is how you reprocess a day of events after fixing a bug.

Testing without writing a producer: the Amazon Kinesis Data Generator (KDG), an open-source AWS Labs web page, sends fake records built from a template (random names, prices, timestamps) to a data stream or Firehose stream at a rate you choose. Its CloudFormation template creates a Cognito user so the page can sign in to your account. It is the quickest way to see a dashboard or a Firehose delivery working before your real producers exist.

The Kinesis Client Library, Producer Library and Agent

• Kinesis Client Library (KCL): the standard way to build a consumer application in Java (with Python, Node.js, .NET and Ruby supported through its multi-language daemon). It discovers shards, spreads them across your workers, records checkpoints, handles resharding and moves work when a worker fails. It keeps its leases and checkpoints in a DynamoDB table it creates per application, which appears on your DynamoDB bill. KCL 3.x, released in late 2024, balances shards by worker CPU use, which lowers the compute you need.
• Kinesis Producer Library (KPL): a high-throughput producer for Java applications. It aggregates many small user records into one Kinesis record (fewer records against the 1,000-a-second shard limit and fewer billed units) and collects records into PutRecords batches, with automatic retries. Consumers must de-aggregate: the KCL does this automatically, and Lambda functions use AWS's small de-aggregation library.
• Kinesis Agent: a Java application for Linux servers that watches log files and sends new lines to a data stream or Firehose, with optional conversion of log lines to JSON, retries and checkpoints.
• Kinesis Agent for Microsoft Windows: the Windows counterpart, which collects Windows event logs, IIS and application log files and performance counters, and sends them to Kinesis Data Streams, Firehose or CloudWatch. Useful when your servers are Windows and your log analytics live in AWS.

All four are on GitHub under the awslabs organization, along with the Kinesis Data Generator and many example consumers.

Kinesis with Lambda: triggers, batch settings and poison records

A Lambda trigger is the simplest way to react to every record. Lambda's event source mapping polls each shard for you (about once a second), batches records, invokes your function, and only moves on when the batch succeeds, which keeps records in order per shard.

aws lambda create-event-source-mapping --function-name store-dashboard \
  --event-source-arn arn:aws:kinesis:us-east-1:111122223333:stream/store-events \
  --starting-position LATEST --batch-size 500 --maximum-batching-window-in-seconds 5 \
  --parallelization-factor 2 --bisect-batch-on-function-error --maximum-retry-attempts 3 \
  --function-response-types ReportBatchItemFailures \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:us-east-1:111122223333:store-events-failed"}}'

• Batch size and batching window: up to 10,000 records per batch, and a window of up to 300 seconds to wait for a fuller batch; bigger batches mean fewer invocations and a lower Lambda bill.
• Parallelization factor (1 to 10): how many batches from one shard Lambda processes at once, still in order per partition key. Raise it when one function cannot keep up with a shard.
• The poison record problem: by default, if your function throws an error, Lambda retries the same batch until it succeeds or the records expire, and that shard stops moving. Set bisect batch on function error, a maximum retry attempts value, a maximum record age, and an on-failure destination (an SQS queue or SNS topic) that receives details of the records that were skipped.
• Partial batch responses: with ReportBatchItemFailures, your function returns the sequence number of the first record that failed, so Lambda retries from there instead of the whole batch.
• Watch the IteratorAge metric: it shows how far behind the newest record your function is. A climbing iterator age means the function is too slow, erroring, or throttled; our Lambda concurrency math explains the throttling side.
• Several Lambda functions on one stream: each standard event source mapping polls every shard, and all of them share the 5 reads per second and 2 MB per second per shard. With more than two or three consumers, give the busy ones an enhanced fan-out consumer and point the event source mapping at it.

The store's dashboard function read each batch, updated running totals per product and per van in a small DynamoDB table, and pushed changes to the screen. A malformed GPS record from a van with a broken tablet would have frozen that shard for a day; with bisect, three retries and an SQS on-failure queue, it cost one skipped record and a message for Ethan to look at.

Kinesis to S3 with Firehose, step by step

1. Open Amazon Data Firehose and choose Create Firehose stream (the console's newer name for a delivery stream).
2. Source: choose Amazon Kinesis Data Streams and pick store-events, or Direct PUT if producers will send straight to Firehose.
3. Destination: choose Amazon S3 and your bucket.
4. Transform (optional): turn on a Lambda function to clean, enrich or filter records, and record format conversion to write Parquet or ORC using a table schema in the AWS Glue Data Catalog.
5. Partitioning: set an S3 prefix such as events/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/, or turn on dynamic partitioning to split by a field in the data (for example by store or event type).
6. Buffer hints: Firehose writes a file when the buffer reaches its size (1 to 128 MB) or its interval (up to 900 seconds), whichever comes first. Larger buffers mean fewer, bigger files, which are cheaper and faster to query.
7. Compression and errors: choose GZIP or Snappy compression, and an error output prefix where Firehose writes records it could not process.
8. Permissions: let the console create the IAM role, which reads the stream and writes to the bucket (and uses the KMS key if the bucket has one).
9. Create it, send some records, and watch files appear in S3 after the buffer interval.

If the destination is unavailable, a Direct PUT Firehose stream keeps records for up to 24 hours while it retries; with a data stream as the source, the records simply wait in the stream for as long as its retention allows. For Redshift and OpenSearch you set a retry duration of up to 7,200 seconds, and records that still fail are backed up to S3. The DeliveryToS3.Success and DeliveryToS3.DataFreshness metrics tell you whether delivery is healthy and how old the oldest undelivered record is.

Firehose also delivers to Amazon Redshift (through S3 and a COPY command), to OpenSearch for log search, to Snowflake, Splunk, Datadog, New Relic and other HTTP endpoints, and to Apache Iceberg tables.

New in 2026: streaming tables and S3 delivery straight from Data Streams

Since August 28, 2026, a data stream can deliver to S3 itself, with no Firehose stream in between. You create a channel on the stream (the CreateChannel API, or the console) with one of two destinations:

• Streaming tables: records are converted to Parquet and written into Apache Iceberg tables in Amazon S3 Tables, with automatic compaction so you do not end up with millions of tiny files. Data is queryable from Athena, EMR, Redshift, Managed Flink or any Iceberg engine within minutes. It needs an AWS Glue Schema Registry schema for the records and an S3 dead-letter queue.
• General purpose S3 buckets: records land in their original format, batched into objects with optional compression and a key layout you define, ideal for raw archives and event replay.

The rules: the stream must be in On-demand Standard or On-demand Advantage mode (provisioned streams cannot use it), the destination must be in the same Region, data freshness is configurable from 5 to 15 minutes, delivery is exactly once per shard, and it does not consume your stream's read capacity or enhanced fan-out slots. Destination encryption needs a customer managed KMS key; the AWS managed aws/kinesis key is not supported for it. Prices: $0.0275 per GB to S3 and $0.035 per GB to streaming tables on On-demand Standard, or $0.011 and $0.014 on On-demand Advantage.

Firehose or the new delivery? Firehose still does things channels do not: JSON to Parquet conversion into plain S3, Lambda transformations, dynamic partitioning, and many destinations beyond S3, and it works with provisioned streams. Channels are simpler and cheaper per GB than Firehose with its 5 KB rounding when your stream is on-demand and you only need raw S3 or Iceberg tables.

ProvisionedThroughputExceededException: what it means and how to fix it

This is the most common Kinesis error, and its name confuses people because it also appears in on-demand mode. It means a request asked a shard for more than it can give at that moment:

• On writes: a shard received more than 1 MB per second or more than 1,000 records per second.
• On reads: consumers called GetRecords more than 5 times per second on a shard, or read more than 2 MB per second from it. A single GetRecords call can return up to 10 MB, and if it does, calls in the next 5 seconds are throttled.
• In on-demand mode: your traffic more than doubled its previous peak within about 15 minutes, before Kinesis finished adding capacity, or one partition key alone sent more than one shard's worth of traffic.

Ethan: "It's not that the stream is full. One lane of it is."

Jake: "Like the one checkout that's open on a Saturday while the other four are closed."

Finding the cause takes a few minutes:

1. Check which side is throttled: in CloudWatch, look at the stream's WriteProvisionedThroughputExceeded and ReadProvisionedThroughputExceeded metrics.
2. Compare the stream's totals with its capacity: IncomingBytes and IncomingRecords per second against shards x 1 MB and shards x 1,000. If the totals are near the limit, you simply need more capacity.
3. Look for a hot shard: if the totals are well under the limit, turn on enhanced shard-level metrics and see whether one shard gets far more traffic than the others. That points to an uneven partition key.
4. Count your readers: list every Lambda event source mapping, KCL application, Firehose stream and tool reading the stream. Five standard consumers each polling once a second already use the entire 5-reads-a-second budget of every shard.
5. Check record sizes and counts: thousands of tiny records hit the 1,000-records limit long before the 1 MB limit.

# Per-shard metrics to find a hot shard (extra CloudWatch metric charges apply)
aws kinesis enable-enhanced-monitoring --stream-name store-events \
  --shard-level-metrics IncomingBytes IncomingRecords WriteProvisionedThroughputExceeded

# Provisioned: double the shards evenly across the hash range
aws kinesis update-shard-count --stream-name store-events --target-shard-count 4 \
  --scaling-type UNIFORM_SCALING

# Or move the stream to on-demand and let Kinesis scale it
aws kinesis update-stream-mode \
  --stream-arn arn:aws:kinesis:us-east-1:111122223333:stream/store-events \
  --stream-mode-details StreamMode=ON_DEMAND

Fixes for write throttling:

• Retry with backoff and jitter, resending only the failed records from PutRecords, as in the producer above. Short bursts of throttling are normal, and retries absorb them.
• Spread your partition keys. If one key (for example a single store ID or "unknown") carries a big share of the traffic, every record for it lands on one shard. Use a key with many distinct values, such as an order or session ID, or add a random suffix to busy keys when you do not need strict ordering for them.
• Add capacity: raise the shard count, split just the hot shard with SplitShard, or switch the stream to on-demand. On On-demand Advantage, set warm throughput before a known spike.
• Send fewer, bigger records: aggregate small events with the KPL or your own batching, which also lowers on-demand and Firehose costs.

Fixes for read throttling:

• Use enhanced fan-out for busy consumers, so each gets its own 2 MB per second per shard and stops competing for the 5 shared reads.
• Slow down polling: custom consumers should wait at least 200 milliseconds between GetRecords calls per shard; the KCL and Lambda already pace themselves.
• Consolidate consumers: one Lambda function that fans results out (to SNS or EventBridge) can replace three that each poll the stream.
• Add shards: read capacity scales with the shard count too.

The store hit write throttling on its first big sale weekend: every click event used the store's location ID as the partition key, and the flagship store generated almost all the traffic. Switching the key to the shopper's session ID spread the load over both shards and the errors stopped without adding a single shard.

Other common Kinesis errors and fixes

• ExpiredIteratorException: a shard iterator is valid for only 5 minutes. Your consumer paused too long between calls; request a new iterator with AFTER_SEQUENCE_NUMBER and the last sequence number you processed.
• ResourceNotFoundException: the stream name, Region or account is wrong (a common mix-up between the CLI's default Region and the stream's Region), or the stream was deleted.
• ResourceInUseException: the stream is still Creating, Updating (during resharding or a mode switch) or Deleting; wait until it is Active. For fan-out consumers, it also appears if you call SubscribeToShard again within 5 seconds.
• LimitExceededException: too many streams being created at once (the limit is 5 in the Creating state), the account's shard quota is reached, or a control-plane API is called too often. DescribeStream allows only 10 calls per second per account; use DescribeStreamSummary and ListShards instead.
• KMSThrottlingException or KMSAccessDeniedException: the stream is encrypted and the producer lacks kms:GenerateDataKey, the consumer lacks kms:Decrypt, or your account hit its KMS request rate.
• InvalidArgumentException: a record over 10 MiB, more than 500 records in one PutRecords call, or a partition key longer than 256 characters.
• Lambda falls further and further behind: a failing record is being retried (set bisect, retries and an on-failure destination), the function is too slow (raise batch size, parallelization factor or memory), or the function is being throttled by its concurrency limit.
• Firehose delivers nothing to S3: the Firehose role lacks s3:PutObject or the KMS key, a Lambda transform is failing (check the error output prefix in your bucket), or you are still inside the buffer interval.
• Duplicate records at the consumer: expected. Kinesis delivers at least once: a producer retry after a timeout or a consumer restart before its checkpoint can repeat records, so make your processing idempotent, for example with a unique event ID.

Kinesis vs Kafka (MSK), SQS, SNS, EventBridge and Glue

Service	What it is best at	Key difference from Kinesis Data Streams
Amazon MSK (Apache Kafka)	Kafka-compatible streaming with the Kafka ecosystem	Topics and partitions instead of shards; Kafka APIs, Kafka Connect and long retention with tiered storage; more to configure
Amazon SQS	Work queues: each message processed once, by one worker	Messages are deleted after processing; no replay, no multiple independent readers
Amazon SNS	Pushing a message to many subscribers at once	No storage; subscribers that are down miss messages unless an SQS queue buffers them
Amazon EventBridge	Routing business and AWS events to targets by rules	Content-based filtering and SaaS sources, but lower throughput and no ordered, per-key stream
AWS Glue	ETL jobs and the Data Catalog	Glue transforms data; Kinesis moves it. Glue streaming jobs can read from a Kinesis stream

Kinesis vs Kafka: choose Kinesis when you want a serverless, AWS-native stream with little to manage and pay-per-use pricing; choose Kafka (on MSK, MSK Serverless, or self-managed) when your team already knows Kafka, you need Kafka-compatible tools and connectors, very long retention, or the same platform across clouds and on-premises. Both deliver ordered, replayable streams; the difference is mostly operations and ecosystem.

Kinesis vs SQS: if a record should be handled by exactly one worker and then forgotten (resize this image, send this email), use SQS. If several systems each need every record, in order, and you may want to replay them, use Kinesis. To push one event to many subscribers without storing it, use SNS; to route events by their content to different targets, use EventBridge.

Amazon Kinesis Video Streams in brief

Kinesis Video Streams (KVS) is the odd one out: it ingests and stores video (and audio, radar or thermal imagery) from cameras, phones and devices, using a producer SDK for C, C++, Java or Android, or GStreamer. You can play it back live or from storage with HLS or MPEG-DASH in a browser, grab clips and still images, send frames to Amazon Rekognition or your own machine learning models, and run WebRTC for real-time two-way video, such as video doorbells and remote inspections.

• Ingestion: $0.0085 per GB; storage: $0.023 per GB-month (or $0.0125 in the warm tier); reading: $0.0085 per GB, or $0.0119 per GB for HLS, MPEG-DASH and clips.
• Images: $10 per million images at 1080p or lower.
• WebRTC: $0.03 per active signaling channel a month, $2.25 per million signaling messages, and $0.12 per 1,000 TURN streaming minutes.

One 1080p camera streaming around the clock at 2 Mbps sends roughly 633 GB a month: about $5.38 to ingest, and about $3.40 to keep a rolling seven days, so roughly $9 a month per camera before anyone watches.

Limits and quotas worth knowing

• Record size: up to 10 MiB of data per record, designed for occasional large records; most records are a few KB.
• Per shard: 1 MB/s or 1,000 records/s of writes; 2 MB/s and 5 GetRecords calls per second of shared reads; up to 10,000 records or 10 MB per GetRecords call.
• PutRecords: up to 500 records per call.
• Retention: from 24 hours (the minimum) to 8,760 hours (365 days).
• On-demand streams: 50 per account by default (adjustable); new streams start at 4 MB/s write and 8 MB/s read.
• Shards: 20,000 per account by default in N. Virginia, Oregon and Ireland, and 1,000 or 6,000 in other Regions, adjustable through Service Quotas.
• Fan-out consumers: 20 per stream (50 with On-demand Advantage).
• Mode switches: twice per stream in 24 hours.
• Shard iterators: expire after 5 minutes.

Firehose limits:

• Record size: up to 1,000 KiB per record; PutRecordBatch takes up to 500 records or 4 MiB per call, whichever is smaller.
• Direct PUT throughput per Firehose stream: 500,000 records, 2,000 requests and 5 MiB per second in N. Virginia, Oregon and Ireland; 100,000 records, 1,000 requests and 1 MiB per second in other Regions. Firehose raises a throttled stream's limit automatically over time, and you can request more in advance. With a data stream as the source, these limits do not apply.
• Firehose streams: 5,000 per Region in N. Virginia, Ohio, Oregon, Ireland and Tokyo, fewer in smaller Regions.
• Dynamic partitioning: 500 active partitions per Firehose stream by default.

Security: KMS encryption, IAM and VPC endpoints

• Encryption at rest: server-side encryption encrypts records with a KMS key as they are written and decrypts them as they are read. Choose the AWS managed aws/kinesis key (no key setup) or your own customer managed key (needed for cross-account access and key policies you control). Producers need kms:GenerateDataKey on the key and consumers need kms:Decrypt; KMS request charges apply.
• Encryption in transit: all Kinesis APIs use HTTPS (TLS).
• IAM: grant producers only the put actions on specific stream ARNs and consumers only the read actions; a resource policy on the stream can grant another account's role access without copying data.
• VPC interface endpoints: producers and consumers in private subnets reach Kinesis through AWS PrivateLink endpoints (com.amazonaws.us-east-1.kinesis-streams for Data Streams and com.amazonaws.us-east-1.kinesis-firehose for Firehose), with no NAT gateway or internet route, and an endpoint policy can limit which streams are reachable. Interface endpoints cost about $0.01 per hour per Availability Zone plus $0.01 per GB processed.

The Kinesis Data Streams API in one list

• Writing: PutRecord (one record) and PutRecords (up to 500).
• Reading: ListShards, GetShardIterator and GetRecords for standard consumers; RegisterStreamConsumer and SubscribeToShard for enhanced fan-out.
• Managing streams: CreateStream, DescribeStreamSummary, UpdateStreamMode, UpdateShardCount, SplitShard, MergeShards, IncreaseStreamRetentionPeriod, StartStreamEncryption, DeleteStream.
• Account and delivery: UpdateAccountSettings (turns On-demand Advantage on or off), UpdateStreamWarmThroughput, and CreateChannel for S3 and streaming tables delivery.

Two practical notes. For checking a stream from scripts, call DescribeStreamSummary and ListShards rather than the older DescribeStream, which allows only 10 calls per second for the whole account and pages through shards slowly. And when you write a consumer, you rarely need the read APIs directly: Lambda and the KCL call ListShards, GetShardIterator and GetRecords (or SubscribeToShard) for you and handle the edge cases, so your code only sees batches of records.

With Terraform, the same stream and an S3 Firehose stream look like this:

resource "aws_kinesis_stream" "store_events" {
  name             = "store-events"
  retention_period = 24
  encryption_type  = "KMS"
  kms_key_id       = "alias/aws/kinesis"
  stream_mode_details {
    stream_mode = "ON_DEMAND"
  }
}

resource "aws_kinesis_firehose_delivery_stream" "store_archive" {
  name        = "store-archive"
  destination = "extended_s3"
  kinesis_source_configuration {
    kinesis_stream_arn = aws_kinesis_stream.store_events.arn
    role_arn           = aws_iam_role.firehose.arn
  }
  extended_s3_configuration {
    role_arn           = aws_iam_role.firehose.arn
    bucket_arn         = aws_s3_bucket.archive.arn
    prefix             = "events/"
    buffering_size     = 64
    buffering_interval = 300
    compression_format = "GZIP"
  }
}

Best practices and ways to cut the bill

• Start on-demand, then measure. Once traffic is steady, move to provisioned with 20 to 30 percent headroom; steady streams are often several times cheaper that way.
• Choose partition keys with many values that still keep together what must stay in order (an order ID, a device ID).
• Batch small events into bigger records to beat the 1 KB on-demand rounding, the 1,000-records-a-second limit and Firehose's 5 KB rounding.
• Retry only failed records from PutRecords, with backoff and jitter.
• Make consumers idempotent, since Kinesis delivers at least once.
• Give each busy consumer enhanced fan-out instead of piling standard consumers onto one stream.
• Keep retention as short as you need; archive to S3 for anything long-term instead of paying for a year of stream retention.
• Alarm on WriteProvisionedThroughputExceeded, ReadProvisionedThroughputExceeded, GetRecords.IteratorAgeMilliseconds and Lambda's IteratorAge.
• Encrypt streams with KMS and use a VPC interface endpoint so producers in private subnets reach Kinesis without the internet; producers in other Regions pay normal data transfer.
• Delete test streams and Firehose streams when you finish, and tag everything with an owner.

Frequently asked questions

The back-office screen went live on a Friday. By Saturday lunchtime the owner had moved two staff from the warehouse to the showroom because the screen showed forty people browsing recliners, and a van that had been sitting in one spot for half an hour turned out to have a flat tire. The stream started in on-demand mode, moved to two provisioned shards after three weeks, and the whole pipeline, stream, Firehose archive and Lambda dashboard, settled at about $62 a month. Jake asked whether the screen could also show which customers were about to buy. "Give it a few months of data in S3," Ethan said, "and we'll find out."

Revision note. Written October 21, 2026. Prices are US East (N. Virginia) list prices on that date. We will update this page when Kinesis prices or features change.