What Is Amazon MSK? Managed Kafka Pricing, Serverless vs Express Brokers, Setup With IAM, MSK vs Kinesis and Error Fixes

Amazon MSK (Managed Streaming for Apache Kafka) runs real, open-source Apache Kafka for you inside your VPC: AWS creates the brokers across Availability Zones, manages the metadata layer, patches, replaces failed brokers and handles upgrades, while your producers and consumers use the normal Kafka clients, tools and connectors. There are three ways to run it. MSK Provisioned with Standard brokers charges per broker-hour (from $0.0456 for a kafka.t3.small or $0.204 for a kafka.m7g.large) plus $0.10 per GB-month of storage; Express brokers (from $0.408 an hour) add up to three times the throughput per broker, faster scaling and no storage to manage, plus $0.01 per GB written; and MSK Serverless costs $0.75 per cluster-hour (about $540 a month) plus $0.0015 per partition-hour and $0.10 per GB in and $0.05 per GB out. A small three-broker production cluster costs roughly $450 to $750 a month. MSK supports Kafka up to 4.1, with KRaft replacing ZooKeeper. Below: what MSK is and how it works, Standard vs Express vs Serverless, every price with worked examples, a step-by-step setup with IAM authentication, MSK Connect, tiered storage and S3 delivery, MSK vs Kinesis, Confluent and self-managed Kafka, limits, and fixes for the Kafka errors everyone hits.

The furniture store's new warehouse management system arrived with one technical requirement: it published every stock movement, pick, pack and delivery scan to Apache Kafka topics, and the store had to provide the Kafka. The vendor's other customers ran Kafka on their own servers, with a part-time engineer who knew how. The store had Ethan, who did not want to babysit ZooKeeper at weekends. Jake asked whether the store's Kinesis stream would do. Ethan said the vendor's software only speaks Kafka, and opened Amazon MSK.

New to streaming on AWS? The cards below cover Kinesis, subnets, Secrets Manager, Lambda and billing alerts, the pieces an MSK cluster usually sits next to.

What is Amazon MSK?

Apache Kafka is the most widely used open-source platform for event streaming: applications write events (an order placed, a parcel scanned, a sensor reading) to named topics, Kafka stores them durably in order, and any number of other applications read them, each at its own pace, for as long as the data is retained. Thousands of tools speak Kafka: connectors for databases and data lakes, stream processors such as Kafka Streams and Flink, and client libraries for every language. Running Kafka yourself is famously hard: brokers, the metadata quorum, disk sizing, partition rebalancing, rolling upgrades, security and monitoring.

Amazon MSK full form: MSK stands for Managed Streaming for Apache Kafka. AWS previewed it at re:Invent 2018 and made it generally available in May 2019; MSK Serverless, MSK Connect, tiered storage and Express brokers came later.

Amazon MSK (Amazon Managed Streaming for Apache Kafka) is AWS's managed Kafka. Unlike services that only imitate an API, MSK runs the actual Apache Kafka software, so client code, command-line tools, Kafka Connect connectors and monitoring tools work unchanged. AWS handles:

• Provisioning brokers across two or three Availability Zones in your VPC.
• The metadata layer: ZooKeeper on older versions, or KRaft, Kafka's built-in metadata quorum, on 3.7 and later (and only KRaft from 4.0).
• Failure handling: detecting and replacing failed brokers and their storage.
• Patching and upgrades: rolling, one broker at a time, so the cluster stays available.
• Security: encryption at rest and in transit, IAM, SASL/SCRAM and mutual TLS authentication.
• Monitoring: CloudWatch metrics and optional Prometheus endpoints.

What MSK does not do for you: it does not design your topics, choose your partition counts or keys, size your cluster for your traffic, write your producers and consumers, or watch your consumer lag. It does not rebalance partitions automatically when you add Standard brokers, and it does not decide your retention or your security policies. Those remain your decisions, and they are where most Kafka problems actually come from. The good news is that they are application decisions you would face on any Kafka, while the undifferentiated work (servers, disks, patching, failures, metadata quorums) is no longer yours.

What it is used for: event-driven microservices, change data capture from databases, log and metrics pipelines, clickstream, IoT and telemetry, payment and order events, feeding data lakes and real-time analytics, and integrating vendor software that requires Kafka, as the store's warehouse system did.

Jake: "So it's Kafka, the real one, but AWS gets the 3 a.m. phone call."

Ethan: "AWS replaces the broken broker. We still get called if our own consumer falls behind. That part's ours."

How MSK works: brokers, topics, partitions and replication

1. A cluster is a set of brokers (Kafka servers) spread evenly across two or three Availability Zones, in subnets you choose.
2. Topics are split into partitions, the unit of parallelism and ordering: records with the same key go to the same partition and stay in order.
3. Each partition is replicated (usually three copies, one per AZ). One replica is the leader that takes writes; followers copy it. With acks=all and min.insync.replicas=2, a write is confirmed only when two copies have it.
4. Producers connect to the cluster's bootstrap brokers, discover the leaders, and send records.
5. Consumers in a consumer group divide the partitions between them and commit offsets, their position in each partition, so they can stop and resume.
6. Retention decides how long records stay: hours, days, or indefinitely; with tiered storage, older data moves to a cheaper tier automatically.
7. If a broker fails, leadership moves to replicas in other AZs, MSK replaces the broker, and it catches up.

The number of partitions decides how far a topic can scale: a consumer group can have at most one active consumer per partition, so a topic with 12 partitions supports 12 parallel consumers. Choose partitions for the throughput you expect in a year, not today, because adding partitions later changes which partition each key lands on.

Kafka words you will meet in MSK

• Broker: a Kafka server that stores partitions and serves your clients. An MSK cluster has several, spread across AZs.
• Topic: a named stream of records, such as stock-movements or orders.
• Partition: an ordered, append-only slice of a topic; your unit of parallelism and ordering.
• Offset: a record's position in its partition; your consumers commit offsets to remember where they stopped.
• Consumer group: a set of consumers that share a topic's partitions, each partition read by one member at a time; different groups read the same data independently.
• Replication factor and ISR: how many copies each partition has, and the in-sync replicas that are fully caught up; min.insync.replicas sets how many must confirm a write.
• Retention and compaction: retention deletes records after a time or size; compaction keeps only the latest record per key, useful for tables of current state.
• KRaft and ZooKeeper: the metadata layer that tracks brokers, topics and leaders; KRaft is built into Kafka and replaces ZooKeeper from Kafka 4.0.

Your workload	A sensible MSK starting point	Rough monthly cost
Development and testing	3 x kafka.t3.small Standard, 100 GB each	$130
Small production, steady traffic	3 x kafka.m7g.large Standard, storage auto scaling	$600 to $800
Unknown or spiky traffic under 200 MBps	MSK Serverless	$600 to $1,000+
Busy production, minimal operations	3 to 6 x express.m7g.xlarge or larger	$1,800 and up
Long retention for replay or audit	Standard brokers with tiered storage, or data delivery to S3	Brokers + $0.06 per GB-month tiered

These are starting points, not answers: your message sizes, number of consumer groups and retention will move you up or down a size, which is why a short load test with your real traffic is worth more than any table.

MSK Provisioned (Standard and Express brokers) vs MSK Serverless

	Standard brokers	Express brokers	MSK Serverless
You choose	Broker size and count, EBS storage per broker	Broker size and count (m7g only)	Nothing: capacity scales automatically
Storage	EBS per broker, up to 16 TiB each, plus optional tiered storage	Unlimited, managed, no sizing	Managed, unlimited retention
Throughput	Depends on broker size and disks	Up to 3x per broker vs Standard; 15.6 to 500 MBps in per broker sustained	Up to 200 MBps in, 400 MBps out per cluster
Configuration control	Full Kafka configuration	Most settings, with managed defaults	Limited
Authentication	IAM, SASL/SCRAM, mTLS or none	IAM, SASL/SCRAM, mTLS	IAM only
Price model	Broker-hours + GB-month of provisioned storage	Broker-hours + GB written + GB-month stored	Cluster-hours + partition-hours + GB in/out + storage
Best for	Cost-sensitive steady clusters, full tuning	High-throughput production with little operations	New apps, dev and test, variable traffic

• Standard brokers are classic Kafka brokers on Graviton (m7g), Intel (m5) or burstable (t3.small, for development only) instances with EBS volumes you size, with storage auto scaling available. You get every Kafka setting and the lowest price per broker-hour, and you plan disk, partitions and rebalancing.
• Express brokers, launched in late 2024, decouple storage from the brokers: there is no disk to size, scaling and recovery are much faster, and each broker handles far more throughput, so you need fewer of them. They cost twice a Standard broker of the same size per hour, plus $0.01 per GB written, and replication traffic between brokers is free.
• MSK Serverless removes capacity planning: create a cluster, create topics, connect with IAM. It suits workloads up to its quotas (200 MBps in, 2,400 partitions) and costs a fixed $540 a month per cluster before data, so it is not the cheapest option for tiny workloads.

Should you use MSK? A quick decision guide

Before you compare prices, answer three questions about your workload:

1. Do you need Kafka itself? If your software, vendor or team requires the Kafka protocol, Kafka Connect connectors or Kafka Streams, you need Kafka, and MSK is the managed way to run it on AWS. If you only need a stream of events between your own apps, Kinesis Data Streams or SQS may serve you with less to manage.
2. How steady is your traffic? If your throughput is unknown or spiky and stays within 200 MBps, start with MSK Serverless. If it is steady, provisioned brokers will cost you less per GB.
3. How much does your team want to operate? If you want to tune every setting and keep the broker price low, choose Standard brokers. If you want Kafka with as little operations work as possible at higher throughput, choose Express brokers.

Most teams that answer "yes" to the first question end up on provisioned Standard or Express brokers for production and Serverless for development or small side projects. Whatever you pick, you can move later: your clients only need a new bootstrap string.

How many brokers do you need?

Sizing a Standard cluster comes down to four numbers you can estimate from your own traffic: peak write throughput in MB/s, the number of consumer groups reading everything, retention in days, and partition count. As a starting rule of thumb, you plan for each kafka.m7g.large broker to handle a sustained few tens of MB/s of writes with replication factor 3, keep CPU under about 60 percent at your peak so the cluster survives losing a broker, and size storage as daily writes x retention days x 3 replicas, plus 20 percent headroom. Your read traffic counts too: three consumer groups reading every record triple your outbound network.

• Start with three brokers (one per AZ) and the smallest size that keeps your CPU and network comfortable in a load test with your real message sizes.
• Scale up before you scale out for CPU-bound clusters; scale out (more brokers, then reassign partitions) when you need more partitions or disk.
• Use the MSK sizing spreadsheet or a load test rather than guessing; the kafka-producer-perf-test and kafka-consumer-perf-test tools that ship with Kafka give you realistic numbers in an afternoon.
• On Express brokers, size from the published sustained throughput per broker size (15.6 MBps in for express.m7g.large up to 500 MBps for 16xlarge) and you can skip storage planning entirely.

Amazon MSK pricing

US East (N. Virginia) prices in October 2026:

Item	Price	Per month (720 h)
Standard broker kafka.t3.small (dev only)	$0.0456 per hour	$32.83
Standard broker kafka.m7g.large	$0.204 per hour	$146.88
Standard broker kafka.m7g.xlarge	$0.408 per hour	$293.76
Standard broker kafka.m5.large	$0.21 per hour	$151.20
Standard broker storage (EBS)	$0.10 per GB-month
Provisioned storage throughput	$0.08 per MB/s-month
Tiered storage	$0.06 per GB-month + $0.0015 per GB retrieved
Express broker express.m7g.large	$0.408 per hour	$293.76
Express broker express.m7g.xlarge	$0.816 per hour	$587.52
Express data written / storage	$0.01 per GB / $0.10 per GB-month
Serverless cluster	$0.75 per cluster-hour	$540.00
Serverless partitions	$0.0015 per partition-hour	$1.08 per partition
Serverless data	$0.10 per GB in, $0.05 per GB out, $0.10 per GB-month stored
MSK Connect	$0.11 per MCU-hour (1 vCPU, 4 GB)	$79.20 per MCU
MSK Replicator	$0.30 per hour + $0.08 per GB (+ $0.02 per GB cross-Region)	$216 + data
Private connectivity (multi-VPC)	$0.0225 per hour per auth scheme + $0.006 per GB	$16.20 + data
Data delivery to S3 / S3 Tables	$8 per TB / $10 per TB delivered

• Standard storage is the EBS volume on each broker, so with three replicas, 1 TB of data needs about 3 TB of provisioned storage across the cluster. Express storage is billed on the data you keep, with replication handled for you.
• Data transfer: traffic between brokers for replication is free; traffic between your clients and brokers in different Availability Zones is billed at normal EC2 rates, which can be a significant hidden cost for busy clusters. Keep consumers in the same AZs as the brokers they read from, or use rack awareness for follower fetching.
• No free tier: MSK has no free allowance; accounts created from July 15, 2025 can use Free Tier credits.

Worked examples: what MSK really costs

Example 1: a development cluster. Three kafka.t3.small Standard brokers with 100 GB each: 3 x $32.83 + 300 GB x $0.10 = $128.50 a month. Burstable brokers are fine for development and tests, not for production traffic.

Example 2: the store's warehouse events on Standard brokers. Three kafka.m7g.large brokers (one per AZ), 1 TB of retained data (7 days of events) replicated three times, so 3 TB of EBS: 3 x $146.88 + 3,000 x $0.10 = $740.64 a month.

Example 3: the same on Express brokers. Three express.m7g.large brokers, 1 TB written a month and 1 TB stored: 3 x $293.76 + 1,000 x $0.01 + 1,000 x $0.10 = $991.28 a month, about a third more, with no disks to size, faster recovery and headroom for three times the traffic.

Example 4: MSK Serverless. One cluster with 60 partitions, 1 TB written and 2 TB read a month, 100 GB stored: $540 + 60 x $1.08 + $100 + $100 + $10 = $814.80 a month, and nothing to plan.

Ethan: "Seven hundred forty on Standard, nine hundred ninety on Express, eight hundred fifteen serverless."

Jake: "And Kinesis?"

Ethan: "Cheaper, about fifty dollars on two provisioned shards for our traffic. But the warehouse software only speaks Kafka, so the real question is which Kafka."

Example 5: a connector. An S3 sink connector on MSK Connect with 2 MCUs running all month: 2 x 720 x $0.11 = $158.40 a month; for plain delivery to S3, MSK's own data delivery at $8 per TB costs $8 for the same 1 TB.

Example 6: long retention with tiered storage. Keeping a year of events (about 12 TB) on Standard brokers' EBS with three replicas would need 36 TB at $0.10: $3,600 a month. With tiered storage, the last week stays on EBS and the rest moves to the low-cost tier at $0.06 per GB-month for a single copy: about 11 TB x $0.06 = $660, plus the smaller EBS volumes. Roughly $1,000 a month instead of $3,600.

The store chose Standard m7g.large brokers: steady, modest traffic suits them, and Ethan was happy to size disks once and turn on storage auto scaling. A billing alert went on the same day.

Create an MSK cluster, step by step

1. Prepare the network: a VPC with private subnets in three Availability Zones, and a security group for the brokers that allows the client ports (9098 for IAM, 9096 for SASL/SCRAM, 9094 for TLS) only from your applications' security group.
2. Open Amazon MSK in the console and choose Create cluster, then Custom create to see every option.
3. Cluster type: Provisioned (then choose Standard or Express brokers) or Serverless.
4. Kafka version: the latest supported version (4.1 at the time of writing, with KRaft metadata) unless your clients need an older one.
5. Brokers: size (kafka.m7g.large is a sensible production start), number of zones (three), and brokers per zone (one, for three in total).
6. Storage (Standard only): EBS size per broker, with storage auto scaling turned on and a maximum, and tiered storage if you need long retention.
7. Security: choose authentication (IAM access control is the simplest on AWS), keep encryption in transit (TLS) and at rest (with your KMS key if required).
8. Configuration: start with the default, or a custom configuration with settings such as auto.create.topics.enable=false, default.replication.factor=3 and min.insync.replicas=2.
9. Monitoring: choose the CloudWatch metrics level (per-topic and per-partition metrics help debugging) and optionally broker logs to CloudWatch or S3.
10. Create the cluster. It takes 15 to 30 minutes. Then choose View client information for the bootstrap broker string.

# Bootstrap brokers for IAM authentication (port 9098)
aws kafka get-bootstrap-brokers --cluster-arn arn:aws:kafka:us-east-1:111122223333:cluster/warehouse-events/abcd1234-5678-90ef-ghij-klmnopqrstuv-1

# Create a topic from a client machine in the VPC (Kafka CLI tools + the MSK IAM library on the classpath)
bin/kafka-topics.sh --bootstrap-server b-1.warehouseevents.abc123.c2.kafka.us-east-1.amazonaws.com:9098 \
  --command-config client.properties --create --topic stock-movements --partitions 12 --replication-factor 3

MSK ports: 9092 for plaintext (only if you allow unauthenticated, unencrypted traffic), 9094 for TLS, 9096 for SASL/SCRAM, 9098 for IAM, and 9194, 9196 and 9198 for the same over public access. ZooKeeper-based clusters also expose 2181 (and 2182 for TLS) for ZooKeeper, which modern tools no longer need.

Your first producer and consumer from the command line

To check your cluster end to end before you write any code, you can use the Kafka command-line tools from an EC2 instance (or a CloudShell environment in the VPC) that has Java installed:

1. Download Apache Kafka matching your cluster's version and unpack it.
2. Add the IAM library: copy the aws-msk-iam-auth JAR into Kafka's libs folder.
3. Create client.properties with the four IAM lines shown in the next section.
4. Give your instance role an IAM policy with kafka-cluster:Connect, topic and group permissions for your cluster.
5. Create a topic, then produce and consume with the commands below, in two terminal windows.

# Terminal 1: type messages, one per line
bin/kafka-console-producer.sh --bootstrap-server b-1.warehouseevents.abc123.c2.kafka.us-east-1.amazonaws.com:9098 \
  --producer.config client.properties --topic stock-movements

# Terminal 2: read them from the beginning
bin/kafka-console-consumer.sh --bootstrap-server b-1.warehouseevents.abc123.c2.kafka.us-east-1.amazonaws.com:9098 \
  --consumer.config client.properties --topic stock-movements --from-beginning --group test-readers

# See your consumer group's lag per partition
bin/kafka-consumer-groups.sh --bootstrap-server b-1.warehouseevents.abc123.c2.kafka.us-east-1.amazonaws.com:9098 \
  --command-config client.properties --describe --group test-readers

If your messages appear in the second window, your network, authentication and permissions are all correct, and any problem you meet later is in your application code. If they do not, the error list further down covers the usual causes in the order you are likely to hit them.

Authentication and client configuration

• IAM access control (recommended on AWS): clients sign in with their IAM role, and IAM policies grant actions such as kafka-cluster:Connect, kafka-cluster:ReadData and kafka-cluster:WriteData on specific topics and consumer groups. No passwords or certificates to manage. Required for Serverless.
• SASL/SCRAM: username and password stored in Secrets Manager (the secret name must start with AmazonMSK_ and use a customer managed KMS key), with Kafka ACLs for authorization. Good for clients outside AWS that cannot use IAM.
• Mutual TLS: client certificates issued by AWS Private CA, with Kafka ACLs.
• Unauthenticated: possible on Provisioned clusters inside a closed network, but rarely a good idea.

A Java client using IAM needs the open-source aws-msk-iam-auth library and these properties (the same file works for the Kafka CLI tools as client.properties):

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

In Python, the aws-msk-iam-sasl-signer-python package generates the token for clients such as confluent-kafka or kafka-python through the OAUTHBEARER mechanism:

from aws_msk_iam_sasl_signer import MSKAuthTokenProvider
from confluent_kafka import Producer

def oauth_cb(_config):
    token, expiry_ms = MSKAuthTokenProvider.generate_auth_token("us-east-1")
    return token, expiry_ms / 1000

producer = Producer({
    "bootstrap.servers": "b-1.warehouseevents.abc123.c2.kafka.us-east-1.amazonaws.com:9098",
    "security.protocol": "SASL_SSL",
    "sasl.mechanisms": "OAUTHBEARER",
    "oauth_cb": oauth_cb,
    "acks": "all",
})
producer.produce("stock-movements", key="SOFA-3S-GRY", value=b'{"bin":"A-12","qty":-1}')
producer.flush()

Public access can be turned on for Provisioned clusters that use authentication and TLS, giving brokers public endpoints on ports 9194 to 9198; keep it off unless a client truly cannot reach the VPC, and prefer private connectivity (multi-VPC PrivateLink) for other AWS accounts.

MSK Connect, Replicator and data delivery to S3

• MSK Connect runs Kafka Connect connectors as a managed service: upload a connector plugin (for example a JDBC source, a Debezium change-data-capture source for MySQL or PostgreSQL, or an S3 sink), configure it, and MSK Connect runs and scales the workers. Capacity is in MSK Connect Units (1 vCPU and 4 GB each) at $0.11 an hour, with auto scaling between a minimum and maximum.
• MSK Replicator copies topics between MSK clusters, in the same Region or across Regions, keeping topic configuration and consumer group offsets in sync, for disaster recovery, migration or sharing data across Regions. It costs $0.30 an hour plus $0.08 per GB, with up to 1 GB per second per Replicator.
• Data delivery sends topic data straight to Amazon S3 (raw JSON, strings or bytes, with optional GZIP or ZSTD compression) or to Iceberg streaming tables in S3 Tables, with a data freshness of 5 to 15 minutes, at $8 per TB to S3 or $10 per TB to S3 Tables. For simple archiving it replaces an S3 sink connector at a fraction of the cost.
• Other integrations: Lambda event source mappings read MSK topics in batches; Amazon Data Firehose can read from MSK; Managed Service for Apache Flink and Spark on EMR process streams; EventBridge Pipes route events to other targets; and the AWS Glue Schema Registry validates Avro, JSON Schema and Protobuf records.

Tiered storage, retention and scaling

On Standard brokers, every byte of retained data sits on EBS, three times over. Tiered storage moves log segments older than a local retention time (for example one day) to a low-cost tier backed by AWS storage, billed at $0.06 per GB-month for one copy plus $0.0015 per GB read back. Consumers read old data transparently, just slower for the first fetch. It suits long retention for replay, audits and reprocessing, and needs a supported Kafka version and broker type.

• Storage auto scaling grows broker EBS volumes when disk use crosses a threshold you set; volumes never shrink, so set a sensible maximum.
• Provisioned storage throughput raises EBS throughput above the baseline for brokers that write heavily ($0.08 per MB/s-month).
• Adding brokers to a Standard cluster does not move existing partitions; reassign partitions (with Cruise Control or the reassignment tool) so new brokers take load. Express brokers rebalance much faster, which is a large part of their appeal.
• Changing broker size is a rolling update with no downtime when clients are configured with retries and replication factor 3.

Monitoring MSK

• Broker health: CpuUser (keep under about 60 percent so a broker failure does not overload the others), KafkaDataLogsDiskUsed (alarm well before 85 percent), memory and network throughput.
• Replication: UnderReplicatedPartitions and OfflinePartitionsCount should be zero; anything else needs attention.
• Consumers: MaxOffsetLag and SumOffsetLag per consumer group show how far behind consumers are; rising lag means consumers are too slow or stuck.
• Connections: ClientConnectionCount and ConnectionCreationRate, against the IAM limits of 3,000 connections and 100 new connections per second per broker.
• Open monitoring: Prometheus JMX and node exporters on the brokers, for Grafana dashboards, plus broker logs to CloudWatch Logs, S3 or Firehose.

Your alarms matter more than your dashboards: you will rarely look at a dashboard at 3 a.m., but you will get the alert. Start with four alarms on every production cluster you run: disk use above 75 percent, CPU above 60 percent, any under-replicated partitions for more than a few minutes, and consumer lag above whatever your business can tolerate for each important consumer group.

Migrating your existing Kafka to MSK

1. Inventory your cluster: Kafka version, topics and partition counts, retention settings, ACLs, client libraries and their versions, and peak throughput.
2. Create your MSK cluster on a version your clients support, with matching topic configurations (create topics explicitly; turn auto-creation off).
3. Replicate the data: run MirrorMaker 2 on MSK Connect (or MSK Replicator if your source is another MSK cluster) to copy topics and consumer group offsets continuously from your old cluster to MSK.
4. Move your consumers first: point each consumer group at MSK, starting from the translated offsets, and check its lag and output.
5. Move your producers: switch them to MSK once their consumers are reading there, then stop replication.
6. Decommission the old cluster after a safe waiting period, when you are sure nothing still writes to it.

Plan the authentication change with your client owners: if your old cluster used no authentication or SASL/PLAIN, your clients need new security settings for IAM or SCRAM, and that is usually the slowest part of a migration.

Kafka versions, upgrades and end of support

MSK supports a range of Apache Kafka versions, from the 3.x line up to 4.0 (added in May 2025) and 4.1. Kafka 4.0 removed ZooKeeper entirely, so 4.x clusters run in KRaft mode, Kafka's built-in metadata quorum, which MSK has offered since 3.7; version 3.9 is the last that supports both. You upgrade a Provisioned cluster in place with a rolling update, one broker at a time. Older versions reach end of support on announced dates; after that, clusters are upgraded automatically or must be upgraded before the deadline, so keep clusters on a supported version and test client compatibility first. Note that a ZooKeeper-based cluster cannot be switched to KRaft in place; moving to 4.x means a new KRaft cluster and a migration, for example with MSK Replicator.

MSK vs Kinesis, self-managed Kafka, Confluent Cloud and SQS

• MSK vs Kinesis Data Streams: both are durable, ordered, replayable streams. Kinesis is AWS-native, serverless or shard-based, simpler, and often cheaper for small and medium workloads (a one-shard stream costs about $11 a month). MSK runs real Kafka: choose it when software requires Kafka, when your team knows Kafka, when you need the Kafka ecosystem (Connect, Streams, Schema Registry, exactly-once semantics), very long retention, or a portable architecture across clouds and on-premises.
• MSK vs self-managed Kafka on EC2: self-managed avoids MSK's markup on broker instances but makes you responsible for provisioning, the metadata quorum, upgrades, failure recovery, security and monitoring; MSK is usually cheaper once engineering time is counted.
• MSK vs Confluent Cloud: Confluent Cloud is a fully managed Kafka service from Kafka's original creators, available on AWS, with extras such as ksqlDB, a large managed connector catalog, stream governance and cluster linking, priced on its own model. MSK is AWS-billed, VPC-native and integrates with IAM; Confluent offers a richer platform and multi-cloud consistency.
• MSK vs SQS: SQS is a queue: each message goes to one consumer and is deleted. Kafka keeps an ordered log that many consumer groups read independently and can replay. Use SQS for task queues, Kafka for event streams.
• MSK vs EventBridge: EventBridge routes events by rules to targets with no infrastructure; Kafka handles far higher throughput with ordering and retention.

MSK limits and quotas

• Brokers: 90 per account per Region by default; 30 per cluster for ZooKeeper-based clusters, 60 for KRaft-based clusters.
• Standard broker storage: 1 GiB to 16 TiB per broker.
• IAM connections: 3,000 per broker and 100 new connections per second (4 per second on t3 brokers).
• Express brokers: sustained ingress from 15.6 MBps (large) to 500 MBps (16xlarge) per broker, up to 15 MB/s per partition, and 1,000 to 20,000 recommended partitions per broker by size.
• Serverless (per cluster): 200 MBps in, 400 MBps out, 5 MBps in and 10 MBps out per partition, 2,400 partitions (120 for compacted topics), 3,000 client connections, 500 consumer groups, 8 MiB messages, unlimited retention, and 10 serverless clusters per account.
• MSK Connect: 60 workers per account and 10 per connector by default. Replicator: 15 per account, 750 topics and 1 GB/s each.

Common MSK and Kafka errors and fixes

• "Timeout expired while fetching topic metadata" or clients hang on connect: wrong bootstrap string or port for the authentication type, the broker security group does not allow the client, or the client is outside the VPC. Use the bootstrap brokers for your auth method (9098 for IAM) and test with nc -vz broker 9098 from the client.
• SaslAuthenticationException or "Access denied": with IAM, the client's role lacks kafka-cluster:Connect on the cluster or the topic and group actions; with SCRAM, the secret is not associated with the cluster or not encrypted with a customer managed key.
• TopicAuthorizationException: the IAM policy or ACL does not cover that topic name or consumer group; check the ARN patterns, including the cluster UUID.
• Disk full, brokers unhealthy: KafkaDataLogsDiskUsed reached 100 percent; turn on storage auto scaling, lower retention, enable tiered storage, or add brokers and rebalance.
• NotEnoughReplicasException: a broker is down or replicas are out of sync, so writes with acks=all cannot meet min.insync.replicas; it usually clears after the broker recovers.
• RecordTooLargeException: a message exceeds message.max.bytes (8 MiB maximum on Serverless); split large payloads or store them in S3 and send a pointer.
• Consumer lag keeps growing: too few partitions or consumers, slow processing, or frequent rebalances; add consumers up to the partition count, tune max.poll.records, or scale processing.
• LEADER_NOT_AVAILABLE on first write: the topic is being created or a leader election is in progress; retries resolve it. Create topics explicitly rather than relying on auto-creation.
• Too many connections on IAM clusters: applications open a new client per request; reuse producers and consumers, and set reconnect.backoff.ms.

Best practices

• Three AZs, replication factor 3, min.insync.replicas=2, acks=all for data you cannot lose.
• Use IAM authentication and least-privilege policies per topic and group.
• Size partitions for the future, but stay within the per-broker partition recommendations.
• Keep CPU under about 60 percent and disk under about 80 percent, with alarms.
• Turn off topic auto-creation and manage topics as code.
• Keep clients in the brokers' AZs or use follower fetching, to cut cross-AZ data transfer costs.
• Stay on a supported Kafka version, and plan the move to KRaft-based 4.x clusters.
• Use tiered storage or data delivery to S3 for long retention instead of ever-larger broker disks.

A security checklist for your cluster: your brokers sit in private subnets; your security group allows only your application security groups on only the ports you use; your clients authenticate with IAM (or SCRAM or mTLS) and never with plaintext; your encryption at rest uses a KMS key you control if your compliance rules require it; your IAM policies name specific topics and consumer groups rather than *; and your broker logs go to CloudWatch Logs or S3 so you can investigate when something looks wrong. If you can tick every item, your cluster is in better shape than most self-managed Kafka in the wild.

Finally, keep your Kafka knowledge in one place: a short runbook with your bootstrap strings, how to add a broker, how to reassign partitions, which alarms page whom, and how to roll back a client change. Your future self, or whoever covers your holiday, will thank you for it.

Frequently asked questions

The warehouse system went live on three kafka.m7g.large brokers in three Availability Zones, with IAM authentication, storage auto scaling, and MSK's data delivery archiving every stock event to S3. The vendor's engineers connected on the first try once they had the right port. In the first month, one consumer fell behind during a stock-take, the lag alarm fired, and adding two consumers fixed it. The bill was $749, including the S3 delivery. Jake asked why the store now had both Kinesis and Kafka. "Kinesis is ours," Ethan said. "Kafka is what the warehouse speaks. Now they can talk to each other."

Revision note. Written October 21, 2026. Prices are US East (N. Virginia) list prices on that date. We will update this page when MSK prices or features change.