Kafka storage calculator: partitions, brokers and monthly cost on self-managed, MSK and Confluent Cloud

How it is calculated

W      = m × s × ρ            bytes per second on the wire and on disk
W_peak = p × W
In     = R × W_peak           into the brokers (producers plus replication)
Out    = (R − 1) × W_peak + G × W_peak   out of the brokers (replication plus consumers)

per copy    = W × 86,400 × retention days
all replicas = per copy × R
provisioned = all replicas ÷ u        u = disk utilisation target

minimum    = max( ceil(W_peak ÷ producer ceiling), ceil(W_peak ÷ consumer ceiling), minimum partitions )
partitions = minimum rounded up to a multiple of the broker count

by storage = ceil(provisioned ÷ disk per broker)
by network = ceil(max(In, Out) ÷ (NIC bandwidth × 0.5))
brokers    = max(by storage, by network, R, AZ count) rounded up to a multiple of the AZ count

replication = followers in other AZs × W × seconds per month     (R − 1 followers when R ≤ AZ count)
clients     = (1 + G) × W × seconds per month × (AZ − 1) ÷ AZ

each component = (quantity − free allowance) × rate
broker hours = brokers × 730         storage = provisioned GB, or stored GB where usage-priced
Confluent: capacity units = max(peak in ÷ in per unit, peak out ÷ out per unit, partitions ÷ partitions per unit)
monthly total = sum of the unrounded components, rounded once

Assumptions

• m is messages per second, s the uncompressed message size, ρ the compression ratio (compressed ÷ original), p the peak factor, R the replication factor and G the number of consumer groups. MB is 10^6 bytes, GB is 10^9 bytes and TB is 10^12 bytes (Confluent bills binary gigabytes, 2^30 bytes).
• Compression ratios in the data table are typical for JSON payloads and are modelled, not measured. They depend on payload type, batch size and codec settings. Measure yours and enter it as the override.
• Network is planned at half of line rate at peak. Broker sizes use the sustained bandwidth, not the short 'up to' burst some instance sizes advertise.
• Each consumer group reads the full stream once. Followers are assumed to sit in other availability zones (rack awareness); clients are assumed to be spread evenly over the zones, so a client reaches a leader in another zone with probability (AZ − 1) ÷ AZ.
• Partition guidance counts partition replicas (leaders plus followers) per broker, as the MSK guidance does. The partition constraint expresses the brokers needed to stay within it. It warns but does not add brokers.
• Hourly components bill 730 hours a month; traffic volumes use a 30-day month. Confluent capacity is sized for the peak and billed for every hour, which is conservative.
• Not included: reserved or savings-plan discounts, volume discounts, support plans, taxes, KRaft controllers on self-managed clusters, monitoring, operations staff, and cloud-provider network charges on Confluent Cloud (PrivateLink, peering).

The data behind it

Three ways to reduce this number

1. Compress on the producer with zstd or lz4: it cuts disk, replication and consumer traffic in proportion to the ratio you reach.
2. Shorten retention, or tier cold data to object storage where your platform supports it, so you provision less disk.
3. Let consumers read from followers in their own availability zone (fetch from closest replica) so client traffic stops crossing zones.

Use it from code

The same engine is available as a REST endpoint and as an MCP tool named calc_kafka_throughput_storage_planner. Your scenario as an API call:

curl -X POST 'https://api.x12i.com/v1/calc/kafka-throughput-storage-planner' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
  "input": {
    "messages_per_second": 50000,
    "peak_factor": 2,
    "avg_message_bytes": 1024,
    "compression": "zstd",
    "replication_factor": 3,
    "retention_days": 7,
    "consumer_groups": 2,
    "disk_utilization_target_pct": 70,
    "per_partition_producer_mb_s": 10,
    "per_partition_consumer_mb_s": 20,
    "min_consumer_parallelism": 6,
    "az_count": 3,
    "broker_profile": "8tb-10gbps",
    "platforms": [
      "self_managed",
      "msk",
      "confluent_cloud"
    ],
    "architecture_style": "event_driven"
  },
  "options": {
    "explain": false
  }
}'

Copy as API call Schema: GET /v1/calc/kafka-throughput-storage-planner/schema

Frequently asked questions

Related calculators