Scaling Kafka: A Practical Guide from Thousands to Millions of Messages per Second

What actually breaks as traffic grows, and the levers that fix it.

Kafka has a reputation for scaling effortlessly, and the reputation is mostly earned. But horizontally scalable is not the same as scales without thought. Teams keep hitting the same walls: a topic with too few partitions, consumers that fall behind, one overloaded broker, a rebalance storm at 2 a.m. This post walks through how Kafka scales, in the order you will usually need to think about it.

Start with the mental model

The partition is the unit of everything in Kafka. It is the unit of parallelism (a partition is read by at most one consumer within a group), the unit of ordering (messages are ordered only within a partition), and the unit of storage and replication (each partition is an append-only log on a broker, copied to other brokers).

That gives you a simple way to reason about capacity: total throughput is roughly the number of partitions, spread across brokers, times what each partition can sustain. Scaling Kafka is mostly about keeping work evenly distributed across partitions, brokers, and consumers.

Step 1: Size your partitions deliberately

A common rule of thumb: measure the throughput a single partition gives you on the producer side (p) and on the consumer side (c), then compute:

partitions ≈ target throughput / min(p, c)

Then add headroom for growth. Why not just pick 1,000 and move on? Over-partitioning has real costs:

More open file handles and memory use on every broker
Longer leader elections and slower recovery after a broker failure
More replication traffic and request overhead
Slower consumer group rebalances

Two decisions are also hard to undo. You can never reduce a topic's partition count, and increasing it changes the key-to-partition mapping, which breaks per-key ordering across old and new data. Pick a number with two to three times headroom up front rather than resizing under pressure.

Choose keys carefully too. Skewed keys (one enormous customer, a null key bucket, a default value) create hot partitions, where one broker and one consumer do all the work while the rest sit idle. If lag is uneven across partitions, suspect key skew first.

Step 2: Tune the producers

Most producer throughput comes from batching and compression, not from adding more producers.

# Batching: trade a few ms of latency for much higher throughput
linger.ms=10
batch.size=131072

# Compression: less network, disk, and replication traffic
# (lz4 favors speed, zstd favors ratio)
compression.type=lz4

# Durability
acks=all
enable.idempotence=true

A few notes on these settings:

Batching: letting the producer wait briefly to fill batches means fewer requests and better compression ratios. Raise linger.ms and batch.size together and measure.
Compression: batches are compressed by the producer and stored compressed by the broker, so it saves network, disk, and replication bandwidth at once. lz4 favors speed, zstd favors ratio.
Durability: acks=all combined with min.insync.replicas=2 and a replication factor of 3 is the standard safe setup. Idempotence prevents duplicates on retries and is on by default in modern clients.
Message size: Kafka is happiest with messages in the kilobyte range. For large payloads, store the blob in object storage and send a reference.

Step 3: Scale consumers without causing rebalances

Parallelism within a consumer group is capped by partition count. A topic with 12 partitions can keep at most 12 consumers busy in one group; any extra consumers sit idle.

When consumers fall behind, work through this in order:

Find the real bottleneck. Most lag comes from slow downstream work such as database writes and HTTP calls, not from Kafka itself.
Batch the processing. Handle records in batches and commit offsets only after the work completes.
Tune fetching. Adjust fetch.min.bytes, fetch.max.wait.ms, and max.poll.records to match your workload.
Add partitions and consumers only if you are truly at the parallelism ceiling.

Rebalances deserve special attention, because every join, leave, or missed heartbeat can pause consumption. To reduce them:

Use the cooperative sticky assignor so only the affected partitions move.
Use static membership (group.instance.id) so routine restarts do not trigger a rebalance.
Keep max.poll.interval.ms comfortably above your worst-case batch processing time.
On Kafka 4.0 and later, look at the new consumer group protocol (KIP-848, enabled with group.protocol=consumer), which largely removes stop-the-world rebalances.

Step 4: Scale the brokers

Adding a broker does not automatically help. A new broker joins empty, and existing partitions stay where they are until you move them. In practice:

Reassign partitions after adding brokers, using kafka-reassign-partitions or a tool like Cruise Control that plans moves for you.
Throttle replication during moves so rebalancing does not starve live traffic.
Balance leaders, not just replicas. Leaders take all the produce traffic, so a skewed leader count means a skewed load.
Spread replicas across racks or availability zones with broker.rack, so losing a zone does not lose a partition.

On the hardware side:

Network usually saturates first. With a replication factor of 3, every byte produced becomes roughly two more bytes between brokers, before counting consumer reads.
Protect the page cache. Kafka leans on the operating system's page cache for fast reads, so give brokers a modest JVM heap and leave the rest of the memory to the OS.
Disks matter more for recovery than steady state. Sequential writes are fast even on modest disks, but fast storage helps when brokers catch up after a restart or when consumers read old data.

Step 5: Plan for storage growth

Retention is capacity planning in disguise:

disk needed ≈ ingest rate × retention period × replication factor

At 50 MB/s of ingress, 7 days of retention, and a replication factor of 3, that is about 4.3 TB per day, 30 TB for a week, and roughly 90 TB on disk, before compression. Numbers like that are why storage strategy matters:

Shorten retention where consumers do not need the history.
Use log compaction for changelog-style topics that only need the latest value per key.
Consider tiered storage (KIP-405), production-ready since Kafka 3.9, which moves older segments to object storage so you can scale storage separately from brokers.

Step 6: Use the modern control plane

Kafka 4.0 removed ZooKeeper entirely in favor of KRaft, where the brokers manage cluster metadata themselves. The practical benefits are simpler operations, faster controller failover, and a much higher partition ceiling per cluster. If you are still on ZooKeeper, migrating to KRaft is the prerequisite for moving to the 4.x line.

Monitor what matters

You cannot scale what you cannot see. These are the signals worth alerting on:

Consumer lag, per partition, along with its trend rather than just its current value
Under-replicated partitions, which should sit at zero
ISR shrink and expand rate, an early warning of struggling brokers or networks
Request handler and network processor idle percentage, where sustained low values mean a saturated broker
Produce and fetch request latency, especially p99
Disk usage and I/O, plus partition and leader counts per broker to catch skew

Common pitfalls

Hot partitions from skewed keys, hiding behind healthy cluster-wide averages
Over-partitioning because more felt safer
Giant messages that bloat batches and memory
Rebalance storms from slow consumers that miss their poll deadline
Scaling the cluster to fix a slow consumer. If lag comes from downstream work, more brokers will not help
Skipping load tests. Test with production-like message sizes, compression, and real consumer processing times

When one cluster is not enough

Eventually you may want multiple clusters: for regional proximity, for isolating noisy workloads, or to limit the blast radius of an incident. MirrorMaker 2 and cluster linking replicate data between them. That adds operational complexity, so exhaust the options above before splitting a cluster that a well-tuned, well-balanced setup could still handle.

A scaling checklist

Measure per-partition throughput on both producer and consumer sides.
Size partitions with headroom, and audit your keys for skew.
Turn on batching and compression, and use acks=all with idempotence.
Make consumers cheap to restart: cooperative rebalancing and static membership.
After adding brokers, reassign partitions and rebalance leaders, with replication throttled.
Budget disk from ingest rate, retention, and replication factor, and consider tiered storage.
Run on KRaft, and alert on lag, under-replicated partitions, and broker idle percentages.

Conclusion

Kafka scales well because its design is simple: independent partition logs, distributed across brokers, read in parallel by consumer groups. Nearly every scaling problem comes down to one of those three things being unevenly loaded. Size partitions with care, batch and compress on the producer side, keep consumers stable, rebalance brokers deliberately, and watch the right metrics. Do that, and Kafka will take you a long way before you need anything more exotic.