Part 7 of Apache Kafka from Zero to Production.
A Kafka cluster that works in a demo is not yet ready for production. The difference is a set of decisions made up front, such as how many partitions to create and how long to keep data, and a set of habits kept up afterward, such as watching the right metrics and restarting brokers safely. This final post covers both.
Sizing partitions
The partition count sets a topic's maximum parallelism, because one consumer group can have at most one active consumer per partition (Part 3). It is also hard to change later. You can add partitions to a topic but never remove them, and adding them changes which partition a key maps to (Part 2).
A practical way to choose a number:
Estimate the target throughput for the topic, both in and out.
Measure how much a single partition can handle on your hardware and with your message sizes, for producers and for consumers.
Divide target by per-partition throughput for each side, and take the larger result.
Round up to leave room for growth, and for the largest consumer group you expect to run.
More partitions are not free. Each one adds open files, memory, replication traffic, and recovery work when a broker restarts. Since KRaft replaced ZooKeeper, clusters handle far more partitions than before, but the cost still grows with the count. Start with a measured number, not a very large one "just in case".
Setting retention
Kafka does not delete records when they are read. It deletes them when they age out. By default, a topic keeps data for 7 days, controlled by retention.ms on the topic (or log.retention.hours on the broker). You can also cap size per partition with retention.bytes, which is unlimited by default.
Two details affect how retention behaves in practice:
Deletion works on whole segment files. Only closed segments are eligible, and the segment currently being written is never deleted. Segments roll at 1 GiB or 7 days by default, so data can outlive its retention time slightly.
Retention is a cleanup policy, not the only one. With
cleanup.policy=compact, Kafka keeps the latest record for each key instead of deleting by age. This suits topics that represent current state, like a customer's latest profile, and a record with a null value (a tombstone) removes its key.
Retention drives your disk budget. As an example, a topic that receives 20 MB per second and keeps 7 days of data holds about 12 TB of records (20 MB/s over 604,800 seconds). With a replication factor of 3, that is roughly 36 TB across the cluster, before compression and before headroom. Leave generous free space, because a full disk takes a broker down. Tiered storage, production-ready since Kafka 3.9, can move older segments to object storage so that brokers keep less data locally.
Securing the cluster
A default Kafka install accepts plaintext connections from anyone who can reach it. Production security has three layers:
Encryption in transit. Use TLS on the listeners, so traffic between clients and brokers, between brokers, and from controllers is encrypted.
Authentication. Clients prove who they are, using mutual TLS or SASL. SCRAM and OAuth are common SASL choices, and Kerberos is supported too. Avoid SASL/PLAIN unless it runs over TLS.
Authorization. Access control lists (ACLs) decide what each authenticated user may do. Grant each application only the access it needs.
A client connecting over SASL and TLS needs settings like these:
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="order-service" password="<secret>";
ssl.truststore.location=/etc/kafka/truststore.jks
ssl.truststore.password=<secret>And ACLs are managed with kafka-acls.sh. The --producer and --consumer shortcuts grant the right bundle of permissions:
bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \
--add --allow-principal User:order-service --producer --topic orders
bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \
--add --allow-principal User:billing --consumer --topic orders --group billing-processorsKafka does not encrypt data at rest itself, so use encrypted disks or volumes. Keep secrets out of config files in source control, and rotate credentials and certificates on a schedule.
Monitoring what matters
Brokers and clients expose their metrics over JMX. A common setup exports them to Prometheus with the JMX exporter and graphs them in Grafana. The signals below catch most serious problems early:
Metric | What it tells you | Healthy value |
|---|---|---|
| Partitions whose followers have fallen out of the ISR | 0 |
| Partitions below | 0 |
| Partitions with no available leader | 0 |
| Whether the cluster has an active controller | Exactly 1 across the cluster |
| How often replicas are dropping out of sync | Near 0 outside restarts |
| How much spare capacity the request threads have | Ideally above 0.3; a sustained drop toward 0 means a saturated broker |
Consumer lag | How far consumers are behind (Part 3) | Stable, not climbing |
Beyond Kafka's own metrics, watch disk usage and disk I/O, network throughput, JVM garbage collection pauses, and open file descriptors on every broker. Alert on conditions that need action, such as disk space or any nonzero offline partition count, rather than on every dashboard wiggle.
Operating the cluster day to day
Rolling restarts. Restart one broker at a time, and wait for
UnderReplicatedPartitionsto return to 0 before moving to the next. Controlled shutdown, which is on by default, moves leadership away from a broker before it stops.Adding brokers. New brokers start empty, and Kafka does not move existing partitions to them on its own. Use
kafka-reassign-partitions.shor a tool such as Cruise Control to rebalance, and expect extra network and disk load while it runs.Upgrades. Upgrade brokers with a rolling restart, read the upgrade notes for your version first, and raise the cluster's metadata version afterward so new features turn on.
Quotas. Client quotas keep one noisy application from starving the rest. For example,
kafka-configs.sh --alter --add-config 'producer_byte_rate=10485760' --entity-type clients --entity-name order-servicelimits that client to about 10 MB per second of writes.Hardware. Kafka leans heavily on the operating system's page cache, so leave most of the machine's memory to the OS rather than a very large JVM heap. Fast disks help, and at scale network bandwidth is often the first limit you hit.
Self-managed or managed?
You can run Kafka yourself on virtual machines or Kubernetes, or use a managed service such as Confluent Cloud, Amazon MSK, Google Cloud Managed Service for Apache Kafka, or Aiven.
Self-managed | Managed service | |
|---|---|---|
Operations effort | You handle upgrades, scaling, patching, and on-call | The provider handles most of it |
Control | Full control over versions, configuration, and networking | Limited to what the provider exposes |
Cost | Lower service cost, higher staffing cost | Higher service cost, lower staffing cost |
Time to start | Days to weeks to build and harden | Minutes to hours |
Managed does not mean hands-off. You still own your topic design, partition counts, client configuration, and lag monitoring. A good rule is to choose managed unless you have a specific reason to run it yourself, such as strict data-residency rules, deep customization needs, or a team that already operates distributed systems well.
Settings at a glance
Setting | Default | Common production choice |
|---|---|---|
| 1 | 3 |
| 1 | 2 |
| true | false, so topics are created deliberately |
| 1 | Set per topic from measured throughput |
| 168 (7 days) | Set per topic with |
| false | false |
Mistakes to avoid
Guessing partition counts. Too few limits scale, and thousands of unused partitions add overhead. Measure first.
Leaving topic auto-creation on. A typo in a client config quietly creates a new topic with default settings.
Having no disk alerts. A full disk stops a broker, and retention math is easy to get wrong.
Running open listeners. Plaintext and unauthenticated access should never reach production.
Restarting brokers too quickly. Moving on before replicas catch up can take partitions below the minimum ISR.
Assuming managed means no operations. The provider runs the brokers, but your design and client choices still decide reliability.
Where to go from here
This is Part 7, the final part of Apache Kafka from Zero to Production. Over the series, you went from topics, partitions, and offsets, through producers, consumers, replication, and delivery guarantees, to Connect, Streams, and running a cluster for real. The best next step is hands-on: stand up a three-broker cluster, build a producer and consumer against it, and then stop a broker and watch Kafka recover.
Frequently Asked Questions
How do I choose the right number of partitions for a topic?
Estimate the target inbound and outbound throughput, measure the maximum throughput a single partition can sustain on your hardware, divide the target by the per‑partition throughput for producers and consumers, take the larger result, and round up to allow growth and the largest expected consumer group.
What are the costs of having many partitions?
Each partition adds open files, memory usage, replication traffic, and recovery work during broker restarts; more partitions increase overall resource consumption even though KRaft allows higher counts.
What is Kafka's default data retention policy?
By default Kafka retains records for 7 days (retention.ms or log.retention.hours set to 168 hours) and does not limit size per partition unless retention.bytes is configured.
How is retention implemented on disk?
Retention deletes whole segment files that are closed; the active segment being written is never deleted, and segments roll at 1 GiB or 7 days by default, so data may persist slightly beyond the configured age.
How can I estimate the disk space needed for a topic?
Multiply the topic's ingest rate by the retention period (e.g., 20 MB/s × 604,800 s ≈ 12 TB), then multiply by the replication factor (e.g., ×3 ≈ 36 TB) and add headroom for compression and operational safety.
What security layers should be enabled for a production Kafka cluster?
Enable TLS encryption for all listeners, configure authentication (mutual TLS or SASL mechanisms such as SCRAM‑SHA‑512, OAuth, or Kerberos), and set up ACLs for authorization; also use encrypted disks and keep secrets out of source‑controlled files.
Which SASL mechanisms are commonly used with Kafka?
SCRAM‑SHA‑512 and OAuth are common choices; Kerberos is also supported, while SASL/PLAIN should only be used over TLS.
What key metrics indicate a healthy Kafka broker?
UnderReplicatedPartitions = 0, UnderMinIsrPartitionCount = 0, OfflinePartitionsCount = 0, ActiveControllerCount = 1, IsrShrinksPerSec ≈ 0, RequestHandlerAvgIdlePercent > 0.3, and stable consumer lag that is not increasing.
What is the recommended process for rolling broker restarts?
Restart one broker at a time, wait until UnderReplicatedPartitions returns to 0 before proceeding, and rely on controlled shutdown (enabled by default) to move leadership away before the broker stops.
What are the main differences between self‑managed and managed Kafka services?
Self‑managed gives full control over versions, configuration, and networking but requires handling upgrades, scaling, patching, and on‑call duties; managed services handle most operational tasks, reduce staffing effort, and start faster, though they limit configuration flexibility and may cost more.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!