{"schemaVersion":"1.0","type":"TechArticle","types":["Article","TechArticle"],"slug":"running-kafka-in-production-sizing-retention-security-and-monitoring-83hbz","url":"https://zyvop.com/running-kafka-in-production-sizing-retention-security-and-monitoring-83hbz","title":"Running Kafka in Production: Sizing, Retention, Security, and Monitoring","subtitle":"Practical guidance on sizing partitions, setting retention, securing the cluster, monitoring what matters, and choosing between self-managed and managed Kafka.","tldr":"Running Kafka well takes more than a working demo. This final post covers how many partitions to use, how to set retention, how to lock down access, which metrics to watch, and when a managed service is the better choice.","keywords":["monitoring","Kafka Operations","Capacity Planning","security","DevOps","Tutorial","Apache Kafka from Zero to Production"],"entities":["Samod Alex","monitoring","Kafka Operations","Capacity Planning","security","DevOps","Tutorial","Apache Kafka from Zero to Production","ZyVOP"],"keyTakeaways":["Part 7 of Apache Kafka from Zero to Production.","A Kafka cluster that works in a demo is not yet ready for production.","The difference is a set of decisions made up front, such as how many partitions to create and how long to keep data, and a set of habits kept up afterward, such as watching the right metrics and restarting brokers safely."],"headings":["Sizing partitions","Setting retention","Securing the cluster","Monitoring what matters","Operating the cluster day to day","Self-managed or managed?","Settings at a glance","Mistakes to avoid","Where to go from here","Frequently Asked Questions","How do I choose the right number of partitions for a topic?","What are the costs of having many partitions?","What is Kafka's default data retention policy?","How is retention implemented on disk?","How can I estimate the disk space needed for a topic?","What security layers should be enabled for a production Kafka cluster?","Which SASL mechanisms are commonly used with Kafka?","What key metrics indicate a healthy Kafka broker?","What is the recommended process for rolling broker restarts?","What are the main differences between self‑managed and managed Kafka services?"],"outboundLinks":[],"contentText":"Part 7 of Apache Kafka from Zero to Production. A Kafka cluster that works in a demo is not yet ready for production. The difference is a set of decisions made up front, such as how many partitions to create and how long to keep data, and a set of habits kept up afterward, such as watching the right metrics and restarting brokers safely. This final post covers both. Sizing partitions The partition count sets a topic's maximum parallelism, because one consumer group can have at most one active consumer per partition (Part 3). It is also hard to change later. You can add partitions to a topic but never remove them, and adding them changes which partition a key maps to (Part 2). A practical way to choose a number: Estimate the target throughput for the topic, both in and out. Measure how much a single partition can handle on your hardware and with your message sizes, for producers and for consumers. Divide target by per-partition throughput for each side, and take the larger result. Round up to leave room for growth, and for the largest consumer group you expect to run. More partitions are not free. Each one adds open files, memory, replication traffic, and recovery work when a broker restarts. Since KRaft replaced ZooKeeper, clusters handle far more partitions than before, but the cost still grows with the count. Start with a measured number, not a very large one \"just in case\". Setting retention Kafka does not delete records when they are read. It deletes them when they age out. By default, a topic keeps data for 7 days, controlled by retention.ms on the topic (or log.retention.hours on the broker). You can also cap size per partition with retention.bytes, which is unlimited by default. Two details affect how retention behaves in practice: Deletion works on whole segment files. Only closed segments are eligible, and the segment currently being written is never deleted. Segments roll at 1 GiB or 7 days by default, so data can outlive its retention time slightly. Retention is a cleanup policy, not the only one. With cleanup.policy=compact, Kafka keeps the latest record for each key instead of deleting by age. This suits topics that represent current state, like a customer's latest profile, and a record with a null value (a tombstone) removes its key. Retention drives your disk budget. As an example, a topic that receives 20 MB per second and keeps 7 days of data holds about 12 TB of records (20 MB/s over 604,800 seconds). With a replication factor of 3, that is roughly 36 TB across the cluster, before compression and before headroom. Leave generous free space, because a full disk takes a broker down. Tiered storage, production-ready since Kafka 3.9, can move older segments to object storage so that brokers keep less data locally. Securing the cluster A default Kafka install accepts plaintext connections from anyone who can reach it. Production security has three layers: Encryption in transit. Use TLS on the listeners, so traffic between clients and brokers, between brokers, and from controllers is encrypted. Authentication. Clients prove who they are, using mutual TLS or SASL. SCRAM and OAuth are common SASL choices, and Kerberos is supported too. Avoid SASL/PLAIN unless it runs over TLS. Authorization. Access control lists (ACLs) decide what each authenticated user may do. Grant each application only the access it needs. A client connecting over SASL and TLS needs settings like these: security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-512 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=\"order-service\" password=\"<secret>\"; ssl.truststore.location=/etc/kafka/truststore.jks ssl.truststore.password=<secret>And ACLs are managed with kafka-acls.sh. The --producer and --consumer shortcuts grant the right bundle of permissions: bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \\ --add --allow-principal User:order-service --producer --topic orders bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config admin.properties \\ --add --allow-principal User:billing --consumer --topic orders --group billing-processorsKafka does not encrypt data at rest itself, so use encrypted disks or volumes. Keep secrets out of config files in source control, and rotate credentials and certificates on a schedule. Monitoring what matters Brokers and clients expose their metrics over JMX. A common setup exports them to Prometheus with the JMX exporter and graphs them in Grafana. The signals below catch most serious problems early: Metric What it tells you Healthy value UnderReplicatedPartitions Partitions whose followers have fallen out of the ISR 0 UnderMinIsrPartitionCount Partitions below min.insync.replicas, so acks=all writes fail 0 OfflinePartitionsCount Partitions with no available leader 0 ActiveControllerCount Whether the cluster has an active controller Exactly 1 across the cluster IsrShrinksPerSec How often replicas are dropping out of sync Near 0 outside restarts RequestHandlerAvgIdlePercent How much spare capacity the request threads have Ideally above 0.3; a sustained drop toward 0 means a saturated broker Consumer lag How far consumers are behind (Part 3) Stable, not climbing Beyond Kafka's own metrics, watch disk usage and disk I/O, network throughput, JVM garbage collection pauses, and open file descriptors on every broker. Alert on conditions that need action, such as disk space or any nonzero offline partition count, rather than on every dashboard wiggle. Operating the cluster day to day Rolling restarts. Restart one broker at a time, and wait for UnderReplicatedPartitions to return to 0 before moving to the next. Controlled shutdown, which is on by default, moves leadership away from a broker before it stops. Adding brokers. New brokers start empty, and Kafka does not move existing partitions to them on its own. Use kafka-reassign-partitions.sh or a tool such as Cruise Control to rebalance, and expect extra network and disk load while it runs. Upgrades. Upgrade brokers with a rolling restart, read the upgrade notes for your version first, and raise the cluster's metadata version afterward so new features turn on. Quotas. Client quotas keep one noisy application from starving the rest. For example, kafka-configs.sh --alter --add-config 'producer_byte_rate=10485760' --entity-type clients --entity-name order-service limits that client to about 10 MB per second of writes. Hardware. Kafka leans heavily on the operating system's page cache, so leave most of the machine's memory to the OS rather than a very large JVM heap. Fast disks help, and at scale network bandwidth is often the first limit you hit. Self-managed or managed? You can run Kafka yourself on virtual machines or Kubernetes, or use a managed service such as Confluent Cloud, Amazon MSK, Google Cloud Managed Service for Apache Kafka, or Aiven. Self-managed Managed service Operations effort You handle upgrades, scaling, patching, and on-call The provider handles most of it Control Full control over versions, configuration, and networking Limited to what the provider exposes Cost Lower service cost, higher staffing cost Higher service cost, lower staffing cost Time to start Days to weeks to build and harden Minutes to hours Managed does not mean hands-off. You still own your topic design, partition counts, client configuration, and lag monitoring. A good rule is to choose managed unless you have a specific reason to run it yourself, such as strict data-residency rules, deep customization needs, or a team that already operates distributed systems well. Settings at a glance Setting Default Common production choice default.replication.factor 1 3 min.insync.replicas 1 2 auto.create.topics.enable true false, so topics are created deliberately num.partitions 1 Set per topic from measured throughput log.retention.hours 168 (7 days) Set per topic with retention.ms unclean.leader.election.enable false false Mistakes to avoid Guessing partition counts. Too few limits scale, and thousands of unused partitions add overhead. Measure first. Leaving topic auto-creation on. A typo in a client config quietly creates a new topic with default settings. Having no disk alerts. A full disk stops a broker, and retention math is easy to get wrong. Running open listeners. Plaintext and unauthenticated access should never reach production. Restarting brokers too quickly. Moving on before replicas catch up can take partitions below the minimum ISR. Assuming managed means no operations. The provider runs the brokers, but your design and client choices still decide reliability. Where to go from here This is Part 7, the final part of Apache Kafka from Zero to Production. Over the series, you went from topics, partitions, and offsets, through producers, consumers, replication, and delivery guarantees, to Connect, Streams, and running a cluster for real. The best next step is hands-on: stand up a three-broker cluster, build a producer and consumer against it, and then stop a broker and watch Kafka recover. Frequently Asked Questions How do I choose the right number of partitions for a topic? Estimate the target inbound and outbound throughput, measure the maximum throughput a single partition can sustain on your hardware, divide the target by the per‑partition throughput for producers and consumers, take the larger result, and round up to allow growth and the largest expected consumer group. What are the costs of having many partitions? Each partition adds open files, memory usage, replication traffic, and recovery work during broker restarts; more partitions increase overall resource consumption even though KRaft allows higher counts. What is Kafka's default data retention policy? By default Kafka retains records for 7 days (retention.ms or log.retention.hours set to 168 hours) and does not limit size per partition unless retention.bytes is configured. How is retention implemented on disk? Retention deletes whole segment files that are closed; the active segment being written is never deleted, and segments roll at 1 GiB or 7 days by default, so data may persist slightly beyond the configured age. How can I estimate the disk space needed for a topic? Multiply the topic's ingest rate by the retention period (e.g., 20 MB/s × 604,800 s ≈ 12 TB), then multiply by the replication factor (e.g., ×3 ≈ 36 TB) and add headroom for compression and operational safety. What security layers should be enabled for a production Kafka cluster? Enable TLS encryption for all listeners, configure authentication (mutual TLS or SASL mechanisms such as SCRAM‑SHA‑512, OAuth, or Kerberos), and set up ACLs for authorization; also use encrypted disks and keep secrets out of source‑controlled files. Which SASL mechanisms are commonly used with Kafka? SCRAM‑SHA‑512 and OAuth are common choices; Kerberos is also supported, while SASL/PLAIN should only be used over TLS. What key metrics indicate a healthy Kafka broker? UnderReplicatedPartitions = 0, UnderMinIsrPartitionCount = 0, OfflinePartitionsCount = 0, ActiveControllerCount = 1, IsrShrinksPerSec ≈ 0, RequestHandlerAvgIdlePercent > 0.3, and stable consumer lag that is not increasing. What is the recommended process for rolling broker restarts? Restart one broker at a time, wait until UnderReplicatedPartitions returns to 0 before proceeding, and rely on controlled shutdown (enabled by default) to move leadership away before the broker stops. What are the main differences between self‑managed and managed Kafka services? Self‑managed gives full control over versions, configuration, and networking but requires handling upgrades, scaling, patching, and on‑call duties; managed services handle most operational tasks, reduce staffing effort, and start faster, though they limit configuration flexibility and may cost more.","contentHash":"sha256:dd1b7cddc2aa13877ba05bc459756753d19611d3cc141b16109f0d23ae3ca49f","authorName":"Samod Alex","authorUrl":"https://zyvop.com/author/samod","authorSameAs":[],"category":"Tutorial","tags":["monitoring","Kafka Operations","Capacity Planning","security","DevOps"],"audience":"Software engineers and developers building applications with Tutorial","tone":"Practical and evidence-based engineering guidance","readingTimeMinutes":8,"wordCount":1868,"faqs":[{"question":"How do I choose the right number of partitions for a topic?","answer":"Estimate the target inbound and outbound throughput, measure the maximum throughput a single partition can sustain on your hardware, divide the target by the per‑partition throughput for producers and consumers, take the larger result, and round up to allow growth and the largest expected consumer group."},{"question":"What are the costs of having many partitions?","answer":"Each partition adds open files, memory usage, replication traffic, and recovery work during broker restarts; more partitions increase overall resource consumption even though KRaft allows higher counts."},{"question":"What is Kafka's default data retention policy?","answer":"By default Kafka retains records for 7 days (retention.ms or log.retention.hours set to 168 hours) and does not limit size per partition unless retention.bytes is configured."},{"question":"How is retention implemented on disk?","answer":"Retention deletes whole segment files that are closed; the active segment being written is never deleted, and segments roll at 1 GiB or 7 days by default, so data may persist slightly beyond the configured age."},{"question":"How can I estimate the disk space needed for a topic?","answer":"Multiply the topic's ingest rate by the retention period (e.g., 20 MB/s × 604,800 s ≈ 12 TB), then multiply by the replication factor (e.g., ×3 ≈ 36 TB) and add headroom for compression and operational safety."},{"question":"What security layers should be enabled for a production Kafka cluster?","answer":"Enable TLS encryption for all listeners, configure authentication (mutual TLS or SASL mechanisms such as SCRAM‑SHA‑512, OAuth, or Kerberos), and set up ACLs for authorization; also use encrypted disks and keep secrets out of source‑controlled files."},{"question":"Which SASL mechanisms are commonly used with Kafka?","answer":"SCRAM‑SHA‑512 and OAuth are common choices; Kerberos is also supported, while SASL/PLAIN should only be used over TLS."},{"question":"What key metrics indicate a healthy Kafka broker?","answer":"UnderReplicatedPartitions = 0, UnderMinIsrPartitionCount = 0, OfflinePartitionsCount = 0, ActiveControllerCount = 1, IsrShrinksPerSec ≈ 0, RequestHandlerAvgIdlePercent > 0.3, and stable consumer lag that is not increasing."},{"question":"What is the recommended process for rolling broker restarts?","answer":"Restart one broker at a time, wait until UnderReplicatedPartitions returns to 0 before proceeding, and rely on controlled shutdown (enabled by default) to move leadership away before the broker stops."},{"question":"What are the main differences between self‑managed and managed Kafka services?","answer":"Self‑managed gives full control over versions, configuration, and networking but requires handling upgrades, scaling, patching, and on‑call duties; managed services handle most operational tasks, reduce staffing effort, and start faster, though they limit configuration flexibility and may cost more."}],"primaryTopic":"Tutorial","publishedAt":"2026-10-17T07:34:00.029Z","updatedAt":"2026-10-17T06:07:56.929Z","canonicalUrl":"https://zyvop.com/running-kafka-in-production-sizing-retention-security-and-monitoring-83hbz"}