Inside the Kafka Cluster: Brokers, Replication, and KRaft

Part 4 of Apache Kafka from Zero to Production.

Kafka keeps working when a server dies because every partition is copied across several brokers, and because the cluster has a built-in way to agree on who is in charge. Parts 2 and 3 covered the clients. This post goes inside the cluster to show how data is stored, copied, and recovered.

Brokers and partitions

A Kafka cluster is a set of brokers. Each broker stores some of the partitions of each topic as files on disk, and each partition is split into segment files that are written sequentially and deleted as a whole once they pass the retention limit. Spreading partitions across brokers is what lets a topic grow beyond the size and throughput of one machine.

For every partition, exactly one broker acts as the leader. Producers write to the leader, and consumers read from it by default. Because different partitions have different leaders, the work is spread across the whole cluster rather than landing on one broker.

Replication: copies of every partition

The replication factor sets how many copies of each partition exist. One copy is the leader, and the others are followers. Followers do nothing but fetch new records from the leader, the same way a consumer would, and append them to their own logs.

Here is a topic called orders with three partitions and a replication factor of 3 on a three-broker cluster:

Every broker holds a copy of every partition, but each one leads a different partition. If you lose any single broker, two copies of its data survive elsewhere.

In-sync replicas and the high watermark

Not every follower is always up to date. The leader keeps track of the in-sync replicas (ISR): itself plus every follower that has kept pace. A follower that falls behind for longer than replica.lag.time.max.ms (30 seconds by default) is dropped from the ISR and added back once it catches up.

The ISR drives two important behaviors:

• The high watermark. Consumers can only read up to the highest offset that every in-sync replica has copied. This guarantees that a consumer never sees a record that could disappear if the leader failed.

• Write acknowledgements. With acks=all, the producer's write is confirmed once all in-sync replicas have it, as described in Part 2. The topic setting min.insync.replicas sets the smallest ISR size at which writes are still accepted.

When a broker fails

Here is what happens if Broker 2, the leader of orders-1, crashes:

1. The cluster controller notices that Broker 2 has stopped sending heartbeats.

2. For each partition Broker 2 led, the controller picks a new leader from that partition's ISR. Here, Broker 1 or Broker 3 takes over orders-1.

3. The controller records the change, and brokers and clients refresh their metadata.

4. Producers retry their failed requests against the new leader (Part 2), and consumers carry on from their committed offsets (Part 3).

5. When Broker 2 comes back, it discards any records the new leader does not have, catches up, and rejoins the ISR.

Because the new leader came from the ISR, it already holds every record that was acknowledged under acks=all, so nothing acknowledged is lost.

What if no in-sync replica is available? By default, Kafka leaves the partition offline until one returns, which protects consistency at the cost of availability. Setting unclean.leader.election.enable=true allows an out-of-sync replica to take over instead. The partition comes back sooner, but records the old leader had acknowledged can be lost. It is off by default for good reason.

KRaft: the cluster's own control plane

Something has to track which brokers are alive, which topics exist, and who leads each partition. For years that job belonged to a separate system, Apache ZooKeeper. Kafka now does it itself with KRaft, which uses the Raft consensus protocol. Kafka 4.0 removed ZooKeeper entirely, so clusters still running on it must be migrated to KRaft before upgrading.

In KRaft, a small group of controller nodes keeps the cluster metadata in an internal log, replicated among themselves by Raft. One controller is the active leader of the quorum, and the others are hot standbys that already hold the full metadata, so failover is fast. Brokers fetch metadata from the controllers instead of asking an external system.

Each node's role is set with process.roles, which can be broker, controller, or both. Running both roles on one node is fine for development, but production clusters normally use dedicated controllers. Run three controllers to tolerate one failure, or five to tolerate two. Use an odd number, because a quorum needs a majority: four controllers survive no more failures than three.

Try it: look inside your cluster

On a three-broker cluster, create a topic with replication and a minimum ISR:

bin/kafka-topics.sh --create --topic orders \
  --partitions 3 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --bootstrap-server localhost:9092

Then inspect where everything lives:

bin/kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092
bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status

The first command prints, for each partition, its Leader, its Replicas, and its Isr. Stop one broker and run it again: the leaders move, and the stopped broker disappears from the Isr column. The second command shows the controller quorum, including which controller is the current leader. On the single local broker from Part 1, every partition has just one replica, so you will see the same columns without any redundancy.

Settings at a glance

Mistakes to avoid

• Running production topics with a replication factor of 1. One disk or broker failure loses the partition.

• Setting min.insync.replicas equal to the replication factor. Then a single broker restart blocks writes. A factor of 3 with a minimum of 2 survives one broker down without losing data or write availability.

• Enabling unclean leader election to fix an outage. It trades acknowledged data for availability, so do it only when you accept that loss.

• Putting all replicas in one rack or zone. Set broker.rack so copies spread across failure domains.

• Using an even number of controllers. Two or four controllers add cost without adding fault tolerance.

Next in the series

This is Part 4 of Apache Kafka from Zero to Production. In Part 5, we look at delivery guarantees: at-most-once, at-least-once, and exactly-once, and the settings that decide which one you get.