Kafka Producers: Getting Events In

Part 2 of Apache Kafka from Zero to Production.

A Kafka producer looks simple, since you hand it a record and move on, but a handful of settings decide whether your events arrive in order, arrive only once, and arrive fast. This post builds on the topics, partitions, and offsets from Part 1 and shows how to configure producers with confidence.

What happens when you call send()

A record has a topic, a value, and optionally a key, headers, and a timestamp. When you send it, the producer serializes the key and value, chooses a partition, and appends the record to an in-memory batch for that partition. A background sender thread then ships batches to the broker that leads the partition.

That means send() is asynchronous. It returns almost immediately, and you learn the outcome later through a callback or a returned future. Most producer tuning is really about what happens between those two moments.

Keys decide where records land

• With a key: the default partitioner hashes the key and maps it to a partition. Every record with the same key goes to the same partition, so those records stay in order. Using a customer ID or order ID as the key is the classic pattern.

• Without a key: the producer spreads records across partitions, filling one batch at a time. Load is balanced, but there is no ordering between records.

Two things to watch. First, adding partitions to an existing topic changes which partition a key maps to for new records, which breaks per-key ordering across the change, so plan partition counts with growth in mind (Part 7 covers sizing). Second, a hot key, such as one customer producing most of your traffic, piles work onto a single partition no matter how many you have.

Batching and compression

Sending one record per network request would be wasteful, so the producer groups records headed for the same partition into a batch. Two settings control when a batch goes out: batch.size, the most data a batch holds, and linger.ms, how long the producer waits for more records before sending what it has. Whichever limit is reached first sends the batch.

Waiting a little adds a few milliseconds of latency but sends far fewer, larger requests, which raises throughput and lowers load on the brokers. Kafka 4.0 raised the linger.ms default from 0 to 5 ms for exactly this reason.

Compression works on whole batches, so bigger batches compress better. The producer supports gzip, snappy, lz4, and zstd, and compression is off by default. lz4 and zstd are common choices: you spend a little CPU to save network bandwidth and broker disk.

How sure do you want to be?

The acks setting decides when the producer considers a write successful:

• acks=0: the producer does not wait for any reply. It is the fastest option, but you cannot know whether a write was lost.

• acks=1: the partition leader confirms it wrote the record. If the leader fails before followers copy it, the record can be lost.

• acks=all: the leader waits until all in-sync replicas have the record. This is the default since Kafka 3.0.

acks=all pairs with the topic setting min.insync.replicas. With a replication factor of 3 and min.insync.replicas set to 2, a write succeeds as long as two replicas are in sync, and fails with an error instead of quietly landing on a single broker. Part 4 explains in-sync replicas in depth.

Retries and idempotent writes

Networks fail. If a producer sends a batch and the acknowledgement never arrives, it retries, but the first attempt may have succeeded after all, which creates a duplicate. Retries can also reorder records when several requests are in flight at once.

The idempotent producer solves both problems. The broker gives each producer an ID, the producer numbers every record it sends to each partition, and the broker discards anything it has already stored while keeping the order intact. It works with up to five requests in flight, so you do not give up throughput. It has been on by default since Kafka 3.0 and requires acks=all.

There are two limits to know about. Retries continue only until delivery.timeout.ms expires, after which your callback receives the error. And idempotence covers a single producer session: a restarted producer gets a new ID, so protection across restarts needs transactions, which Part 5 covers.

Settings at a glance

A minimal producer

Here is a Java producer that sends one record and reports where it landed. Since acks=all and idempotence are already the defaults, they are not set explicitly.

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("compression.type", "lz4");
props.put("linger.ms", "20");

try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    ProducerRecord<String, String> record =
        new ProducerRecord<>("orders", "customer-42", "order-1001: 1 x book");

    producer.send(record, (metadata, exception) -> {
        if (exception != null) {
            System.err.println("Send failed: " + exception.getMessage());
        } else {
            System.out.printf("Wrote to partition %d at offset %d%n",
                metadata.partition(), metadata.offset());
        }
    });
}

Because customer-42 is the key, every later record for that customer lands in the same partition, in order. Closing the producer at the end of the try block flushes anything still waiting in a batch.

Mistakes to avoid

• Ignoring the callback. A send that fails after all retries is only reported there. Without a callback, you can lose data without knowing.

• Waiting on every send. Calling get() on the returned future after each record turns an asynchronous pipeline into a slow, one-at-a-time one and defeats batching.

• Creating a producer per message. A producer is thread-safe and holds connections and buffers, so create one and share it.

• Skipping keys when order matters. Without a key, related records can end up in different partitions with no ordering between them.

• Sending very large records. By default, requests above roughly 1 MB are rejected. Store big payloads elsewhere and send a reference.

Next in the series

This is Part 2 of Apache Kafka from Zero to Production. In Part 3, we look at the other side of the log: consumers, offsets, consumer groups, and what happens during a rebalance.