All About Exactly Once in Apache Kafka Delivery Guarantee

Apache Kafka is a powerful event streaming platform that provides several different delivery guarantees to ensure that messages are delivered in a reliable and timely manner. One of the most important of these delivery guarantees is exactly once delivery, which ensures that each message sent by a producer will be delivered to a consumer exactly once. In this article, we’ll take a closer look at exactly once delivery in Apache Kafka and how it can be configured to meet the specific needs of your application.

What is Exactly Once Delivery?

Exactly once delivery is a delivery guarantee offered by Apache Kafka that ensures that each message sent by a producer will be delivered to a consumer exactly once. This is achieved by using a combination of transactional producers and idempotent producers, as well as careful management of acknowledgements by the consumer. With exactly once delivery, producers can ensure that messages are sent exactly once, even if the producer crashes or fails. Consumers can also ensure that messages are processed exactly once, even if the consumer crashes or fails.

For example, consider a scenario where a producer sends a message to a topic and the consumer acknowledges the receipt of that message. If the consumer crashes or fails during processing, the message will be redelivered, and the consumer will process the message exactly once. This ensures that the message is delivered exactly once, with no duplicates or lost messages.

Advantages of Exactly Once Delivery

Exactly once delivery has several advantages, including:

• Higher reliability and consistency, as messages are not lost or duplicated.
• Higher availability, as messages are stored on disk for redelivery.
• Better error handling, as messages can be redelivered if necessary.

Disadvantages of Exactly Once Delivery

Exactly once delivery also has some disadvantages, including:

• Higher resource requirements, as messages are stored on disk for redelivery.
• Higher latency, as messages must be acknowledged by the consumer before they are considered sent.

Configuring Exactly Once Delivery

Exactly once delivery is not the default delivery guarantee in Apache Kafka, but it can be configured by setting the following properties:

• acks configuration property set to all
• retries configuration property set to the maximum possible value
• enable.idempotence configuration property set to true

Here’s a simple example of how you might configure a producer for exactly once delivery in Java:

// Configure the producer
Properties producerProperties = new Properties();
producerProperties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
producerProperties.setProperty(ProducerConfig.ACKS_CONFIG, "all");
producerProperties.setProperty(ProducerConfig.RETRIES_CONFIG, Integer.toString(Integer.MAX_VALUE));
producerProperties.setProperty(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");

// Create the producer
KafkaProducer<String, String> producer = new KafkaProducer<>(producerProperties);

In this example, the producer is configured to use the acks configuration property set to all, which means that the producer will wait for all in-sync replicas to acknowledge receipt of a message before considering it sent. The retries configuration property is set to the maximum possible value to ensure that the producer will continue to retry sending a message if necessary. The enable.idempotence configuration property is set to true

Exactly once delivery is a valuable delivery guarantee offered by Apache Kafka that ensures that each message sent by a producer will be delivered to a consumer exactly once. This delivery guarantee is especially useful in applications where reliability, consistency, and accuracy are critical. To configure exactly once delivery in Apache Kafka, simply set the acks, retries, and enable.idempotence configuration properties appropriately. With these properties set, you can be confident that your messages will be delivered exactly once, even in the face of failures or crashes.