0

I am trying to understand a behavior which is a bit confusing. We are using Spring boot and KafkaListener approach for the consumers. The configuration has auto commit set to true with the default interval of 5 seconds.

As per the docs, with Kafka auto commit enabled, the commit happens automatically every 5 seconds in the background.

We have a consumer that polls for messages and processes them in about 100 milliseconds. So within the 5 second interval there will be about 50 polls of the messages. When things are working fine, i don't see any issues. There are a couple of scenarios that i am trying to understand:

  1. We continue to poll for messages and say polled for 4.5 seconds and the pod crashes in the middle of processing a polled batch, before the 5 seconds auto commit interval is reached. This in theory should reprocess all the messages since the last commit 4.5 seconds ago, but it reprocesses only the polled batch that did not finish successfully. Does that mean there is some incremental commit that happens in the background, and only the current unfinished batch is not included in the commit offset?
  2. Say I have a polled batch that takes longer than the auto commit interval. I had induced an explicit sleep of 10 seconds in the listener. Per docs, this means the batch should get auto committed so if the batch fails after processing, it would have been committed as the interval is passed. However, when i try to do reproduce this by explicitly force killing the pod, the messages got reprocessed/republished as part of the next poll. Per #1 above, may be the background process is not including the current batch as it's not completed yet?

Any explanation to this behavior? My assumption seems to make sense but I want to be sure that's what is happening. Thanks in advance!

Improve this question
5
  • Even if you set enable.auto.commit=true, Spring Kafka’s @KafkaListener containers by default don’t use Kafka’s native auto-commit. They use container-managed commits: offsets are committed only after your listener method finishes. Commented Aug 31 at 21:59
  • Thanks. Does that mean the listener uses batch mode with commit at the end of each successful poll processing? Is there any documentation that explains this further? Commented Aug 31 at 22:08
  • 1
    docs.spring.io/spring-kafka/reference/kafka/receiving-messages/… Commented Aug 31 at 22:12
  • Thank you! I see this comment which is what you are referring to - Starting with version 2.3, the framework sets enable.auto.commit to false unless explicitly set in the configuration. Previously, the Kafka default (true) was used if the property was not set Commented Sep 1 at 2:31
  • Interestingly, i tried to set the auto commit interval explicitly to 6 seconds hoping for it to be effective and then i had a sleep in the listener for 10 seconds - if the auto commit worked, then the message should have been committed. But when i force restarted the pod, the message was received again making me think the auto commit interval doesn't have any effect whether explicitly set or not.. Commented Sep 2 at 1:37
Related questions
0
Kafka Manual commit scenario Springboot
0
How Kafka Listener re delivers messages when started consuming messages with auto commit set to false
34
How does kafka consumer auto commit work?
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.