Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

How to analyze the multi-consumer problem of kafka bulk consumption

Shulou Source: shulou.com Published: 2022-06-01 13:36:08 11月17日 Update

Today, I will talk to you about how to analyze the multi-consumer problem of kafka bulk consumption. Many people may not know much about it. In order to make you understand better, the editor has summarized the following contents for you. I hope you can get something according to this article.

Package com.llw.medical.bs.listener;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.annotation.TopicPartition;import org.springframework.stereotype.Component;import java.util.List;import java.util.Optional;@Componentpublic class KafakaListener {@ KafkaListener (id = "1", topics = {"topic2"}) public void listen (ConsumerRecord record) {Optional kafkaMessage = Optional.ofNullable (record.value ()) If (kafkaMessage.isPresent ()) {Object message = kafkaMessage.get (); System.out.println ("- record =" + record); System.out.println ("- message =" + message) } @ KafkaListener (id = "2", topicPartitions = {@ TopicPartition (topic = "topic1", partitions = {"1", "2", "3"} / / partitionOffsets = @ PartitionOffset (partition = "1", initialOffset = "4"))}) public void listen2 (ConsumerRecord record) {Optional kafkaMessage = Optional.ofNullable (record.value ()) If (kafkaMessage.isPresent ()) {Object message = kafkaMessage.get (); System.out.println ("- record 1 =" + record); System.out.println ("- message 1 =" + message) } / / id= "4", / / id= "4" @ KafkaListener (id= "4", groupId = "1", topics= "topic1", / * topicPartitions = {@ TopicPartition (topic = "topic1", partitions = {"0"} / / partitionOffsets = @ PartitionOffset (partition = "1", initialOffset = "4")) * / / *} * / containerFactory = "kafkaBatchListener6") public void listen3 (List record: records) {Optional kafkaMessage = Optional.ofNullable (record.value ()) If (kafkaMessage.isPresent ()) {Object message = kafkaMessage.get (); System.out.println ("- record 4 =" + record); / / System.out.println ("- message 4 =" + message) } finally {/ / ack.acknowledge () }} / / id= "5" @ KafkaListener (id= "5", groupId = "1", topics= "topic1", / * topicPartitions = {@ TopicPartition (topic = "topic1", partitions = {"0"} / / partitionOffsets = @ PartitionOffset (partition = "1", initialOffset = "4"))} * / containerFactory = "kafkaBatchListener6") public void listen2 (List record: records) {Optional kafkaMessage = Optional.ofNullable (record.value ()) If (kafkaMessage.isPresent ()) {Object message = kafkaMessage.get (); System.out.println ("- record 6 =" + record); / / System.out.println ("- message 6 =" + message) } finally {/ / ack.acknowledge () }} / / https://www.cnblogs.com/linjiqin/p/13171789.html @ KafkaListener (id = "6", groupId = "1", topics= "topic1", / * topicPartitions = {@ TopicPartition (topic = "topic1", partitions = {"0"} / / partitionOffsets = @ PartitionOffset (partition = "1", initialOffset = "4"))} * / containerFactory = "kafkaBatchListener6") public void listen4 (List record: records) {Optional kafkaMessage = Optional.ofNullable (record.value ()) If (kafkaMessage.isPresent ()) {Object message = kafkaMessage.get (); System.out.println ("- record 3 =" + record); / / System.out.println ("- message 6 =" + message) } finally {/ / ack.acknowledge ();}

A partition can only have one consumer. If multiple consumers will be broadcast mode, each consumer will have a piece of data. Kafka is a primary key for publish and subscribe mode, not a queue mode.

In spring boot integration, if you use the topicPartitions annotation parameter to specify partition, there will be a problem of message repeated consumption. It is best to use topics annotation and specify groupId.

After reading the above, do you have any further understanding of how to analyze the multi-consumer problem of kafka bulk consumption? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Tags: Consumption consumer question content model fee analysis annotation parameter multiple data more best message knowledge article industry information channel queue Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno macOS Xiaomi Apple Shulou Information MySQL