Kafka of message queue (consumption semantics)

1. An introduction to consumption semantics

   at last once: consume at least once (multiple consumption of a message may result in repeated consumption of data)

     reason: when Proudcer generated data, it was already written in broker, but due to the network exception of broker, no ACK was returned. If Producer thinks that the data has not been written successfully, producer will write again, which is equivalent to a piece of data, and has been written multiple times.

   at most once: consume at most once. For messages, it is possible to consume once or not at all.

    reason: when generating data, producer may not succeed in writing data. If broker skips this message, the data will be lost and consumer will not be able to consume.

   exactly once: yes and only once. This is what we need, that is, accurate consumption.

The scenario of consumption semantics in 2.kafka

   at last once: you can first read the data, process the data, and finally record the offset. Of course, if you crash before recording the offset, the new consumer will repeatedly consume this data, resulting in "at least one time"

   at most once: you can read the data first, then record the offset, and finally process the data. In this way, it is possible to crash the data before it is processed in time after the offset, causing the new consumer to continue to process from this offset. Then the data that has not been processed before it has time to be processed will never be processed, resulting in "maximum consumption".

   exactly once: this can be solved by dividing the submission into two phases: once the offset is saved, and once the message is processed successfully.

How to write data accurately in 3.kafka?

Idempotent operation is guaranteed when writing data on the A:Producer side:

Idempotency: only one piece of data is written to the same data no matter how many times it is operated. if it is written repeatedly, the execution is not successful.

When B:broker writes data, atomic operations are guaranteed to either succeed or fail. (retry again and again without success)