What is the water level of Kafka?
This article mainly explains "what is the water level of Kafka". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the water level of Kafka".
What is the water level?
Water level is described in kafka.
The offset of visible data in a partition.
Maybe you need to know these things:
Hw (water level) you can understand it as a global (minimum offset of all replicas) offset
For a partition.
LEO represents the maximum offset of all messages in this copy
It's for one copy, that is, each copy has LEO, and it's different.
The smallest LEO of all copies is the water level.
Water level
Why the water level?
To ensure the consistency of the data
The Operation of Water level in Kafka
When Leader receives a message and writes it successfully
Its LEO is and immediately + 1.
The copy will be synchronized with Leader regularly.
For each message synchronized, its own LEO corresponds to + 1
And the water level is the smallest LEO of all copies.
So it will increase slowly.
The relationship with ACK
In fact, there is no inevitable relationship.
ACK is used to ensure that data is not lost.
Hw is used to ensure the consistency of client consumption.
But when ACK=-1
Because it will wait until the data is fully written to all copies.
To return to success.
In other words, the LEO of all copies is + 1
Its water level must also be + 1.
At this time, data loss can be avoided.
Otherwise, there is a risk of data loss.
Why ACK! =-1 data may be lost
For example, one partition, four copies, one, two, two, three, four:
1:LEO = 10 (Leader)
2:LEO = 8
3:LEO = 7
4:LEO = 6
At this point hw = 6
When No.1 Leader is dead.
The 3rd was elected as the new Leader.
Then all surviving copies will first erase all data except hw.
Then synchronize the data after the water level from Leader
For example, copy 2:
Clear your own data first. 7, 8, and Leo becomes 6.
Then go to Leader to synchronize the data after 6.
Even if the old Leader 1 recovers, the 6-10 data is still there.
You also need to clear all the data after the hw during the downtime.
Resynchronize data from Leader
Thank you for your reading, the above is the content of "what is the water level of Kafka". After the study of this article, I believe you have a deeper understanding of what the water level of Kafka is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!
