Kafka partition

1. The function of introducing partition under topic:

Topic is a logical concept and partition is a physical concept.

For performance reasons, if the messages in the topic are stored in only one broker, then the broker will become a bottleneck and cannot scale horizontally. Kafka allocates partition to different servers in the cluster as much as possible through the algorithm.

Partition can also be understood as the encapsulation of segment. One partition corresponds to multiple segment. A segment contains a data file and an index file

Second, kafka partition allocation strategy:

Partition.assignment.strategy= range (default) or roundrobin

Range strategy: sort by partition order, consumers by alphabetical order.

The number of partitions divided by the total number of consumer threads determines how many partitions each consumer thread consumes. If not, the first few consumer threads will consume an extra partition.

Suppose there are 3 consumers and 11 divisions.

C1-0 will consume 0, 1, 2, 3 zones

C1-2 will consume 4, 5, 6, 7 zones

C1-3 will consume 8, 9, 10 zones

Roundrobin strategy: partitions are sorted by hashcode, consumers are sorted alphabetically

Suppose there are 3 consumers and 11 divisions.

C1-0 will consume 0, 3, 6, 9 zones

C1-2 will consume 1, 4, 7, 10 zones

C1-3 will consume 2, 5, 8 zones

Note:

1. A partition can only be consumed by one consumer, but a consumer can consume data from multiple partitions.

2. The new api reserves the possibility of implementing the allocation policy class org.apache.kafka.clients.consumer.RangeAssignor.

3. Zone modification. / kafka-topics.sh-- alter-- topic topic1-- zookeeper zkip:2181/kafka-- partitions 6