Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

[kafka] Custom partition rules

Shulou Source: shulou.com Published: 2022-06-03 08:08:15 11月23日 Update

(1) the data generated by the producer is partitioned by using custom key as hashcode.

Topic can be physically divided into one or more Partition in 1.Kafka

Each partition physically corresponds to a folder, named in the "topicName_partitionIndex" manner

All messages (.log) and index (.index) files for this partition are stored under the folder

# for example, Topic:test0320 uses 3 partitions and 2 replicas 186 187 188-test0320-0 test0320-0 test0320-1test0320-2 test0320-1test0320-2

two。 When producing data, producers can customize Key rules for each message, so that when messages are sent to broker, they will choose which partition to store in according to the partition rules.

3. Partition rules are classes that implement the kafka.producer.Partitioner interface

Tags: Rules data files messages production folders physics producers storage copies multiple that is interfaces methods time indexes generation selection Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno Linux MariaDB Xiaomi OPPO Reno Shulou Information