How to understand the background and architecture of kafka

This article introduces how to understand the background and structure of kafka. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

Background and architecture introduction

I. background

What is kafka, a distributed messaging system developed by LinkedIn, written in scala, that can scale horizontally and have high throughput.

Originally developed from LinkedIn as the basis for their activity flow and operational data processing pipeline, it was used by a number of companies and slowly grew.

Second, a brief introduction

Kafka is a distributed, publish / subscribe based messaging system.

1. The access performance of constant time complexity can be guaranteed even for data above TB level.

2. High throughput: even on very cheap commercial machines, a single machine can support the transmission of more than 100k messages per second.

3. Partition and distributed message consumption are supported, but messages can only be transmitted sequentially within each partition. And support online horizontal expansion.

Why use message middleware? you must have come into contact with rabbitmq, activemq, redis, etc., and there are probably a lot of feelings. I will not describe in detail, this can only personally participate in the large technical architecture, myself in it, and feel the difference between not using message middleware and using it.

III. Structure

Before we understand the architecture, let's take a look at the basic nouns.

Broker: the server with kafka installed is a broker.

Topic: message analogy, it is best to order a topic for a class of data to be stored and transmitted.

Partition: partition. Topic can be divided into several partitions.

Producer: send the message, the sender.

Consumer: consumer news, consumer.

Consumer Group: each Consumer belongs to a specific Consumer Group.

From the figure, the kafka cluster consists of several producer and consumer as well as broker, as well as a zk cluster. Producer sends data to broker,Consumer through push mode and pulls data through pull mode.

When Producer sends messages to broker, it chooses which partition can not be assigned according to the partition mechanism. If the setting is reasonable, all messages can

It can not be evenly distributed to different partition, and the load balance is realized.

This is how to understand the background and architecture of kafka. I hope the above content can be helpful to you and learn more knowledge. If you think the article is good, you can share it for more people to see.