Kafka installation and deployment
1 download and install JDK
Refer to the installation of jdk, jdk1.6+ is required.
2 installation of Kafka 2.1 download kafka
Download address: http://mirrors.hust.edu.cn/apache/kafka/0.10.0.1/kafka_2.11-0.10.0.1.tgz
2.2 extract the installation package
$tar zvxf kafka_2.11-0.10.0.1.tgz
2.3 copy the extracted folder to the installation directory
$cp-r kafka_2.11-0.10.0.1 / software/
2.4 create a soft link to the kafka installation directory
$ln-s / software/kafka_2.11-0.10.0.1/software/kafka
2.5 configure link variables
Modify the contents of / etc/profile file:
Export KAFKA_HOME=/software/kafka
Export PATH=$KAFKA_HOME/bin:$FLUME_HOME/bin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:
$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
Export CLASSPATH=.:$KAFKA_HOME/libs/*:$FLUME_HOME/lib/*:$HBASE_HOME/lib/*:$JAVA_
HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
2.6 start kafka
1) start kafka before starting zookeeper
$zookeeper-server-start.sh / software/kafka/config/zookeeper.properties &
2) start kafka
$kafka-server-start.sh / software/kafka/config/server.properties &
3 experience single node Kafka3.1 to create topic
Kafka-topics.sh-- create-- zookeeperlocalhost:2181-- replication-factor 1-- partitions 1-- topic test
Topic created by query
$kafka-topics.sh-list-zookeeperlocalhost:2181
3.2 send messages
$kafka-console-producer.sh-broker-listlocalhost:9092-topic test
This is a message
This is another message
3.3 consumption messages
$kafka-console-consumer.sh-bootstrap-serverlocalhost:9092-zookeeper localhost:2181-topic test-from-beginning
This is a message
This is another message
4 build a multi-broker Kafka cluster 4.1 replication configuration file
$cd / software/kafka/config/
$cp server.properties s1.properties
$cp server.properties s2.properties
$cp server.properties s3.properties
4.2 modify the configuration file
Config/s1.properties:
Broker.id=1
Listeners=PLAINTEXT://:9093
Log.dir=/tmp/kafka-logs-1
Config/s2.properties:
Broker.id=2
Listeners=PLAINTEXT://:9094
Log.dir=/tmp/kafka-logs-2
Config/s3.properties:
Broker.id=3
Listeners=PLAINTEXT://:9095
Log.dir=/tmp/kafka-logs-3
4.3 start kafka
$kafka-server-start.sh / software/kafka/config/s1.properties &
$kafka-server-start.sh / software/kafka/config/s2.properties &
$kafka-server-start.sh / software/kafka/config/s3.properties&
4.4 create topic
$kafka-topics.sh-create--zookeeper localhost:2181-replication-factor 3-partitions 1-topic my-replicated-topic
View topic description
$kafka-topics.sh-describe-zookeeper localhost:2181-topic my-replicated-topic
4.5 send messages
$kafka-console-producer.sh-broker-listlocalhost:9093-topic my-replicated-topic
My test msg 1
My test msg 2
My test msg 3
4.6 consumption message
$kafka-console-consumer.sh--zookeeper localhost:2181-from-beginning-topic my-replicated-topic
5 Fault tolerance Test 5.1 find S1 process
$ps-ef | grep s1.properties
5.2 Kill process S1
Kill pid
5.3 View topic description
$> kafka-topics.sh-describe-zookeeperlocalhost:2181-topic my-replicated-topic
Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas:1,2,0 Isr: 2,0
5.4 launch consumer consumption theme message
$> kafka-console-consumer.sh--zookeeper localhost:2181-from-beginning-topic my-replicated-topic
My test message 1 my test message 26 fully distributed build 6.1 distribute kafka installation folders to the remaining cluster servers
$xsync kafka_2.11-0.10.0.1 /
$xsync kafka
6.2 distribute environment variables
# xsync / etc/profile
6.3 modify the server.properties properties file and distribute it
Broker.id=203 # modify the id of broker
Log.dirs=/home/hadoop/kafka/logs # the storage path of the modification log
Zookeeper.connect=s201:2181,s202:2181,s203:2181 # modify the connection path of zookeeper
6.4 start the Kafka cluster
Start the kafka cluster on each server that deploys kafka
$kafka-server-start.sh / software/kafka/config/server.properties &
$kafka-server-start.sh / software/kafka/config/server.properties &
$kafka-server-start.sh / software/kafka/config/server.properties &
