Flume extracts mysql data to kafka

For kafka+zookeeper building, see the article tutorial url.

Flume installation:

1. Download address

2. Installation-the following figure

Create new databases and tables

3. Configure new conf/mysql-flume.conf

[root@node191 apache-flume-1.8.0-bin] # cat conf/mysql-flume.conf a1.channels = ch-1a1.sources = src-1a1.sinks = k1#sql source# For each one of the sources The type is defineda1.sources.src-1.type = org.keedio.flume.source.SQLSourcea1.sources.src-1.hibernate.connection.url = jdbc:mysql://192.168.3.191:3306/chenhuachao# Hibernate Database connection propertiesa1.sources.src-1.hibernate.connection.user = roota1.sources.src-1.hibernate.connection.password = abcd@1234a1.sources.src-1.hibernate.connection.autocommit = truea1.sources.src-1.hibernate.dialect = org.hibernate.dialect.MySQL5Dialecta1. Sources.src-1.hibernate.connection.driver_class = com.mysql.jdbc.Drivera1.sources.src-1.run.query.delay=5000a1.sources.src-1.status.file.path = / opt/apache-flume-1.8.0-bina1.sources.src-1.status.file.name = sqlSource.status# Custom querya1.sources.src-1.start.from = 0a1.sources.src-1.custom.query = select `id` `name` from testa1.sources.src-1.batch.size = 1000a1.sources.src-1.max.rows = 1000a1.sources.src-1.hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvidera1.sources.src-1.hibernate.c3p0.min_size=1a1.sources.src-1.hibernate.c3p0.max_size=10### # a1.channels.ch-1.type = memorya1.channels.ch-1.capacity = 10000a1.channels.ch-1.transactionCapacity = 10000a1.channels.ch-1.byteCapacityBufferPercentage = 20a1.channels.ch-1.byteCapacity = 800000 yuan # # a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.topic = TestTopica1.sinks.k1.brokerList = 192.168.3.191:9092192.168.3.193:9092192.168.3.194:9092a1.sinks.k1.requiredAcks = 1a1.sinks.k1.batchSize = 20a1.sinks.k1.channel = c1a1.sinks.k1.channel = ch-1a1.sources.src-1.channels=ch-1

4. Add the mysql driver to the lib directory of flume

$wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.35.tar.gz$ tar xzf mysql-connector-java-5.1.35.tar.gz$ cp mysql-connector-java-5.1.35-bin.jar lib/

5. Add topic of kafka

Bin/kafka-topics.sh-create-zookeeper localhost:2181-replication-factor 3-partitions 1-topic TestTopic

6. Start flume agent

. / bin/flume-ng agent-n A1-c conf-f conf/mysql-flume.conf-Dflume.root.logger=INFO,console

7. Operate the database, add new table data, and view topic data