How to write flume Multiplexing

This article mainly introduces "how to write flume multiplexing". In daily operation, I believe many people have doubts about how to write flume multiplexing. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts of "how to write flume multiplexing". Next, please follow the editor to study!

I. the purpose of the compilation

Through the understanding of flume components, in addition to using interceptors, you can also modify or customize the source code. This test uses changes to the source code. Use virtual to build flume + kafka environment, write simple multiplexing, mainly use the exec of flume to collect data sources and put them into logger, hdfs, kafka queues.

II. Description of the compilation environment

1. Jdk version: jdk1.7.0_80 Note: according to the requirements in pom.xml, flume-ng-1.6.0 needs to be compiled in jdk1.7, which may cause problems in jdk1.8.

2. Flume version: flume-ng-1.6.0-cdh6.7.0

3. Maven version: apache-maven-3.3.9

III. Modification of Flume source code

3. Compile flume-ng-1.6.0-cdh6.7.0 source code with maven

Download flume

Cd / opt/sourcecode

Wget http://archive.cloudera.com/cdh6/cdh/5/flume-ng-1.6.0-cdh6.7.0-src.tar.gz

Tar-zxvf flume-ng-1.6.0-cdh6.7.0-src.tar.gz

3.2. Modify the source code:

Cd / opt/sourcecode/flume-ng-1.6.0-cdh6.7.0/flume-ng-core/src/main/java/org/apache/flume/source

Add the following code to ExecSource.java:

Vi ExecSource.java

/ / create a map (Note: add 336 lines in the source code)

True

Always

Warn

Conjars

Conjars

Http://conjars.org/repo

Default

At this point, the study on "how to write flume multiplexing" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!