Shulou(Shulou.com)06/03 Report--

Data acquisition is an important part of the whole process of big data analysis. Typical data acquisition tools include ETL tools, log collection tools, data migration tools and so on.

Flume is a highly available, highly reliable, distributed mass log collection, aggregation and transmission system.

1. Install Flume

Download: http://www.apache.org/dist/flume/

Hadoop@dblab:/usr/local$ sudo wget http://www.apache.org/dist/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz

Hadoop@dblab:/usr/local$ sudo tar-zxvf apache-flume-1.7.0-bin.tar.gz

Hadoop@dblab:/usr/local$ sudo mv apache-flume-1.7.0-bin. / flume

two。 Configure environment variables

Hadoop@dblab:/usr/local$ sudo vim / .bashrc

Export FLUME_HOME=/usr/local/flume

Export FLUME_CONF_DIR=$FLUME_HOME/conf

Export JAVA_HOME=/usr/lib/jvm/default-java

Export PATH=$PATH:$HIVE_HOME/bin

Hadoop@dblab:/usr/local$ source / .bashrc

Hadoop@dblab:/usr/local/flume/conf$ mv flume-env.sh.template flume-env.sh

Hadoop@dblab:/usr/local/flume/conf$ sudo vim flume-env.sh

# add the following statement at the beginning of the flume-env.sh file:

Export JAVA_HOME=/usr/lib/jvm/default-java

3. Start Flume

Hadoop@dblab:/usr/local/flume$ cd / usr/local/flume

Hadoop@dblab:/usr/local/flume$. / bin/flume-ng version

Error: main class org.apache.flume.tools.GetJavaProperty cannot be found or cannot be loaded

Flume 1.7.0

Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git

Revision: 511d868555dd4d16e6ce4fedc72c2d1454546707

Compiled by bessbd on Wed Oct 12 20:51:10 CEST 2016

From source with checksum 0d21b3ffdc55a07e1d08875872c00523

Hadoop@dblab:/usr/local/flume$ cd / usr/local/hbase/conf

Hadoop@dblab:/usr/local/hbase/conf$ sudo vim hbase-env.sh

# export HBASE_CLASSPATH=/usr/local/hadoop/conf # comment the line to solve the above problem

Hadoop@dblab:/usr/local/flume$. / bin/flume-ng version

Flume 1.7.0

Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git

Revision: 511d868555dd4d16e6ce4fedc72c2d1454546707

Compiled by bessbd on Wed Oct 12 20:51:10 CEST 2016

From source with checksum 0d21b3ffdc55a07e1d08875872c00523

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.