Installation and use of data acquisition tools
Data acquisition is an important part of the whole process of big data analysis. Typical data acquisition tools include ETL tools, log collection tools, data migration tools and so on.
Flume is a highly available, highly reliable, distributed mass log collection, aggregation and transmission system.
1. Install Flume
Download: http://www.apache.org/dist/flume/
Hadoop@dblab:/usr/local$ sudo wget http://www.apache.org/dist/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz
Hadoop@dblab:/usr/local$ sudo tar-zxvf apache-flume-1.7.0-bin.tar.gz
Hadoop@dblab:/usr/local$ sudo mv apache-flume-1.7.0-bin. / flume
two。 Configure environment variables
Hadoop@dblab:/usr/local$ sudo vim / .bashrc
Export FLUME_HOME=/usr/local/flume
Export FLUME_CONF_DIR=$FLUME_HOME/conf
Export JAVA_HOME=/usr/lib/jvm/default-java
Export PATH=$PATH:$HIVE_HOME/bin
Hadoop@dblab:/usr/local$ source / .bashrc
Hadoop@dblab:/usr/local/flume/conf$ mv flume-env.sh.template flume-env.sh
Hadoop@dblab:/usr/local/flume/conf$ sudo vim flume-env.sh
# add the following statement at the beginning of the flume-env.sh file:
Export JAVA_HOME=/usr/lib/jvm/default-java
3. Start Flume
Hadoop@dblab:/usr/local/flume$ cd / usr/local/flume
Hadoop@dblab:/usr/local/flume$. / bin/flume-ng version
Error: main class org.apache.flume.tools.GetJavaProperty cannot be found or cannot be loaded
Flume 1.7.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 511d868555dd4d16e6ce4fedc72c2d1454546707
Compiled by bessbd on Wed Oct 12 20:51:10 CEST 2016
From source with checksum 0d21b3ffdc55a07e1d08875872c00523
Hadoop@dblab:/usr/local/flume$ cd / usr/local/hbase/conf
Hadoop@dblab:/usr/local/hbase/conf$ sudo vim hbase-env.sh
# export HBASE_CLASSPATH=/usr/local/hadoop/conf # comment the line to solve the above problem
Hadoop@dblab:/usr/local/flume$. / bin/flume-ng version
Flume 1.7.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 511d868555dd4d16e6ce4fedc72c2d1454546707
Compiled by bessbd on Wed Oct 12 20:51:10 CEST 2016
From source with checksum 0d21b3ffdc55a07e1d08875872c00523
