Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

Sqoop2 installation startup

Shulou Source: shulou.com Published: 2022-06-03 07:04:08 11月23日 Update

(1) Installation environment

Operating system: Linux (centos 6.5)

JDK version: 1.7.0_45

Hadoop version: hadoop 2.6.0

Sqoop2 Version: sqoop-1.99.4-bin-hadoop200

hadoop installation directory: /usr/local/hadoop/hadoop-2.2.0

Sqoop2 installation directory: /usr/local/sqoop/sqoop-1.99.4-bin-hadoop200

Hadoop and Sqoop are both the same user hadoop Below, hadoop user's home directory: /home/hadoop

(2) Modify Sqoop2 configuration file

1. First modify the configuration file/home/hadoop/sqoop-1.99.4-bin-hadoop200/server/conf/sqoop.properties Specify the path to which the hadoop configuration file belongs.

Replace the original configuration with the following:

# Hadoop configuration directory

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/

Amend to read:

# Hadoop configuration directory

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/hadoop-2.6.0/etc/hadoop/

2. Modify the configuration file/home/hadoop/sqoop-1.99.4-bin-hadoop200/server/conf/catalina.properties.

Here immediately add all *.jar packages under/home/hadoop/hadoop-2.2.0/share/hadoop to the classpath of sqoop2.

Replace the original configuration with the following:

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../ lib/*.jar,/usr/lib/hadoop/*.jar,/usr/lib/hadoop/lib/*.jar,/usr/lib/hadoop-hdfs/*.jar,/usr/lib/hadoop-hdfs/lib/*.jar,/usr/lib/hadoop-mapreduce/*.jar,/usr/lib/hadoop-mapreduce/lib/*.jar,/usr/lib/hadoop-yarn/*.jar,/usr/lib/hadoop-yarn/lib/*.jar

Amend to read:

common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../ lib/*.jar,/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/*.jar,/usr/local/hadoop/hadoop-2.6.0/share/hadoop/common/lib/*.jar,/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/*.jar,/usr/local/hadoop/hadoop-2.6.0/share/hadoop/hdfs/lib/*.jar,/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/*.jar,/usr/local/hadoop/hadoop-2.6.0/share/hadoop/mapreduce/lib/*.jar,/usr/local/hadoop/hadoop-2.6.0/share/hadoop/tools/*.jar,/usr/local/hadoop/hadoop-2.6.0/share/hadoop/tools/lib/*.jar,/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/*.jar,/usr/local/hadoop/hadoop-2.6.0/share/hadoop/yarn/lib/*.jar,/usr/local/hadoop/hadoop-2.6.0/share/hadoop/httpfs/tomcat/lib/*.jar

(3) Modify environmental variables

Modify ~/.bashrc directly and add the following at the end of the file:

export SQOOP_HOME=/usr/local/sqoop/sqoop-1.99.4-bin-hadoop200

export PATH=$SQOOP_HOME/bin:$PATH

export CATALINA_HOME=$SQOOP_HOME/server

export LOGDIR=$SQOOP_HOME/logs

Of course, after completing the above modifications, you need to execute the command under the hadoop user to make the configuration take effect:

Start Stop sqoop-server:

$SQOOP2_HOME/bin/sqoop.sh server start$SQOOP2_HOME/bin/sqoop.sh server stop or $SQOOP2_HOME/bin/sqoop2-server start$SQOOP2_HOME/bin/sqoop2-server stop

Verify successful startup:

Method 1: jps View Progress: Bootstrap

Method 2: http://hadoop000:12000/sqoop/version

wget -qO - hadoop000:12000/sqoop/version

Start SQOOP client:

$SQOOP2_HOME/bin/sqoop.sh client or $SQOOP2_HOME/bin/sqoop2-shell

Configure the server for clients:

sqoop:000> set server --host hadoop000 --port 12000 --webapp sqoop

View server-side information:

sqoop:000> show server --all

Tags: Configuration file method version user directory client client server environment path service success operating system information content variable command ownership end Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno NVidia Shulou Tech Info Linux Docker Xiaomi