Hadoop3 cluster mode
Hadoop3 cluster mode includes independent mode, pseudo-distributed mode and fully distributed mode.
1 independent mode
Installation
A) download jdk-8u65-linux-x64.tar.gz
B) tar on
$> su centos; cd ~
$> mkdir downloads
$> cp / mnt/hdfs/downloads/bigdata/jdk-8u65-linux-x64.tar.gz ~ / downlooads
$> tar-xzvf jdk-8u65-linux-x64.tar.gz
C) create / soft folder
$> sudo mkdir / soft
$> sudo chown centos:centos / soft
D) move the files opened by tar to / soft
$> mv ~ / downloads/jdk-1.8.0_65 / soft/
E) create symbolic links
$> ln-s / soft/jdk-1.8.0_65 / soft/jdk
F) verify that the jdk installation was successful
$> cd / soft/jdk/bin
$>. / java-version
Configure jdk environment variables
1. Edit / etc/profile
$> sudo nano / etc/profile
...
Export JAVA_HOME=/soft/jdk
Exprot PATH=$PATH:$JAVA_HOME/bin
two。 Make the environment variable effective immediately
$> source / etc/profile
3. Go to any directory and test whether it is ok
$> cd ~
$> java-version
Pay attention
The folder you create must be the same as the users you log in.
Install HADOOP softwar
1. Install hadoop
A) download hadoop-2.7.3.tar.gz
B) tar on
$> su centos; cd ~
$> cp / mnt/hdfs/downloads/bigdata/hadoop-2.7.3.tar.gz ~ / downloads
$> tar-xzvf hadoop-2.7.3.tar.gz
C) none
D) move the files opened by tar to / soft
$> mv ~ / downloads/hadoop-2.7.3 / soft/
E) create symbolic links
$> ln-s / soft/hadoop-2.7.3 / soft/hadoop
F) verify that the hadoop installation was successful
$> cd / soft/hadoop/bin
$>. / hadoop version
two。 Configure hadoop environment variables
$> sudo nano / etc/profile
...
Export JAVA_HOME=/soft/jdk
Exprot PATH=$PATH:$JAVA_HOME/bin
Export HADOOP_HOME=/soft/hadoop
Export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
3. Take effect
$> source / etc/profile
4. Go to any directory and test whether it is ok
$> cd ~
$> hadoop version
5.nothing!
There is no need to enable a separate hadoop process.
6.hafs dfs-ls / home has the same effect as ls in linux
Pseudo distribution pattern (Pseudodistributed mode)
A) enter the ${HADOOP_HOME} / etc/hadoop directory
Three modes coexist
1. Create three configuration directories, which are equivalent to the hadoop directory
${hadoop_home} / etc/local
${hadoop_home} / etc/pesudo
${hadoop_home} / etc/full
two。 Create a symbolic connection
$> ln-spesudo hadoop
B) Edit core-site.xml
Fs.defaultFS
Hdfs://localhost/
C) Edit hdfs-site.xml
Dfs.replication
one
D) Edit mapred-site.xml
Note: cp mapred-site.xml.template mapred-site.xml
Mapreduce.framework.name
Yarn
E) Edit yarn-site.xml
Yarn.resourcemanager.hostname
Localhost
Yarn.nodemanager.aux-services
Mapreduce_shuffle
Configuration: core-site.xml hdfs-site.xmls yarn-site.xml mapred-site.xml
F) configure SSH
