Building a fully distributed environment based on hadoop0.20.2
The three servers are configured with IP as follows:
192.168.11.131
192.168.11.132
192.168.11.133
Configure hostnames separately
Master:
# hostnamectl set-hostname master
The other two are configured as slave1 and slave2 respectively.
Each server shuts down the selinux and firewall:
# vi / etc/sysconfig/selinux
SELINUX=enforcing-- > SELINUX=disabled
# setenforce 0
# systemctl stop firewalld
# systemctl disable firewalld
Replace the Yum source:
[root@master ~] # mkdir apps
Upload package
Wget-1.14-15.el7.x86_64.rpm
[root@master apps] # rpm-ivh wget-1.14-15.el7.x86_64.rpm
[root@master apps] # cd / etc/yum.repos.d/
[root@master yum.repos.d] # wget http://mirrors.aliyun.com/repo/Centos-7.repo
[root@master yum.repos.d] # mv Centos-7.repo CentOS-Base.repo
[root@master yum.repos.d] # scp CentOS-Base.repo root@192.168.11.132:/etc/yum.repos.d/
[root@master yum.repos.d] # scp CentOS-Base.repo root@192.168.11.133:/etc/yum.repos.d/
Each server executes
# yum clean all
# yum makecache
# yum update
Ntp time synchronization:
As the ntp server, master is configured as follows
# yum install-y ntp
Ntpserver:
Modification time of master as ntp master server
# date-s "2018-05-27 23:03:30"
# vi / etc/ntp.conf
Add two lines under the comment
# restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
Server 127.127.1.0
Fudge 127.127.1.0 stratum 11
Below the comment
# server 0.centos.pool.ntp.org iburst
# server 1.centos.pool.ntp.org iburst
# server 2.centos.pool.ntp.org iburst
# server 3.centos.pool.ntp.org iburst
# systemctl start ntpd.service
# systemctl enable ntpd.service
Slave1 and slave2 as ntp clients are configured as follows
# vi / etc/ntp.conf
Add two lines under the same comment
# restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
Server 192.168.11.131
Fudge 127.127.1.0 stratum 11
Add comments on four lines
# server 0.centos.pool.ntp.org iburst
# server 1.centos.pool.ntp.org iburst
# server 2.centos.pool.ntp.org iburst
# server 3.centos.pool.ntp.org iburst
# systemctl start ntpd.service
# systemctl enable ntpd.service
Synchronization time error
# ntpdate 192.168.11.131
25 Jun 07:39:15 ntpdate [25429]: the NTP socket is in use, exiting
Resolve:
# lsof-iPUR 123
-bash: lsof: command not found
# yum install-y lsof
# lsof-iPUR 123
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
Ntpd 1819 ntp 16u IPv4 33404 0t0 UDP *: ntp
Ntpd 1819 ntp 17u IPv6 33405 0t0 UDP *: ntp
Ntpd 1819 ntp 18u IPv4 33410 0t0 UDP localhost:ntp
Ntpd 1819 ntp 19u IPv4 33411 0t0 UDP slave1:ntp
Ntpd 1819 ntp 20u IPv6 33412 0t0 UDP localhost:ntp
Ntpd 1819 ntp 21u IPv6 33413 0t0 UDP slave1:ntp
# kill-9 1819
Update time again
# ntpdate 192.168.11.131
24 Jun 23:37:27 ntpdate [1848]: step time server 192.168.11.131 offset-28828.363808 sec
# date
Sun Jun 24 23:37:32 CST 2018
Useradd:
# groupadd hduser
# useradd-g hduser hduser
# passwd hduser
Ssh password-free authentication:
All nodes generate authorized_keys:
# su hduser
$cd
$ssh-keygen-t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/ home/hduser/.ssh/id_rsa):
Created directory'/ home/hduser/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in / home/hduser/.ssh/id_rsa.
Your public key has been saved in / home/hduser/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:KfyLZTsN3U89CbFAoOsrkI9YRz3rdKR4vr/75R1A7eE hduser@master
The key's randomart image is:
+-[RSA 2048]-+
| .o. | |
|. . .. | |
|. .. oo |
| | o.o.oo. | |
| o + .s.. .. E. |
| | + o.B.. Oo. |
| | o =. = o +.. |
|. . O * oo. O o. |
| | oo==+. | . . | |
+-[SHA256]-+
$cd .ssh /
$cp id_rsa.pub authorized_keys
All nodes authenticate each other:
Master:
[hduser@master .ssh] $ssh-copy-id-I id_rsa.pub hduser@slave1
[hduser@master .ssh] $ssh-copy-id-I id_rsa.pub hduser@slave2
Verify:
[hduser@master .ssh] $ssh slave1
Last failed login: Wed Jun 27 04:55:44 CST 2018 from 192.168.11.131 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Wed Jun 27 04:50:05 2018
[hduser@slave1 ~] $exit
Logout
Connection to slave1 closed.
[hduser@master .ssh] $ssh slave2
Last login: Wed Jun 27 04:51:53 2018
[hduser@slave2 ~] $
Slave1:
[hduser@slave1 .ssh] $ssh-copy-id-I id_rsa.pub hduser@master
[hduser@slave1 .ssh] $ssh-copy-id-I id_rsa.pub hduser@slave2
Slave2:
[hduser@slave2 .ssh] $ssh-copy-id-I id_rsa.pub hduser@master
[hduser@slave2 .ssh] $ssh-copy-id-I id_rsa.pub hduser@slave1
Upload package:
[hduser@master ~] $cd src
[hduser@master src] $ll
Total 356128
-rw-r--r-- 1 root root 44575568 Jun 16 17:24 hadoop-0.20.2.tar.gz
-rw-r--r-- 1 root root 288430080 Mar 16 2016 jdk1.7.0_79.tar
Configure jdk:
[hduser@master src] $tar-xf jdk1.7.0_79.tar-C..
[hduser@master src] $cd..
[hduser@master ~] $vi .bashrc
Add
Export JAVA_HOME=/home/hadoop/jdk1.7.0_79
Export JRE_HOME=$JAVA_HOME/jre
Export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib/rt.jar
Export PATH=$PATH:$JAVA_HOME/bin
[hduser@master ~] $source .bashrc
[hduser@master ~] $java-version
Java version "1.7.079"
Java (TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot (TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
The configuration of the other two nodes is the same as above
Configure hadoop:
Decompress each node
Tar-zxf hadoop-0.20.2.tar.gz-C..
Master:
[hduser@master conf] $pwd
/ home/hduser/hadoop-0.20.2/conf
[hduser@master conf] $vi hadoop-env.sh
Export JAVA_HOME=/home/hduser/jdk1.7.0_79
[hduser@master conf] $vi core-site.xml
Fs.default.name
Hdfs://master:9000
[hduser@master conf] $vi hdfs-site.xml
Dfs.replication
two
[hduser@master conf] $vi mapred-site.xml
Mapred.job.tracker
Master:9001
[hduser@master conf] $vi masters
# localhost
Master
[hduser@master conf] $vi slaves
# localhost
Slave1
Slave2
Copy the configuration file to the other two nodes
[hduser@master conf] $scp hadoop-env.sh slave1:~/hadoop-0.20.2/conf/
[hduser@master conf] $scp core-site.xml slave1:~/hadoop-0.20.2/conf/
[hduser@master conf] $scp hdfs-site.xml slave1:~/hadoop-0.20.2/conf/
[hduser@master conf] $scp mapred-site.xml slave1:~/hadoop-0.20.2/conf/
[hduser@master conf] $scp masters slave1:~/hadoop-0.20.2/conf/
[hduser@master conf] $scp slaves slave1:~/hadoop-0.20.2/conf/
[hduser@master conf] $scp hadoop-env.sh slave2:~/hadoop-0.20.2/conf/
[hduser@master conf] $scp core-site.xml slave2:~/hadoop-0.20.2/conf/
[hduser@master conf] $scp hdfs-site.xml slave2:~/hadoop-0.20.2/conf/
[hduser@master conf] $scp mapred-site.xml slave2:~/hadoop-0.20.2/conf/
[hduser@master conf] $scp masters slave2:~/hadoop-0.20.2/conf/
[hduser@master conf] $scp slaves slave2:~/hadoop-0.20.2/conf/
Format the file system
[hduser@master conf] $cd.. / bin
[hduser@master bin] $. / hadoop namenode-format
Start the service
[hduser@master bin] $. / start-all.sh
[hduser@master bin] $jps
1681 JobTracker
1780 Jps
1618 SecondaryNameNode
1480 NameNode
[hduser@slave1 conf] $jps
1544 Jps
1403 DataNode
1483 TaskTracker
[hduser@slave2 conf] $jps
1494 TaskTracker
1414 DataNode
1555 Jps
