Hadoop ha cluster building (5 sets)

Yum install ntpdate lrzsz-y

Systemctl stop firewalld

Systemctl disable firewalld

Systemctl stop NetworkManager

Systemctl disable NetworkManager

Setenforce 0

Mkdir / home/myapps & & cd / home/myapps

Hosts file configuration:

Cat > > / etc/hosts / home/myapps/zookeeper-3.4.11/data/myid

[root@node2 zookeeper-3.4.11] # echo "2" > / home/myapps/zookeeper-3.4.11/data/myid

[root@node3 zookeeper-3.4.11] # echo "3" > / home/myapps/zookeeper-3.4.11/data/myid

Start the cluster:

[root@node1 zookeeper-3.4.11] # / home/myapps/zookeeper-3.4.11/bin/zkServer.sh start

[root@node2 zookeeper-3.4.11] # / home/myapps/zookeeper-3.4.11/bin/zkServer.sh start

[root@node3 zookeeper-3.4.11] # / home/myapps/zookeeper-3.4.11/bin/zkServer.sh start

Build a hadoop cluster:

Cd / home/myapps/hadoop/hadoop-2.7.5/etc/hadoop

Vim hadoop-env.sh

25 export JAVA_HOME=/home/myapp/java

Vim yarn-env.sh

23 export JAVA_HOME=/home/myapp/java

Vim hdfs-site.xml

Dfs.nameservices mycluster dfs.ha.namenodes.mycluster nn1,nn2 dfs.namenode.rpc-address.mycluster.nn1 node1:8020 dfs.namenode.rpc-address.mycluster.nn2 node2:8020 dfs.namenode.http-address.mycluster.nn1 node1:50070 dfs.namenode.http-address.mycluster.nn2 node2:50070 dfs.namenode.shared.edits.dir qjournal://node3:8485;node4:8485 Node5:8485/mycluster dfs.client.failover.proxy.provider.mycluster org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.ha.fencing.methods sshfence dfs.ha.fencing.ssh.private-key-files / root/.ssh/id_dsa dfs.journalnode.edits.dir / home/myapps/hadoop/node/local/data dfs.ha.automatic-failover.enabled true

Vim core-site.xml

Fs.defaultFS hdfs://mycluster hadoop.tmp.dir / home/myapps/hadoop/data/hadoop/temp Abase for other temporarydirectories. Ha.zookeeper.quorum node1:2181,node2:2181,node3:2181

Vim yarn-site.xml

Yarn.resourcemanager.hostname node1 yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce_shuffle.class org.apache.hadoop.mapred.ShuffleHandler

Vim mapred-site.xml

Mapreduce.framework.name yarn Execution framework set to Hadoop YARN.

Vim slaves

Node3

Node4

Node5

Copy the modified hadoop to each node:

For ip in 23 45; do scp-r / home/myapps/hadoop root@node$ {ip}: / home/myapps/; done

Start zookeeper first:

Authentication: jps View process QuorumPeerMain

Hadoop cluster startup:

1. Perform verification in node3,node4,node5: cd / home/myapps/hadoop/hadoop-2.7.5/sbin/. / hadoop-daemon.sh start journalnode: jps to view the result, if JournalNode appears Indicates initial success 2. Perform formatting on one of the namenode: cd / home/myapps/hadoop/hadoop-2.7.5/bin. / hdfs namenode-format copy the dir generated after successful execution to another namenode: execute node1 after the success of scp-r / home/myapps/hadoop/data root@node2:/home/myapps/hadoop/, execute hdfs zkfc-formatZK3, Start cd / home/myapps/hadoop/hadoop-2.7.5/sbin/. / start-all.sh jps to view the verification results of the process. You can count browser visits: http://node1:50070 NameNode 'node1:9000' (active) http://node2:50070 NameNode' node2:9000' (standby)

Zookeeper startup script:

#! / bin/bash# chkconfig: 2345 20 8 hours description: zk Cluster start and stop case $1 in "start") cat / root/myapps/onekey/zk/slave | while read line do {echo $line ssh $line "source / etc/profile;nohup zkServer.sh start > / dev/null 2 > & 1 &"} & wait done "stop") cat / root/myapps/onekey/zk/slave | while read line do {echo $line ssh $line "source / etc/profile;jps | grep QuorumPeerMain | cut-c 1-5 | xargs kill-s 9"} & wait done;; "status") jps;;*) echo "usage: / etc/init.d/zkd {start | stop | status}";; esac

Hadoop startup script

#! / bin/bash# chkconfig: 2345 20 8 in description: hadoop Cluster start and stop case $1 in "start") / root/myapps/hadoop/hadoop-2.7.4/sbin/start-all.sh;; "stop") / root/myapps/hadoop/hadoop-2.7.4/sbin/stop-all.sh;; "status") jps;;*) echo "usage: / etc/init.d/hadoopd {start | stop | status}";; esac