Alliance of HDFS (NameNode): Federation
The alliance of HDFS (NameNode): Federation, configure HDFS alliance, multiple NameNode nodes survive at the same time, handle different requests of clients according to the routing rules configured by agents, achieve load balancing, and cache more meta-information at the same time. It is different from HA in that multiple NameNode are active, which can store more meta-information.
I. the connection and difference between Alliance and HA
1. There are also multiple NameNode
2. The difference from HA: these NameNode are active status.
3. Function:
(1) caching more meta-information
(2) realize load balancing
Second, build HDFS alliance
(1) Planning
NameNode:bigdata112 bigdata113
DataNode:bigdata114 bigdata115
(2) configure and install hadoop on bigdata112
(2.1) decompression
(2.2) configure the java environment in hadoop-env.sh
Export JAVA_HOME=/training/jdk1.8.0_144
(2.3) configure core-site.xml
Hadoop.tmp.dir / training/hadoop-2.7.3/tmp
(2.4) configure mapred-site.xml
Mapreduce.framework.name yarn (2.5) configure yarn-site.xml
Yarn.resourcemanager.hostname
Bigdata12
Yarn.nodemanager.aux-services
Mapreduce_shuffle
(2.6) configure slaves
Bigdata14 bigdata15
(2.7) configure hdfs-site.xml
Dfs.nameservices ns1 Ns2 dfs.namenode.rpc-address.ns1 bigdata112:9000 dfs.namenode.http-address.ns1 bigdata112:50070 Dfs.namenode.secondaryhttp-address.ns1 bigdata112:50090 dfs.namenode.rpc-address.ns2 bigdata113:9000 Dfs.namenode.http-address.ns2 bigdata113:50070 dfs.namenode.secondaryhttp-address.ns2 bigdata113:50090 dfs.replication 2 dfs.webhdfs.enabled true Dfs.permissions false
(2.8) configure core-site.xml and add routing rules
Fs.viewfs.mounttable.xdl1.homedir / home fs.viewfs.mounttable.xdl1.link./movies hdfs://bigdata114:9000/movies fs.viewfs.mounttable.xdl1.link./mp3 hdfs://bigdata115:9000/mp3 fs.default.name viewfs://xdl1
Note: if there are too many routing rules, you can create a separate xml file (mounTable.xml)
(3) copy the hadoop from 113,114,115 to 113,114,115
(4) format NameNode
Formatting on 112and 113th namenode nodes
Hdfs namenode-format-clusterId xdl1
(5) start
Start Hadoop on 112
Start-all.sh
(6) create a directory for the response on the DataNode node
