Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

How to configure Hadoop HA cluster files

Shulou Source: shulou.com Published: 2022-06-03 07:45:54 11月23日 Update

Today, the editor will bring you an article on configuring Hadoop HA cluster files. The editor thinks it's pretty good, so I'll share it for you as a reference. Let's follow the editor and have a look.

> 1. Hadoop-env.sh configuration

Export JAVA_HOME= (jdk installation address)

> 2. Hdfs-site.xml configuration file

Dfs.nameservices

Mycluster

Dfs.ha.namenodes.mycluster

Master,slave1

Dfs.namenode.rpc-address.mycluster.master

Master:50070

Dfs.namenode.rpc-address.mycluster.slave1

Slave1:50070

Dfs.namenode.http-address.mycluster.master

Master:9000

Dfs.namenode.http-address.mycluster.slave1

Slave1:9000

Dfs.namenode.shared.edits.dir

Qjournal://master:8485;slave1:8485;slave2:8485/mycluster

Dfs.client.failover.proxy.provider.mycluster

Org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

Dfs.ha.fencing.methods

Sshfence

Dfs.ha.fencing.ssh.private-key-files

/ root/.ssh/id_rsa

Dfs.ha.fencing.methods

Shell (/ bin/true)

Dfs.journalnode.edits.dir

/ usr/local/src/hadoop/data/journalnode/jn

Dfs.ha.automatic-failover.enabled

True

> 3. Core-site.xml configuration file

Fs.defaultFS

Hdfs://mycluster

Hadoop.tmp.dir

/ opt/tmp/hadoop/ha

Ha.zookeeper.quorum

Master:2181,slave1:2181,slave2:2181

> 4. Mapred-site.xml configuration

Mapreduce.farmework.name

Yarn

> 5. Yarn-site.xml configuration

Yarn.resourcemanager.ha.enabled

True

Yarn.resourcemanager.ha.automatic-failover.enabled

True

Yarn.resourcemanager.cluster-id

Yarn-rm-cluster

Yarn.resourcemanager.ha.rm-ids

Rm1,rm2

Yarn.resourcemanager.hostname.rm1

Master

Yarn.resourcemanager.hostname.rm2

Server1

Yarn.resourcemanager.zk-address

Master:2181,server1:2181,server2:2181

Yarn.resourcemanager.webapp.address.rm1

Master:8088

Yarn.resourcemanager.webapp.address.rm2

Slave1:8088

Yarn.nodemanager.aux-services

Mapreduce_shuffle

> 6. Slaves configuration (DataNode)

Master

Slave1

Slave2

The above is a summary of how to configure Hadoop HA cluster files, which is more comprehensive, and I also believe that there are quite a few tools that we may see or use in our daily work. Through this article, I hope you can gain more.

Tags: Configuration file cluster good for this content address that is tools articles daily work more read it article reference work Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno macOS OPPO Reno Linux Apple Shulou Information