Hadoop cluster decommission node (offline node, super practical)

I. description

In order to save costs and avoid waste of resources, a node in the offline cluster, that is, recycles a CVM.

Centos 6.6_64bit

Hadoop 2.6.0

Second, the operation procedure (dynamic offline)

The hostname of the offline node is as follows. It operates under the hadoop user, and the configuration files are all in the conf directory.

Host-10-10-10-10 # # looks like it's on the cloud, isn't it?

1. Create a file in the conf directory

Touch excludes

Echo "host-10-10-10-10" > exclude

Less exclude # # requires authentication

two。 Modify the configuration file hdfs-site.conf

Vi hdfs-site.xml

Add the following content, the path according to its own actual situation

Dfs.hosts.exclude

/ usr/local/RoilandGroup/hadoop-2.6.0/etc/hadoop/excludes

3. Modify the configuration file yarn-site.conf

Add the following content, the path according to its own actual situation

Yarn.resourcemanager.nodes.exclude-path

/ usr/local/RoilandGroup/hadoop-2.6.0/etc/hadoop/excludes

4. Refresh the hdfs node (namenode active operation)

Hdfs dfsadmin-refreshNodes

Hdfs dfsadmin-report # # observe whether the node is decommission

5. Refresh the nodemanager node (resourcemanager active operation)

Yarn rmadmin-refreshNodes

6. Modify the slave file

Comment out the hostname

# host-10-10-10-10

7. Synchronize exclude files and slave files

Standby node from scp exclude to namenode/resourcemanager

8. Verify again, make sure it is the result we want, and inform our OPS colleagues that the CVM can be recycled.

Matters needing attention

1. Testing must be done before operating the production environment.

two。 Check the official documents and know how much impact your modified files have on the system.