HADOOP monitoring

Hadoop load balancing

The equalizer (balancer) is a Hadoop daemon that redistributes chunks from busy datanode to relatively idle datanode. At the same time, adhere to the block copy placement strategy, spread the copies to different racks, in order to reduce the data damage rate. It keeps moving blocks until the cluster reaches equilibrium, that is, the utilization of each datanode (the ratio of space used to space capacity on that node) and cluster utilization (the ratio of space used in the cluster to the space capacity of the cluster) are very close, and the gap does not exceed a given threshold.

In order to reduce the load of the cluster and avoid disturbing other users, the equalizer is designed to run in the background. The bandwidth of replicating data between different nodes is also limited. The default value is a very small 1MB/s, which can be reset by the dfs.balance.bandwidthPersec attribute in the hdfs-site.xml file (in bytes)

Hadoop Log Settings

You can view and set the log level through http://192.168.9.110:50070/logLevel. The screenshot is as follows:

3.Hadoop stack trace

You can view the stack information through http://192.168.9.110:50070/stacks. The screenshot is as follows: