Hadoop optimization and adjustment

[io.file.buffer.size] (core-site.xml)

Used to set the size of the cache, larger caches provide more efficient data transfer, but also mean greater memory consumption and latency

The default value is 4KB and is generally set to 64KB (65536)

[dfs.balance.bandwidthPerSec]

The HDFS balancer detects overused or underused DataNode in the cluster and shifts blocks between these DataNode to ensure load balancing. This parameter defines the maximum bandwidth allowed for each DataNode balancing operation in byte, and the network bandwidth is generally in bit.

[dfs.block.size]

The default value is 67108864, or 64MB, and the reference value is 134217728 (128m)

[dfs.DataNode.du.reserved]

Because mapred.local.dir often shares available hard disk resources with DataNode, we need to reserve some hard disk resources for MapReduce tasks. It is recommended that each hard disk reserve the minimum 10GB resources for map tasks, that is, 10737418240.

[dfs.NameNode.handler.count]

NameNode has a worker thread pool for handling remote procedure calls from clients and calls from cluster daemons. The default value is 10, which is generally set to the natural logarithm of the cluster size multiplied by 20, or 20logN.

The obvious symptom of setting this value too small is that DataNode always times out when connecting to NameNode or the connection is rejected.

[dfs.DataNode.failed.volumes.tolerated]

When any one of the local disks of the DataNode fails, the entire DataNode is determined to fail by default. The default value of this parameter is 0, which means that any failure of one disk will make the entire DataNode unavailable. Reference value is 1

[dfs.hosts]

Confirm the DataNode that is allowed to connect and join the cluster through a file with a list of DataNode hostnames

[dfs.host.exclude]

Excluding related nodes from HDFS, you can uninstall DataNode

[fs.trash.interval] (core-site.xml)

Defines the time (minutes) for files in the .Trash directory to be retained before they are permanently deleted. The default is 0, that is, the garbage Recycle Bin function is turned off, and the reference value is 1440 (24 hours).