Configuration and Optimization method of hadoop2.6

This article mainly introduces "the configuration and optimization methods of hadoop2.6". In the daily operation, I believe many people have doubts about the configuration and optimization methods of hadoop2.6. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "configuration and optimization methods of hadoop2.6". Next, please follow the editor to study!

1.vi / opt/hadoop-2.6.0/etc/hadoop/hadoop-env.sh

Export JAVA_HOME=/opt/jdk1.7.0_75

2.vi / opt/hadoop-2.6.0/etc/hadoop/core-site.xml

Fs.default.name

Hdfs://spore:9000

Note: spore is the hostname of the machine.

Hadoop.native.lib

True

Dfs.permissions

False

Hadoop.tmp.dir

/ opt/hadoop-2.6.0/tmp

3.vi / opt/hadoop-2.6.0/etc/hadoop/hdfs-site.xml

Dfs.replication

one

4.vi / opt/hadoop-2.6.0/etc/hadoop/mapred-site.xml

Mapreduce.framework.name

Yarn

5.vi / opt/hadoop-2.6.0/etc/hadoop/slaves

Add hostname for slave

Optimization:

Try to use combiner to reduce the number of key-value pairs, merge key-value pairs locally, reduce network transmission, and the optimization effect is obvious.

Increase the memory of the mapreduce intermediate result cache

Skillfully use compound keys to allow the system to complete sorting, so it is not necessary to sort by yourself.

At this point, the study of "hadoop2.6 configuration and optimization method" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!