[Hadoop] YARN pseudo-distributed deployment and MapReduce case
Foreword:
In previous articles, we introduced the configuration and startup of HDFS components. Yarn is the resource and job scheduling platform for Hadoop clusters. The following describes the pseudo-distributed deployment of Yarn and the simple use of MapReduce.
1. Configuration file and launch [hadoop@hadoop000 hadoop] $pwd/opt/software/hadoop-2.8.1/etc/hadoop [hadoop@hadoop000 hadoop] $cp mapred-site.xml.template mapred-site.xml [hadoop@hadoop000 hadoop] $vi mapred-site.xml mapreduce.framework.name yarn [hadoop@hadoop000 hadoop] $vi yarn-site.xml yarn.nodemanager.aux-services mapreduce_shuffle [hadoop@hadoop000 hadoop] $ Cd / opt/software/hadoop-2.8.1 [hadoop@hadoop000 hadoop-2.8.1] $sbin/start-yarn.shstarting yarn daemonsstarting resourcemanager Logging to / opt/software/hadoop-2.8.1/logs/yarn-hadoop-resourcemanager-hadoop000.out192.168.6.217: starting nodemanager Logging to / opt/software/hadoop-2.8.1/logs/yarn-hadoop-nodemanager-hadoop000.out [hadoop@hadoop000 hadoop-2.8.1] $jps10512 ResourceManager-- process of Yarn 10004 SecondaryNameNode9655 NameNode10936 Jps9806 DataNode10623 NodeManager-- process of Yarn # two processes of Yarn: ResourceManager (Resource Management) NodeManager (Node Management) # refer to the official website: http://hadoop.apache.org/docs/r2.8.4/hadoop-project-dist/hadoop-common/SingleCluster.html2.Yarn website Test
3.MapReduce Job test [hadoop@hadoop000 hadoop-2.8.1] $pwd/opt/software/hadoop-2.8.1 [hadoop@hadoop000 hadoop-2.8.1] $find. /-name "* example*". / lib/native/examples./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.8.1-test-sources.jar./share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.8.1-sources.jar./share/hadoop / mapreduce/lib-examples./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar./share/doc/hadoop/hadoop-mapreduce-examples./share/doc/hadoop/hadoop-yarn/hadoop-yarn-common/apidocs/org/apache/hadoop/yarn/webapp/example./share/doc/hadoop/hadoop-auth-examples./share/doc/hadoop/api/org/apache/hadoop/examples./share/doc/hadoop/api/org/apache/hadoop/security/authentication / examples./etc/hadoop/ssl-server.xml.example./etc/hadoop/ssl-client.xml.example [hadoop@hadoop000 hadoop-2.8.1] $bin/hadoop jar. / share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar pi 5104. Result interpretation
