Shulou(Shulou.com)06/03 Report--

Problems encountered in the first try of hadoop MapReduce

. / hadoop jar. / MapReduce-jar-with-dependencies.jar com.hc360.mp.WordCountHbaseReader / user/hbase/tabelname01 tablename

When executing MapReduce tasks on a hadoop cluster, you need to use a lot of dependent jar. Generally speaking, when you execute, you specify the configuration on the execution command, which will cause the execution command to be too long and prone to error. Sometimes you want to bring several parameter information, so it is more prone to error.

A better way to handle this is to package the required dependent jar into the running jar when generating and running the jar, so that you don't have to specify all the dependent jar at execution time.

. / hdfs dfs-rmr / user/hbase/

Existing files on hdfs cannot be overwritten. Files must be generated as new files that do not exist. You can execute the above command to clear existing files before generating files:

16-07-19 18:26:08 INFO ipc.Client: Retrying connect to server: xxxx/192.168.xx.xx:8032. Already tried 4 time (s); retry policy is etryUpToMaximumCountWithFixedSleep (maxRetries=10, sleepTime=1000 MILLISECONDS)

On the hadoop cluster, if yarn (MapReduce) is not started, the above information will be prompted when the MapReduce task is run

When you execute the shell command of hadoop, if you are prompted with information similar to the above, you can see if the hadoop process has been started.

Java.lang.IllegalArgumentException: Does not contain a valid host:port authority:

Java.net.UnknownHostException

For the hostname configuration of a hadoop cluster, the hostname does not contain an underscore "_", but it can have an underscore "-".

Each machine in the hadoop cluster has to configure the hostname domain name of all the machines in the cluster so that they can access each other.

Here is a bloody lesson to share. When naming hostname, it must be noted that although linux supports some special characters, it is not supported in java, so the naming of hostname must use numbers and letters (separated by periods). If the starting hostname contains a special character underscore, it causes an error in the reduce phase of the hadoop runtime.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.