Shulou(Shulou.com)05/31 Report--

This article will explain in detail what holes there are in the process of using Hive. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

1. The following exception is reported when using the INSERT OVERWRITE DIRECTORY statement

Caused by: java.io.IOException: Cannot get DistCp constructor: org.apache.hadoop.tools.DistCp. () at org.apache.hadoop.hive.shims.Hadoop23Shims.runDistCp (Hadoop23Shims.java:1160) at org.apache.hadoop.hive.common.FileUtils.copy (FileUtils.java:553) at org.apache.hadoop.hive.ql.metadata.Hive.moveFile (Hive.java:2622)... 21 more

Environment: hive-1.2.1 hadoop-2.7.2

Cause of error:

The no-parameter constructor of org.apache.hadoop.tools.DistCp in the hadoop-2.7.2 source code has removed public.

/ * To be used with the ToolRunner. Not for public consumption. * / @ VisibleForTesting DistCp () {}

When the reflection mechanism is used to initialize org.apache.hadoop.tools.DistCp in hive-1.2.1, it is the no-parameter construction method that is called.

Override public boolean runDistCp (Path src, Path dst, Configuration conf) throws IOException {int rc; / / Creates the command-line parameters for distcp String [] params = {"- update", "- skipcrccheck", src.toString (), dst.toString ()}; try {Class clazzDistCp = Class.forName ("org.apache.hadoop.tools.DistCp"); Constructor c = clazzDistCp.getConstructor (); c.setAccessible (true); Tool distcp = (Tool) c.newInstance () Distcp.setConf (conf); rc = distcp.run (params);} catch (ClassNotFoundException e) {throw new IOException ("Cannot find DistCp class package:" + e.getMessage ());} catch (NoSuchMethodException e) {throw new IOException ("Cannot get DistCp constructor:" + e.getMessage ());} catch (Exception e) {throw new IOException ("Cannot execute DistCp process:"+ e, e);} return (0 = rc) }

Solution: use the old version of hadoop-distcp-x.x.x.jar, and I use hadoop-distcp-2.6.2.jar here.

@ VisibleForTesting public DistCp () {} this is the end of the article on "what's wrong with the use of Hive". I hope the above content can be helpful to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.