Org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found
The exception information is as follows:
At org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain (SparkSubmit.scala:569)
At org.apache.spark.deploy.SparkSubmit$.doRunMain$1 (SparkSubmit.scala:166)
At org.apache.spark.deploy.SparkSubmit$.submit (SparkSubmit.scala:189)
At org.apache.spark.deploy.SparkSubmit$.main (SparkSubmit.scala:110)
At org.apache.spark.deploy.SparkSubmit.main (SparkSubmit.scala)
Java.lang.RuntimeException: MetaException (message:java.lang.ClassNotFoundException Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found)
At org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore (Table.java:290)
At org.apache.hadoop.hive.ql.metadata.Table.getDeserializer (Table.java:281)
At org.apache.hadoop.hive.ql.metadata.Table.getCols (Table.java:631)
At org.apache.hadoop.hive.ql.metadata.Table.checkValidity (Table.java:189)
At org.apache.hadoop.hive.ql.metadata.Hive.getTable (Hive.java:1017)
At org.apache.hadoop.hive.ql.metadata.Hive.getTable (Hive.java:950)
At org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation (HiveMetastoreCatalog.scala:201)
At org.apache.spark.sql.hive.HiveContext$$anon$2.org$apache$spark$sql$catalyst$analysis$OverrideCatalog$$super$lookupRelation (HiveContext.scala:262)
At org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply (Catalog.scala:161)
At org.apache.spark.sql.catalyst.analysis.OverrideCatalog$$anonfun$lookupRelation$3.apply (Catalog.scala:161)
At scala.Option.getOrElse (Option.scala:120)
At org.apache.spark.sql.catalyst.analysis.OverrideCatalog$class.lookupRelation (Catalog.scala:161)
At org.apache.spark.sql.hive.HiveContext$$anon$2.lookupRelation (HiveContext.scala:262)
At org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable (Analyzer.scala:174)
At org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$6.applyOrElse (Analyzer.scala:186)
At org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$6.applyOrElse (Analyzer.scala:181)
At org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply (TreeNode.scala:188)
At org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$3.apply (TreeNode.scala:188)
Background:
CREATE TABLE apachelog (
Host STRING
Identity STRING
User STRING
Time STRING
Request STRING
Status STRING
Size STRING
Referer STRING
Agent STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
"input.regex" = "([^] *) (- |\ [^\] *\]) ([^\"] *) (\ "[^\"] *\ ") (- | [0-9] *) (- | [0-9] *) (?: ([^\"] * |\ ".*\") ([^\ "] * |\" .*\ ")?"
)
STORED AS TEXTFILE
Because the use of 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe', instead of' org.apache.hadoop.hive.serde2.RegexSerDe', when creating the table causes the above exception to be reported all the time when using spark-sql or spark-shell access, the corresponding class is always not found, and importing the relevant Jar package still cannot be solved.
Solution:
Import when you want to start spark-shell,spark-sql-- jar xxxxx.jar imports the corresponding jar package. (note: under normal circumstances, everyone should think of importing the corresponding jar. But the problem I encountered is that if the path of the jar is a soft connection path, the above exception will still be reported, and the corresponding class cannot be found, so the actual path of the jar package must be imported. It may be because spark has bug in the processing of soft paths, which is not sure. )
