Shulou(Shulou.com)06/01 Report--

This article will explain in detail how spark writes hive data. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

Package hgs.spark.hiveimport org.apache.spark.SparkConfimport org.apache.spark.SparkContextimport org.apache.spark.sql.SparkSessionimport org.apache.spark.sql.SQLContextimport org.apache.spark.sql.SaveModeimport org.apache.spark.sql.types.StructFieldimport org.apache.spark.sql.types.StructTypeimport org.apache.spark.sql.types.IntegerTypeimport org.apache.spark.sql.types.StringTypeimport org.apache.spark.sql.Rowobject WriteDatatoHive {def main (args: Array [String]): Unit = {val Conf = new SparkConf () .setAppName ("WriteDatatoHive") .setMaster ("local") val context = new SparkContext (conf) val rdd = context.parallelize (List (("wd") 22), ("cm", 25), 2) .map (x = > Row (x.account1) x.account2) val builder = SparkSession.builder () .appName ("hiveApp") .config ("spark.sql.warehouse.dir") "hdfs://bigdata00:9000/user/hive/warehouse/") .enableHiveSupport () .getOrCreate () / / import builder.implicits._ import builder.implicits._ val personShcema = StructType (List (/ / below is a description of a column They are column name and data type Whether it is empty StructField ("name", StringType,true), StructField ("age", IntegerType,true) val personDF = builder.createDataFrame (rdd) PersonShcema) personDF.createOrReplaceTempView ("personm") / / this can store data compatible with hiveSQL builder.table ("personm"). Write.insertInto ("test.person") / / builder.sql ("select * from personm"). Write.option ("spark.sql.hive.convertMetastoreParquet", false) / / .mode (SaveMode.Append) .saveAsTable ("test.person") context.stop ()} case class person (name:String) Age:Int) this is the end of the article on "how spark writes hive data" Hope that the above content can be helpful to you, so that you can learn more knowledge, if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.