Get the App
SLTechnology News&Howtos  ›  Development  › 

How to configure spark sql query hive

Shulou Source: shulou.com Published: 2022-06-03 05:41:58 11月18日 Update

This article introduces the relevant knowledge of "how to configure spark sql query hive". In the actual case operation process, many people will encounter such difficulties. Next, let Xiaobian lead you to learn how to deal with these situations! I hope you can read carefully and learn something!

Configure idea spark sql connection hive

1. First copy hive-site.xml of/etc/hive/conf/to resources of the project

2. To add maven dependencies to the pom.xml file, you need to add spark-sql, spark-hive, spark version 2.2.0, scala version 2.10

org.apache.spark

spark-sql_2.10

2.2.0

org.apache.spark

spark-hive_2.10

2.2.0

3. Set it in the program

val spark = SparkSession

.builder()

.appName("test_note")

.master("local")

.enableHiveSupport()//Set support hive

.getOrCreate()

To this place, you can directly query the hive data of the server

Configure CDH Cluster

Cluster configuration only needs to copy hive-site.xml from/etc/hive/conf/directory to/etc/spark2/conf/directory to spark-submit.

"How to configure Spark SQL query hive" content is introduced here, thank you for reading. If you want to know more about industry-related knowledge, you can pay attention to the website. Xiaobian will output more high-quality practical articles for everyone!

Tags: Configuration query content more version directory knowledge cluster utility learning next dilemma reality situation data files articles servers cases programs Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno Microsoft MariaDB Linux Shulou Technology vpn