Get the App
SLTechnology News&Howtos  ›  Development  › 

How to connect Spark SQL to a remote Hive Metastore

Shulou Source: shulou.com Published: 2022-06-03 05:41:56 11月18日 Update

This article mainly explains "how to connect Spark SQL to remote Hive Metastore". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's learn how to connect Spark SQL to a remote Hive Metastore!

For Spark 1.x, you can set with :

System.setProperty("hive.metastore.uris", "thrift://METASTORE:9083");final SparkConf conf = new SparkConf();SparkContext sc = new SparkContext(conf);HiveContext hiveContext = new HiveContext(sc);

Or

final SparkConf conf = new SparkConf();SparkContext sc = new SparkContext(conf);HiveContext hiveContext = new HiveContext(sc);hiveContext.setConf("hive.metastore.uris", "thrift://METASTORE:9083");

Update If your Hive is Kerberized :

Try setting these before creating the HiveContext :

System.setProperty("hive.metastore.sasl.enabled", "true");System.setProperty("hive.security.authorization.enabled", "false");System.setProperty("hive.metastore. keranthus.principal", hivePrincipal);System.setProperty("hive.metastore.execute.setugi", "true"); At this point, I believe that you have a deeper understanding of "how to connect Spark SQL to remote Hive Metastore", you may wish to actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Tags: Content learning practical deeper interest practicality simple operation methods more friends websites channels inquiries Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno NVidia Redmi Docker Shulou Technology vpn