How to connect Spark SQL to a remote Hive Metastore
This article mainly explains "how to connect Spark SQL to remote Hive Metastore". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's learn how to connect Spark SQL to a remote Hive Metastore!
For Spark 1.x, you can set with :
System.setProperty("hive.metastore.uris", "thrift://METASTORE:9083");final SparkConf conf = new SparkConf();SparkContext sc = new SparkContext(conf);HiveContext hiveContext = new HiveContext(sc);
Or
final SparkConf conf = new SparkConf();SparkContext sc = new SparkContext(conf);HiveContext hiveContext = new HiveContext(sc);hiveContext.setConf("hive.metastore.uris", "thrift://METASTORE:9083");
Update If your Hive is Kerberized :
Try setting these before creating the HiveContext :
System.setProperty("hive.metastore.sasl.enabled", "true");System.setProperty("hive.security.authorization.enabled", "false");System.setProperty("hive.metastore. keranthus.principal", hivePrincipal);System.setProperty("hive.metastore.execute.setugi", "true"); At this point, I believe that you have a deeper understanding of "how to connect Spark SQL to remote Hive Metastore", you may wish to actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!