Shulou(Shulou.com)06/01 Report--

This article mainly talks about "what is the method of integrating SPARK2 and Phoenix". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "what is the method of integrating SPARK2 and Phoenix"?

1. Environment description operating system CentOS Linux release 7.4.1708 (Core) Ambari2.6.xHDP2.6.3.0Spark2.xPhoenix4.10.0-HBase-1.22, conditions

HBase installation completed

Phoenix is enabled, and the Ambari interface is as follows:

Spark 2 installation completed

3. Integration of Spark2 and Phoenix

Steps:

Enter the Ambari Spark2 configuration interface

Locate the custom spark2-defaults and add the following configuration items:

Spark.driver.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-4.10.0-HBase-1.2-client.jar

Spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-4.10.0-HBase-1.2-client.jar

Mark4, Yarn HA problem

If Yarn HA is configured, you need to modify the Yarn HA configuration, otherwise the spark-submit submission task will report the following error:

Exception in thread "main" java.lang.IllegalAccessError: tried to access method org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal () Ljava/lang/Object; from class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider

At org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider.init (RequestHedgingRMFailoverProxyProvider.java:75)

At org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider (RMProxy.java:163)

At org.apache.hadoop.yarn.client.RMProxy.createRMProxy (RMProxy.java:94)

At org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy (ClientRMProxy.java:72)

At org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceStart (YarnClientImpl.java:187)

At org.apache.hadoop.service.AbstractService.start (AbstractService.java:193)

At org.apache.spark.deploy.yarn.Client.submitApplication (Client.scala:153)

At org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start (YarnClientSchedulerBackend.scala:56)

At org.apache.spark.scheduler.TaskSchedulerImpl.start (TaskSchedulerImpl.scala:173)

At org.apache.spark.SparkContext. (SparkContext.scala:509)

At org.apache.spark.SparkContext$.getOrCreate (SparkContext.scala:2516)

At org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply (SparkSession.scala:922)

At org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply (SparkSession.scala:914)

At scala.Option.getOrElse (Option.scala:121)

At org.apache.spark.sql.SparkSession$Builder.getOrCreate (SparkSession.scala:914)

At cn.spark.sxt.SparkOnPhoenix$.main (SparkOnPhoenix.scala:13)

At cn.spark.sxt.SparkOnPhoenix.main (SparkOnPhoenix.scala)

At sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethod)

At sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)

At sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

At java.lang.reflect.Method.i

Modify the Yarn HA configuration:

Change the original configuration:

Yarn.client.failover-proxy-provider=org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider

Change to the current configuration:

Yarn.client.failover-proxy-provider=org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider

If Yarn HA is not configured, you do not need to configure this step

At this point, I believe you have a deeper understanding of "what is the method of integrating SPARK2 and Phoenix". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.