Shulou(Shulou.com)06/03 Report--

This article focuses on "how to submit Spark applications to the cluster remotely". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to submit Spark applications to the cluster remotely.

Describes how to submit Spark tasks to the cluster outside the cluster.

Spark currently supports three types of clusters, namely, Spark self-implemented standalone cluster, mesos cluster and YARN cluster.

Using standalone clustering, use the-master parameter to specify the cluster address when submitting the application, such as-- master spark://masternode:7077

Therefore, only the client of spark is needed, which can be easily submitted to the specified cluster remotely.

Using mesos clusters, also specify the-master parameter as the cluster address, such as-- master mesos://master66:5050. This form is very similar to standalone clusters, and it is also very convenient to submit applications remotely.

With YARN clustering, the-master parameter is specified as YARN, but requires that the environment variable for HADOOP_CONF_DIR or YARN_CONF_DIR be set in advance, which points to the directory of the hadoop-related configuration. The spark client reads and parses the configuration file and uses the configuration information to connect to the cluster manager.

If you want to submit applications to the Yarn cluster remotely, the easiest way is to download these configurations from the cluster and then configure the environment variables.

For more information, please refer to:

Remote Spark Jobs on YARN

Spark submit YARN mode HADOOP_CONF_DIR contents

How to submit a spark job on a remote master node in yarn client mode?

I think maybe we can find out the configuration items that spark needs and specify them on the command line, instead of configuring the environment variables of the directory, so I can flexibly switch between different clusters, and I do need it.

This direction remains to be explored.

An example of submitting an application in the windows system:

1set HADOOP_CONF_DIR=E:LOCALCLUSTERSERVICE-HADOOP-a1e7b238d7f044d6b70185654c855c52 & at this point, I believe you have a better understanding of "how to remotely submit Spark applications to the cluster", so you might as well do it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.