Shulou(Shulou.com)06/01 Report--

This article shows you how the whole startup process of Spark application in Standalone mode in Spark is like, the content is concise and easy to understand, it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

First, the spark role in Standalone mode

As we all know, in Standalone mode, spark has the following five roles:

Sparksubmit,master,Worker,Driver,Executor . What role do these five roles play when we submit the application? let's explain it in detail below.

1,SparkSubmit

Load an entry for a Spark application. This class deals with setting the Classpath of spark-related dependencies (which shows its importance), while shielding the details of different cluster managers and different deployment models to provide a unified interface for users.

2,Master

Main functions:

A), accept the registration of worker and manage Worker

B), accept the Application submitted by SparkSubmit, and schedule to start Driver (send LaunchDriver to Worker).

C), accept the RegisterApplication (appDescription,self) sent by the AppClient on the driver side, and then schedule and send the LaunchExecutor message to the Worker.

3,Worker

A), register with Master and maintain your heartbeat.

B), accept the LaunchDriver message from Master, and start Driver.

C), accept the LaunchExecutor message from Master, and start Executor

4,Driver

A), register App with Master through AppClient, and carry CoraseGrainedExecutorBackend information at this time.

B), accept Executor registration and maintain Executor information.

C), partition Stage, encapsulate task, and schedule task.

5,Executor

A), register yourself with Driver and maintain your heartbeat

B), accept the LaunchTask information of Driver and execute task.

C) to return the task execution result to Driver.

Second, Standalone submits an application source code process

1. The process of starting Driver

Org.apache.spark.launcher.Main

Org.apache.spark.deploy.SparkSubmit

Org.apache.spark.deploy.rest.RestSubmissionClient

Org.apache.spark.deploy.rest.StandaloneRestServer

Org.apache.spark.deploy.master.Master

Org.apache.spark.deploy.worker.Worker

Org.apache.spark.deploy.worker.DriverRunner

Org.apache.spark.deploy.worker.DriverWrapper

User customized program ~ ~

2. The process of starting Executor

Org.apache.spark.SparkContext

Org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend

Org.apache.spark.deploy.client.AppClient

Org.apache.spark.deploy.master.Master

Org.apache.spark.deploy.worker.Worker

Org.apache.spark.deploy.worker.ExecutorRunner

Org.apache.spark.executor.CoarseGrainedExecutorBackend

Org.apache.spark.executor.Executor

3. The process of executing task

Org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend

Org.apache.spark.executor.CoarseGrainedExecutorBackend

Org.apache.spark.executor.Executor

Org.apache.spark.executor.TaskRunner

Third, explain it with the specific flow chart.

The above is what the whole startup process of Spark application in Standalone mode in Spark is like. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.