Shulou(Shulou.com)06/03 Report--

This article mainly explains "the basic flow and characteristics of the operation of Spark". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the basic flow and characteristics of Spark".

1. Spark runs the basic process:

(1) build the running environment of Spark Application (start SparkContext), and SparkContext registers with the resource manager (which can be Standalone, Mesos or YARN) and applies for running Executor resources.

(2) the resource manager allocates Executor resources and starts the Executor,Executor operation to be sent to the resource manager with the heartbeat.

(3) SparkContext is constructed into DAG graph, DAG graph is decomposed into Stage, and Taskset is sent to Task Scheduler. Executor applies to SparkContext for Task,Task Scheduler to issue Task to Executor to run while SparkContext issues the application code to Executor.

(4) Task runs on Executor, and all resources are released after running.

2. The characteristics of Spark running architecture:

(1) each Application gets its own executor process, which resides during the Application and runs tasks in a multithreaded manner.

(2) the Spark task has nothing to do with the resource manager, as long as you can get the executor process and keep communicating with each other.

(3) the Client that submits the SparkContext should be close to the Worker node (the node running Executor), preferably in the same Rack, because there is a lot of information exchange between SparkContext and Executor when the Spark program is running; if you want to run in a remote cluster, it is best to use RPC to submit the SparkContext to the cluster, and do not run SparkContext far away from Worker.

(4) Task adopts the optimization mechanism of data locality and speculative execution.

Thank you for your reading, the above is the content of "the basic flow and characteristics of the operation of Spark". After the study of this article, I believe you have a deeper understanding of the basic process and characteristics of the operation of Spark, and the specific use needs to be verified by practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.