Shulou(Shulou.com)05/31 Report--

This article introduces what the Spark principle architecture diagram is like, the content is very detailed, interested friends can refer to, hope to be helpful to you.

Task is the basic unit that runs on a cluster. A Task is responsible for handling a partition of the RDD. Multiple patition of RDD will be handled by different Task. Of course, the processing logic of these Task is exactly the same. This group of Task forms a Stage. There are two Task types:

Org.apache.spark.scheduler.ShuffleMapTask

Org.apache.spark.scheduler.ResultTask

ShuffleMapTask puts the calculation results into different bucket according to Task's partitioner. ResultTask sends the results back to Driver Application. A Job contains multiple Stage, while a Stage is made up of exactly the same set of Task. The final Stage contains a set of ResultTask.

About how the Spark principle architecture diagram is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.