Shulou(Shulou.com)05/31 Report--

This article is to share with you what the core components of spark are, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

The core components of spark are as follows:

When the SparkContext is initialized, a series of contents are initialized:

View memory usage:

The Block tracker in the core components of the cluster is used to manage the correspondence between block and partition.

The shuffle tracker in the core components of the cluster is used to record the process details of the shuffle operation.

It can also be seen from the cluster that Executor executes tasks in a multithreaded manner and can read data on systems such as HDFS or HBase.

And when the actual Driver Program is running, each partition will be run by a task.

These are the core components of spark, and the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.