Shulou(Shulou.com)05/31 Report--

How to analyze the shuffle module in spark? in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

1. Basic shuffle writer implementation parsing

When shuffleMapTask is executed on Executor, the runTask of shuffleMapTask will eventually be called. The core logic is as follows:

1. Get shuffleManager from sparkEnv

two。 Get writer from manager

3. Call RDD to start the calculation, and the operation result is persisted through writer. Before persistence, the Aggregator is used to determine whether it needs to be aggregated at the Map end, and then the result is written by the FileShuffleManager#forMapTask method. After the writing is completed, the metadata information is written to the MapStatus, and then the downstream Task can obtain the data to be processed through this MapStatus.

In this way, writer obtains the number of downstream partition through ShuffleDependency#partitioner, and each partition downstream corresponds to a file with the format of "shuffle_" + shuffledId+ "_" + mapId+ "_" + reduceId.

Since each shuffle Map Task needs to create a separate file for each downstream Task, the number of files is number (shuffle_map_task) * number (following_task). This causes many files to be created and opened.

Later, Spark introduced Shuffle Consolidate Writer, the principle is that the first Task on the core creates a file, and the shuffle operations of the later Task on the core are appended to the modified file, so that the number of files is number (core) * number (following_task).

This is the answer to the question about how to analyze the shuffle module in spark. I hope the above content can be of some help to you. If you still have a lot of doubts to solve, you can follow the industry information channel to learn more about it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.