Shulou(Shulou.com)05/31 Report--

This article mainly explains "the coexistence of spark and hadoop". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the coexistence of spark and hadoop".

Q1: in order to speed up the execution of spark shuffle, whether it is possible to point spark_local_dirs to a solid state disk, and whether this is effective.

You can point the spark_local_dirs to a solid state disk, which will effectively improve the speed of Spark execution.

At the same time, if you want to improve the running speed of Spark faster, you can specify multiple Shuffle output directories and let Shuffle read and write disks in parallel.

Q2:solidation=true is just merging on the same machine, right?

Solidation=true is merged on the same machine.

When merging, bucket belonging to the same Reducer will be put into the same file, which greatly reduces the number of Shuffler files and improves performance.

Q3: will spark and hadoop coexist in the future?

Spark and hadoop will coexist, Spark+Hadoop= A winning combination

When coexisting, Hadoop mainly uses HDFS for data storage, and Spark is responsible for the integrated and diversified calculation of big data.

Thank you for your reading, the above is the content of "coexistence of spark and hadoop", after the study of this article, I believe you have a deeper understanding of the coexistence of spark and hadoop, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.