How to configure yarn-site.xml minimization in Spark

This article mainly explains "how to minimize the configuration of yarn-site.xml in Spark". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to minimize the configuration of yarn-site.xml in Spark".

Modify the configuration file slaves as follows:

We set the slave nodes in the Hadoop cluster to SparkWorker1 and SparkWorker2, and modify the contents of the slaves file to:

Modify the configuration file core-site.xml as follows:

Modify the contents of the core-site.xml file to:

The above is the minimized configuration of core-site.xml files

Modify the configuration file hdfs-site.xml as follows:

Modify the contents of the hdfs-site.xml file to:

The above is the minimized configuration of hdfs-site.xml files

Modify the configuration file mapred-site.xml as follows:

Make a copy of mapred-site.xml.template and name it mapred-site.xml. Open mapred-site.xml.

Modify the contents of the mapred-site.xml file to:

The above is the mapred-site.xml minimized configuration

Modify the configuration file yarn-site.xml as follows:

Modify the content of yarn-site.xml to:

Thank you for reading, the above is the content of "how to minimize the configuration of yarn-site.xml in Spark". After the study of this article, I believe you have a deeper understanding of how to minimize the configuration of yarn-site.xml in Spark, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!