Why do we choose Spark On Yarn mode in production?

Why do we choose Spark On Yarn in production?

In development, we choose local [2] mode.

Production running task Job, we choose Spark On Yarn mode

Deploying Spark Application into yarn has the following advantages:

1. It is easier to deploy Application and services

Only yarn services are needed, and many applications, including Spark,Storm, do not bring their own services. After they are submitted by the client, they are distributed to each computing node by the distributed caching mechanism provided by yarn.

two。 Resource isolation mechanism

Yarn is only responsible for resource management and scheduling, and it is entirely up to users and themselves to decide which service and Applicatioin to run on the yarn cluster, so it is possible to run multiple similar services and Application on yarn at the same time. Yarn uses Cgroups to achieve resource isolation. Users do not have to worry about resource isolation when developing new services or Application.

3. Flexible management of resources

Yarn can manage multiple services running in the yarn cluster simultaneously through queues, and adjust the corresponding resource usage according to the pressure of different types of applications to achieve flexible resource management.

There are two modes of Spark On Yarn, one is cluster mode, the other is client mode.

Run client mode:

". / spark-shell-master yarn". / spark-shell-master yarn-client ". / spark-shell-master yarn- deploy-mode client"

Running in cluster mode

". / spark-shell-master yarn-cluster". / spark-shell-master yarn- deploy-mode cluster "

The main difference between client and cluster mode: a. Client's driver runs in client processes. B. cluster's driver runs in Application Master