Shulou(Shulou.com)06/03 Report--

Use Docker to install the project:

Local mode is to set sparklocal.spark.driver.host

In yarn mode, only sparklocal.spark.driver.host is set and can be started, but no statement can be executed. It needs to be set again.

Sparkyarn.spark.driver.bindAddress

And blockmanager

You can specify the host and bindAddress corresponding to the host and bindaddress,docker specified access settings when spark starts.

And blockmanager will do

Note: sparkyarn.spark.driver.bindAddress is a parameter unique to spark2.1.0, which was not available before. Therefore, before 2.1.0, the combination of spark and docker, the above approach does not work.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.