Analysis of the problem of Spark Parameter transfer
I. sparkconf this needs to be written in the code, written in the code, then can not change, even if you pass a parameter outside, will eventually take effect in the code, its highest priority
II. Sparksubmit / spark-shell scripts, etc., usually pass parameters for my application here
III. The configuration file is in the conf directory. Note that if you need to make the parameters take effect, you must remove the template: default, global
Parameters can be set in these three places. If we set the same parameter, which place will take effect in the end?
The priority of these three places is: sparkconf > sparksubmit > profile