Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

Analysis of the problem of Spark Parameter transfer

Shulou Source: shulou.com Published: 2022-06-03 07:17:57 11月23日 Update

I. sparkconf this needs to be written in the code, written in the code, then can not change, even if you pass a parameter outside, will eventually take effect in the code, its highest priority

II. Sparksubmit / spark-shell scripts, etc., usually pass parameters for my application here

III. The configuration file is in the conf directory. Note that if you need to make the parameters take effect, you must remove the template: default, global

Parameters can be set in these three places. If we set the same parameter, which place will take effect in the end?

The priority of these three places is: sparkconf > sparksubmit > profile

Tags: Parameters code place three priority file configuration highest global directory script application problem Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno Redmi Shulou Tech Info Shulou Information Shulou Technology MySQL