Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

Spark local operation mode

Shulou Source: shulou.com Published: 2022-06-03 06:57:20 11月23日 Update

Spark sets setMaster=local, does not submit clusters, and enables multithreaded simulation to run locally

Object SparkUtil {privateval logger = Logger.getLogger (getClass.getName, true) def getSparkContext (appName:String, local:Boolean=false, threadNum:Int=4): SparkContext = {val conf = new SparkConf (). SetAppName (appName) if (local) conf.setMaster (s "local [$threadNum]") val sc = new SparkContext (conf) return sc} SparkUtil.getSparkContext (" testApp ", true) / / start 4 threads locally to simulate running

Running in local mode can test the main program logic and may not run correctly in the cluster.

Tags: Run thread cluster pattern program logic test Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno MariaDB vpn Linux Shulou Information Shulou Technology