Spark local operation mode
Spark sets setMaster=local, does not submit clusters, and enables multithreaded simulation to run locally
Object SparkUtil {privateval logger = Logger.getLogger (getClass.getName, true) def getSparkContext (appName:String, local:Boolean=false, threadNum:Int=4): SparkContext = {val conf = new SparkConf (). SetAppName (appName) if (local) conf.setMaster (s "local [$threadNum]") val sc = new SparkContext (conf) return sc} SparkUtil.getSparkContext (" testApp ", true) / / start 4 threads locally to simulate running
Running in local mode can test the main program logic and may not run correctly in the cluster.