How to solve the exception of eclipse spark submitting remote tasks
This article mainly explains "how to solve the eclipse spark submission remote task exception". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to solve the eclipse spark submission remote task exception".
Java.lang.IllegalStateException: Library directory scala-2.11\ jars' does not exist; make sure Spark is built.package my.test;import java.io.IOException;import org.apache.spark.deploy.SparkSubmit;import com.huangyueran.spark.utils.Constant;public class Main {public static void main (String [] args) throws IOException {System.setProperty ("HADOOP_USER_NAME", "hadoop"); System.setProperty ("user.name", "hadoop") / / System.setProperty ("HADOOP_CONF_DIR", "C:\ eclipse-workspace\\ SparkDemo\\ src\\ main\\ resources"); / / System.setProperty ("HADOOP_CONF_DIR", "C:\\ my\\ soft\\ hadoop\ hadoop-2.8.5\\ hadoop-2.8.5\\ etc\\ hadoop") System.out.println ("-" + System.getenv ("HADOOP_CONF_DIR")); System.out.println ("-" + System.getenv ("HADOOP_HOME")); String appName = "wordCount-yarn-cluster"; String className = "my.test.WordCount" String path = "C:\ eclipse-workspace\\ SparkDemo\\ target\\ SparkDemo-1.0-SNAPSHOT.jar"; path = Constant.HDFS_FILE_PREX + "/ user/zzm/SparkDemo-1.0-SNAPSHOT.jar" String [] arg0=new String [] {/ / "--jars", Constant.HDFS_FILE_PREX + "/ user/zzm/spark-lib", "--master", "yarn", / / ip port "--deploy-mode", "cluster", "--name", appName "--class", className,// runs the main classes main / / "--spark.yarn.archive", Constant.HDFS_FILE_PREX + "/ user/zzm/spark-lib", "--executor-memory", "2G", "--total-executor-cores", "10", "--executor-cores", "2" The package of path,// on linux can be changed to the parameters in the path / / "LR", "20180817111111", "66" / / jar on hdfs. Pay attention to the parameter writing here} SparkSubmit.main (arg0);}}
Spark and hadoop need to be configured for local environment before yarn cluster can be submitted locally
Thank you for your reading, the above is the content of "how to solve eclipse spark submitting remote task exception". After the study of this article, I believe you have a deeper understanding of how to solve the problem of eclipse spark submitting remote task exception, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!