Get the App
SLTechnology News&Howtos  ›  Development  › 

How to use eclipse spark

Shulou Source: shulou.com Published: 2022-06-03 06:05:12 11月17日 Update

This article mainly introduces "how to use eclipse spark". In daily operation, I believe many people have doubts about how to use eclipse spark. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to use eclipse spark"! Next, please follow the editor to study!

Package my.test;import java.io.IOException;import org.apache.spark.deploy.SparkSubmit;import com.huangyueran.spark.utils.Constant;public class Main {public static void main (String [] args) throws IOException {System.setProperty ("HADOOP_USER_NAME", "hadoop"); System.setProperty ("user.name", "hadoop") / / System.setProperty ("HADOOP_CONF_DIR", "C:\ eclipse-workspace\\ SparkDemo\\ src\\ main\\ resources"); / / System.setProperty ("HADOOP_CONF_DIR", "C:\\ my\\ soft\\ hadoop\ hadoop-2.8.5\\ hadoop-2.8.5\\ etc\\ hadoop") System.out.println ("-" + System.getenv ("HADOOP_CONF_DIR")); System.out.println ("-" + System.getenv ("HADOOP_HOME")); String appName = "wordCount-yarn-cluster"; String className = "my.test.WordCount" String path = "C:\ eclipse-workspace\\ SparkDemo\\ target\\ SparkDemo-1.0-SNAPSHOT.jar"; path = Constant.HDFS_FILE_PREX + "/ user/zzm/SparkDemo-1.0-SNAPSHOT.jar" String [] arg0=new String [] {/ / "--jars", Constant.HDFS_FILE_PREX + "/ user/zzm/spark-lib", "--master", "yarn", / / ip port "--deploy-mode", "cluster", "--name", appName "--class", className,// runs the main classes main / / "--spark.yarn.archive", Constant.HDFS_FILE_PREX + "/ user/zzm/spark-lib", "--executor-memory", "2G", "--total-executor-cores", "10", "--executor-cores", "2" The package of path,// on linux can be changed to the parameters in the path / / "LR", "20180817111111", "66" / / jar on hdfs. Pay attention to the parameter writing here} SparkSubmit.main (arg0);}}

Stuck all the time:

New configuration solution for spark-defaults.conf:

# spark.yarn.jar hdfs://udp02:8020/user//spark-lib/*.jar

At this point, the study on "how to use eclipse spark" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Tags: Learning parameters more help practical next writing article method theory knowledge port article website material follow path problem face easy to use Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno OPPO Reno MySQL Huawei macOS MariaDB