How to build spark Cluster
This article mainly introduces "how to build spark cluster". In daily operation, I believe many people have doubts about how to build spark cluster. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "how to build spark cluster". Next, please follow the editor to study!
Configure linux
Install epel
Add epel source time under CentOS6.5: 2014-09-20 01:46 Source: cnblogs author: born my talent P useful tip click: 22665 times 0. Install yum priority plug-in yum install yum-priorities1.epel introduction: https://fedoraproject.org/wiki/EPEL/zh-cnrpm-Uvh http://mirrors.ustc.edu.cn/fedora/epel/6/x86_64/epel-release-6-8.noarch.rpmrpm-Uvh http://rpms.famillecollet.com/enterprise/remi-release-6.rpm above URL please modify 2. Check to see if rpm-Q epel-release 3. Import key:rpm-- import / etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-64. Modify / etc/yum.repos.d/epel.repo file to add an attribute priority=11vi / etc/yum.repos.d/epel.repo at the end of [epel]. It means that yum goes to the official source first, but the official does not go to the source of epel to find 5. Rebuild cache yum makecache
Configure the environment
Scp root@10.10.11.191:/root/Download/jdk.tar.gz jdk.tar.gzscp root@10.10.11.191:/root/Download/apache-maven-3.3.9-bin.tar.gz apache-maven-3.3.9-bin.tar.gzscp root@10.10.11.191:/root/Download/spark-2.0.1-SNAPSHOT-bin-nevermore-spark.tgz spark-2.0.1-SNAPSHOT-bin-nevermore-spark.tgzvi / etc/profile+++JAVA_HOME=/opt/env/jdk1.7.0_79MAVEN_HOME=/opt/env/maven-3.3.9CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarPATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATHexport JAVA_HOMEexport CLASSPATH:wqsource / etc/profilevi / etc/hosts10.10.11.191 spark.env:wqssh-keygen-t rsatouch / root/.ssh/authorized_keyschmod 600 ~ / .ssh/authorized_keys./sbin/start- Slave.sh spark://spark.env:7077 has come here The study on "how to build a spark cluster" is over. I hope I can solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!
