Shulou(Shulou.com)06/03 Report--

Environment description: operating system: centos7 64-bit 3 centos7-1 192.168.190.130 master centos7-2 192.168.190.129 slave1 centos7-3 192.168.190.131 slave2 installation spark needs to be installed at the same time: jdk scale 1. Install jdk, configure jdk environment variables here do not talk about how to install and configure jdk, own Baidu. two。 Install scala to download the scala installation package, https://www.scala-lang.org/download/ select the version that meets the requirements to download, and use the client tool to upload to the server. Extract: # tar-zxvf scala-2.13.0-M4.tgz modify the / etc/profile file again and add the following: export SCALA_HOME=$WORK_SPACE/scala-2.13.0-M4 export PATH=$PATH:$SCALA_HOME/bin # source / etc/profile / / make it effective immediately # scala- version / / check whether the scala installation is complete. Install spark spark download address: http://spark.apache.org/downloads.html description: there are different versions of the package to download, choose what you need to download and install Source code: Spark source code, need to compile to use, in addition, Scala 2.11 need to use source code compilation to use Pre-build with user-provided Hadoop: "Hadoop free" version Can be applied to any Hadoop version of Pre-build for Hadoop 2.7and later: a pre-compiled version based on Hadoop 2.7needs to correspond to the locally installed version of Hadoop. Hadoop 2.6 is also optional. Because the hadoop installed here is 3.1.0, I directly install the version of for hadoop 2.7 and later. Note: for the installation of hadoop, please check my previous blog. I will not repeat the description.

# mkdir spark# cd / usr/spark#tar-zxvf spark-2.3.1-bin-hadoop2.7.tgz#vim / etc/profile# add spark environment variables, such as under PATH, export out # source / etc/profile# and enter the conf directory Make a copy of spark-env.sh.template and rename it spark-env.sh#cd / usr/spark/spark-2.3.1-bin-hadoop2.7/conf#cp spark-env.sh.template spark-env.sh#vim spark-env.shexport SCALA_HOME=/usr/scala/scala-2.13.0-M4export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.171-8.b10.el7_5.x86_64export HADOOP_ HOME=/usr/hadoop/hadoop-3.1.0export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport SPARK_HOME=/usr/spark/spark-2.3.1-bin-hadoop2.7export SPARK_MASTER_IP=masterexport SPARK_EXECUTOR_MEMORY=1G# enters the conf directory Copy a copy of slaves.template to be renamed slaves#cd / usr/spark/spark-2.3.1-bin-hadoop2.7/conf#cp slaves.template slaves#vim slaves# add node domain name to slaves file # master / / the domain name is centos7-1 # slave1 / / the domain name is centos7-2 # slave2 / / the domain name is centos7-3

Start spark

# start the hadoop node before starting spark

# cd / usr/hadoop/hadoop-3.1.0/

# sbin/start-all.sh

# jps / / check whether the started thread has started hadoop

# cd / usr/spark/spark-2.3.1-bin-hadoop2.7

# sbin/start-all.sh

Note: spark must also be installed on the slave1\ slave2 node as above, or a copy of the # scp-r / usr/spark root@slave1ip:/usr/spark startup information on the slave1,slave2 node must be directly copied as follows:

Starting org.apache.spark.deploy.master.Master, logging to / usr/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out

Slave2: starting org.apache.spark.deploy.worker.Worker, logging to / usr/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave2.com.cn.out

Slave1: starting org.apache.spark.deploy.worker.Worker, logging to / usr/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave1.com.cn.out

Master: starting org.apache.spark.deploy.worker.Worker, logging to / usr/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-master.out

Test the Spark cluster: open the spark cluster url: http://192.168.190.130:8080/ on the master node with a browser

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.