Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

Spark installation configuration

Shulou Source: shulou.com Published: 2022-06-03 06:29:35 11月23日 Update

The installation of Scala Scala before installing spark has been introduced in the hadoop installation configuration

1. Download the spark installation package at the following address

Http://spark.apache.org/downloads.html

What I chose is

Spark-1.4.1-bin-hadoop2.6.tgz on / root/software

Decompression

Tar zxvf spark-1.4.1-bin-hadoop2.6.tgz

2. Configure system environment variables

Vim / etc/profile

Export SPARK_HOME=/root/sherry/spark-1.4.1

Export PATH=$SPARK_HOME/bin:$PATH

Source / etc/profile

3. Configure the spark file

1) [root@HBMS-DKXD-DEV sherry] # cd spark-1.4.1-bin-hadoop2.6/

[root@HBMS-DKXD-DEV spark-1.4.1-bin-hadoop2.6] # cd conf

[root@HBMS-DKXD-DEV conf] # vim spark-env.sh

Add the following

Export SCALA_HOME=/root/sherry/scala-2.10.4

Export JAVA_HOME=/usr/java/jdk1.7.0_45

Export SPARK_WORKER_MEMORY=1g

Export SPARK_MASTER_IP=10.118.46.22

2) configure slave

I don't have my own configuration.

4. Start the cluster

[root@HBMS-DKXD-DEV spark-1.4.1-bin-hadoop2.6] #. / sbin/start-all.sh

Close

[root@HBMS-DKXD-DEV spark-1.4.1-bin-hadoop2.6] #. / sbin/stop-all.sh

Check whether the installation is successful

Jps

Tags: Configuration success content variables address file environment system cluster first check select Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno Xiaomi MySQL OPPO Reno vpn Shulou Tech Info