Big data: spark cluster building
Create a spark user group, group ID1000
Groupadd-g 1000 spark
For spark users who create user ID 2000 under the spark user group to obtain Chinese documents and complete videos, please add QQ group: 947967114
Useradd-u 2000-g spark spark
Set password
Passwd spark
Modify sudo permissions
Chmod Ubunw / etc/sudoers
Vi / etc/sudoers
find
Root ALL= (ALL) ALL
Add
Spark ALL= (ALL) ALL
Create an app directory to store the spark software environment (jdk, scala, spark)
Mkdir / app
Modify the group and owner of this file
Chown-R spark:spark / app
Create soft
Mkdir / app/soft
Create spark
Mkdir / app/spark
Create / spark/work
Mkdir-p / home/spark/work
Change / spark/work group and owner
Chown-R spark:spark / home/spark/work
Switch users
Su root
Decompress JDK
Cd / tmp/
Tar zxvf jdk-8u192-linux-x64.tar.gz-C / app/soft/
If you don't have permission, use chmod 777-R / tmp to modify permission first.
Cd / app/soft/
Ll-a
Configuration / etc/profile
Sudo vi / etc/profile, all required configurations have been added
JAVA_HOME=/app/soft/jdk1.8.0_192
PATH=$JAVA_HOME/bin:$PATH:$HOME/bin
Export PATH
To make the configuration effective, please add QQ group to get the documents and complete videos in the video. Please add QQ group: 947967114
Source / etc/profile
Install scala:
Tar zxvf / tmp/scala-2.11.12.tgz-C / app/soft/
Configure environment variables
Sudo vi / etc/profile
JAVA_HOME=/app/soft/jdk1.8.0_192
SCALA_HOME=/app/soft/scala-2.11.12/
PATH=$JAVA_HOME/bin:$PATH:$HOME/bin:$SCALA_HOME/bin
Export PATH
Configure ssh unsecret login
Ssh-keygen-t rsa
Cd ~ /
Cd .ssh /
Modify the name of the public key
Master node: mv id_rsa.pub authorized_keys_master.pub
Slave1 node: mv id_rsa.pub authorized_keys_slave1.pub
Slave2 node: mv id_rsa.pub authorized_keys_slave2.pub
Give the public keys of slave1 and slave2 to master
Slave1 node: scp authorized_keys_slave1.pub spark@master:/home/spark/.ssh/
Slave2 node: scp authorized_keys_slave2.pub spark@master:/home/spark/.ssh/
Write the public keys of all three nodes in one file
Cat authorized_keys_master.pub > > authorized_keys
Cat authorized_keys_slave1.pub > > authorized_keys
Cat authorized_keys_slave2.pub > > authorized_keys
Check the total public key file
Vi authorized_keys
Give the total public key file authorized_keys to the slave1 and slave2 nodes
Scp authorized_keys spark@slave1:/home/spark/.ssh
Scp authorized_keys spark@slave2:/home/spark/.ssh
Modify the operation permission of authorized_keys. All three nodes need to be modified.
Chmod 400 authorized_keys
Verify that secret-free login is successful
Ssh master
Ssh slave1
Ssh slave2
Ssh master
Install spark:
Tar-zxf / tmp/spark-2.1.0-bin-hadoop2.6.gz-C / app/spark/
Cd / app/spark/
Ls
Cd spark-2.1.0-bin-hadoop2.6/
Configure environment variables:
Vi / etc/profile
JAVA_HOME=/app/soft/jdk1.8.0_192
SCALA_HOME=/app/soft/scala-2.11.12/
SPARK_HOME=/app/spark/spark-2.1.0-bin-hadoop2.6
PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$JAVA_HOME/bin:$PATH:$HOME/bin:$SCALA_HOME/bin
Export PATH
Configure the core file for spark:
Cd spark-2.1.0-bin-hadoop2.6/
Cd conf/
Configure slaves
Mv slaves.template slaves
Vi slaves add three nodes
Master
Slave1
Slave2
Configure spark-env.sh
Cp spark-env.sh.template spark-env.sh
Vi spark-env.sh
Export JAVA_HOME=/app/soft/jdk1.8.0_192
Export SCALA_HOME=/app/soft/scala-2.11.12
Export SPARK_MASTER_IP=master
Export SPARK_MASTER_PORT=7077
Export SPARK_EXECUTOR_INSTANCES=1
Export SPARK_WORKER_INSTANCES=1
Export SPARK_WORKER_CORES=1
Export SPARK_WORKER_MEMORY=1024M
Export SPARK_MASTER_WEBUI=8080
Export SPARK_CONF_DIR=/app/spark/spark-2.1.0-bin-hadoop2.6/conf/
Change the work and soft permissions of app on all nodes to 777: execute chmod 777-R / app/soft and chmod 777-R / app/spark on all nodes
Scp-r / app/spark/ spark@slave1:/app/
Scp-r / app/soft/ spark@slave1:/app/
At this point, the spark cluster has been built:
Open: start-all.sh to get the video documents and complete video partners, please add QQ group: 947967114
Jps can see the following process:
Master node:
3617 Worker
3507 Master
4156 Jps
Slave1 node:
3361 Worker
3702 Jps
Slave2 node:
3319 Worker
3647 Jps
Enable spark-shell authentication:
Spark-shell-- master spark://master:7077-- executor-memory 1024m-- driver-memory 1024m
After startup, the following content is displayed:
18-11-29 16:13:46 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
18-11-29 16:13:47 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.0.10:4040
Spark context available as' sc' (master = spark://master:7077, app id = app-20181129161336-0000).
Spark session available as' spark'.
Welcome to
_ /
/ /. / _, / _\ version 2.1.0
/ _ /
Using Scala version 2.11.8 (Java HotSpot (TM) 64-Bit Server VM, Java 1.8.0 / 192)
Type in expressions to have them evaluated.
Type: help for more information.
Scala >
You can write the spark code after >:
G NoSuchObjectException
Spark context Web UI available at http://192.168.0.10:4040
Spark context available as' sc' (master = spark://master:7077, app id = app-20181129161336-0000).
Spark session available as' spark'.
Welcome to
_ /
/ /. / _, / _\ version 2.1.0
/ _ /
Using Scala version 2.11.8 (Java HotSpot (TM) 64-Bit Server VM, Java 1.8.0 / 192)
Type in expressions to have them evaluated.
Type: help for more information.
Scala > sc.textFile ("/ app/spark/spark-2.1.0-bin-hadoop2.6/README.md"). FlatMap (.split (")) .map (x = > (xmem1)). ReduceByKey (+ _) .map (x = > (x.Zong2jue x.train1)) .sortByKey (false) .map (x = > (x.split (") .take (10)
Res0: Array [(String, Int)] = Array ((", 71), (the,24), (to,17), (Spark,16), (for,12), (and,9), (# #, 8), (can,7), (run,7)
Scala > Please add QQ group to get the documents in the video and the complete video: 947967114.
