Installation and basic programming of Spark

Linux system: Ubuntu 16.04

Hadoop: 2.7.1

JDK: 1.8

Spark: 2.4.3

one。 Download the installation file

Http://spark.apache.org/downloads.html

Https://archive.apache.org/dist/spark/

Hadoop@dblab:/usr/local$ sudo wget http://mirror.bit.edu.cn/apache/spark/spark-2.4.3/spark-2.4.3-bin-hadoop2.7.tgz

Hadoop@dblab:/usr/local$ sudo tar-zxf spark-2.4.3-bin-hadoop2.7.tgz-C spark

Hadoop@dblab:/usr/local$ sudo chown-R hadoop:hadoop spark/

two。 Configure related files

Hadoop@dblab:/usr/local/spark$. / conf/spark-env.sh.template. / conf/spark-env.sh

Export SPARK_DIST_CLASSPATH=$ (/ usr/local/hadoop/bin/hadoop classpath)

# verify whether Spark is installed successfully

Hadoop@dblab:/usr/local/spark$ bin/run-example SparkPi

Pi is roughly 3.139035695178476

three。 Start Spark Shell

Hadoop@dblab:/usr/local/spark$. / bin/spark-shell

Welcome to

_

/ _ _ / _ / / _ _

_ / _ `/ _ _ /'_ /

/ _ _ /. _ _ /\ _, _ /\ _\ version 2.1.0

/ _ /

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0,212)

Type in expressions to have them evaluated.

Type: help for more information.

Scala >

Scala > 8 / 2 / 5

Res0: Int = 21

four。 Read a file

1. Read local files

Hadoop@dblab:/usr/local/hadoop$. / sbin/start-dfs.sh

Scala > val textFile=sc.textFile ("file:///usr/local/spark/README.md")"

TextFile: org.apache.spark.rdd.RDD [String] = file:///usr/local/spark/README.md MapPartitionsRDD [1] at textFile at: 24

Scala > textFile.first ()

Res0: String = # Apache Spark

two。 Read HDFS file

Hadoop@dblab:/usr/local/hadoop$. / bin/hdfs dfs-put / usr/local/spark/README.md.

Hadoop@dblab:/usr/local/hadoop$. / bin/hdfs dfs-cat README.md

Scala > val textFile=sc.textFile ("hdfs://localhost:9000/user/hadoop/README.md")

TextFile: org.apache.spark.rdd.RDD [String] = hdfs://localhost:9000/user/hadoop/README.md MapPartitionsRDD [3] at textFile at: 24

Scala > textFile.first ()

Res1: String = # Apache Spark

Scala >: quit