SPARK installation
Use the installation package:
Scala-2.10.3.tgz
Spark-0.9.0-incubating-bin-hadoop2.tgz
Hadoop-2.3.0-cdh6.0.0.tar.gz
Jdk1.7.0_45
Download address: http://mirror.bit.edu.cn/apache/spark/spark-0.9.0-incubating/
Spark-0.9.0-incubating-bin-hadoop2.tgz
Http://archive.cloudera.com/cdh6/cdh/5/ downloads sacala and hadoop
Install softwar
Tarzxvf spark-0.9.0-incubating-bin-hadoop2.tgz
Tar zxvf scala-2.10.3.tgz
Configure user environment variable ~ / .bashrc
Export JAVA_HOME=/hadoop/jdk1.7.0_45
Export HADOOP_HOME=/hadoop/hadoop
Export HADOOP_CONF_DIR=/hadoop/hadoop-config
Export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin:$SPARK_HOME/bin:$PATH
Export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
Export HADOOP_HOME_WARN_SUPPRESS=1
Export SCALA_HOME=/hadoop/scala-2.10.3
Export SPARK_EXAMPLES_JAR=/hadoop/spark/examples/target/spark-examples_2.10-0.9.0-incubating.jar
Export SPARK_HOME=/hadoop/spark-0.9.0-incubating-bin-hadoop2
Configure environment variables: / hadoop/spark-0.9.0-incubating-bin-hadoop2/conf
Modify the spark-env.sh file to add the necessary configuration for spark:
Export SCALA_HOME=/hadoop/scala-2.10.3
Export JAVA_HOME=/hadoop/jdk1.7.0_45
Export SPARK_HOME=/hadoop/spark
Copy environment variables to other nodes with immediate effect of source .bash _ profile
Test environment variable scala-version
Configure the conf file:
Vi slaves
Bigdata-2
Bigdata-4
Files are distributed to other nodes
Spark launch:
Cd / hadoop/spark-0.9.0-incubating-bin-hadoop2/sbin./start-all.sh View process: jps22580 NameNode25767 Master27758 Jps23024 ResourceManager22812 SecondaryNameNode Node process: 70869 Worker70150 NodeManager71462 Jps70023 DataNode~ installed ~ spark use: run SparkPi: cd / hadoop/spark/bin
Ll / hadoop/spark/bin
-rw-rw-r--. 1 hadoop hadoop 2601 February 3 03:13 compute-classpath.cmd
-rwxrwxr-x. 1 hadoop hadoop 3330 February 3 03:13 compute-classpath.sh
-rwxrwxr-x. 1 hadoop hadoop 2070 February 3 03:13 pyspark
-rw-rw-r--. 1 hadoop hadoop 1827 February 3 03:13 pyspark2.cmd
-rw-rw-r--. 1 hadoop hadoop 1000 February 3 03:13 pyspark.cmd
-rwxrwxr-x. 1 hadoop hadoop 3055 February 3 03:13 run-example
-rw-rw-r--. 1 hadoop hadoop 2047 February 3 03:13 run-example2.cmd
-rw-rw-r--. 1 hadoop hadoop 1012 February 3 03:13 run-example.cmd
-rwxrwxr-x. 1 hadoop hadoop 5151 February 3 03:13 spark-class
-rwxrwxr-x. 1 hadoop hadoop 3212 February 3 03:13 spark-class2.cmd
-rw-rw-r--. 1 hadoop hadoop 1010 February 3 03:13 spark-class.cmd
-rwxrwxr-x. 1 hadoop hadoop 3038 February 303: 13 spark-shell
-rwxrwxr-x. 1 hadoop hadoop 941 February 3 03:13 spark-shell.cmd
. / run-example org.apache.spark.examples.SparkPi spark://master:7077
