Example Analysis of macOS Spark 2.4.3 standalone Construction
This article is to share with you the example analysis of macOS Spark 2.4.3 standalone building, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.
Based on
Jdk 1.8
Mac os
1. Decompression
Nancylulululu:local nancy$ tar-zxvf / Users/nancy/Downloads/spark-2.4.3-bin-hadoop2.7.tar-C / usr/local/opt/
2. Rename and modify the configuration file
Nancylulululu:conf nancy$ cp slaves.template slaves
Nancylulululu:conf nancy$ cp spark-defaults.conf.template spark-defaults.conf
Nancylulululu:conf nancy$ cp spark-env.sh.template spark-env.sh
Nancylulululu:conf nancy$ vi slaves
Localhost
Nancylulululu:conf nancy$ vi spark-env.sh
SPARK_LOCAL_IP=127.0.0.1
SPARK_MASTER_HOST=127.0.0.1
SPARK_MASTER_PORT=7077
SPARK_WORKER_CORES=2
SPARK_WORKER_MEMORY=1G
3. Start spark
Nancylulululu:spark-2.4.3 nancy$ sbin/start-all.sh
Starting org.apache.spark.deploy.master.Master, logging to / usr/local/opt/spark-2.4.3/logs/spark-nancy-org.apache.spark.deploy.master.Master-1-nancylulululu.out
Localhost: starting org.apache.spark.deploy.worker.Worker, logging to / usr/local/opt/spark-2.4.3/logs/spark-nancy-org.apache.spark.deploy.worker.Worker-1-nancylulululu.out
Jps views a master and a worker service
Nancylulululu:conf nancy$ jps
3908 CoarseGrainedExecutorBackend
3478 Master
4166 Jps
3896 SparkSubmit
3514 Worker
4 、 Launching Applications with spark-submit
Once a user application is bundled, it can be launched using the bin/spark-submit script. This script takes care of setting up the classpath with Spark and its dependencies, and can support different cluster managers and deploy modes that Spark supports:
. / bin/spark-submit\-- class\-- master\-- deploy-mode\-- conf =. # other options\ [application-arguments]
Some of the commonly used options are:
-- class: The entry point for your application (e.g. org.apache.spark.examples.SparkPi)
-- master: The master URL for the cluster (e.g. spark://23.195.26.187:7077)
-- deploy-mode: Whether to deploy your driver on the worker nodes (cluster) or locally as an external client (client) (default: client)
Conf: Arbitrary Spark configuration property in key=value format. For values that contain spaces wrap "key=value" in quotes (as shown).
Application-jar: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a file:// path that is present on all nodes.
Application-arguments: Arguments passed to the main method of your main class, if any
Script:
. / bin/spark-submit\-class org.apache.spark.examples.SparkPi\-master spark://localhost:7077\ / usr/local/opt/spark-2.4.3/examples/jars/spark-examples_2.11-2.4.3.jar\ 100
Result
Pi is roughly 3.1413047141304715
14:23:06 on 19-05-14 INFO SparkUI: Stopped Spark web UI at http://localhost:4040
5. Spark-shell mode
Nancylulululu:spark-2.4.3 nancy$ bin/spark-shell-master spark://localhost:7077
6. Test wordcount
Test file
Nancylulululu:opt nancy$ vi 3.txt
Hello scala hello spark hello mysql hello java hello java hello scala
Test result
Scala > sc.textFile ("/ usr/local/opt/3.txt"). FlatMap (_ .split (")). Map ((_, 1)). ReduceByKey (_ + _). Collect
Res1: Array [(String, Int)] = Array ((scala,2), (mysql,1), (hello,6), (java,2), (spark,1))
Scala >
The above is the example analysis built by macOS Spark 2.4.3 standalone. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.
