Construction of stand-alone Spark environment under Windows 10
[objective]
Windows 10 bare metal builds a Spark environment that enables PySpark Shell to run.
[raw material]
Windows 10 x64
Jdk-8u162-windows-x64.exe
Python-3.6.7-amd64.exe
Spark-2.3.2-bin-hadoop2.7.tgz
Hadoop-2.7.7.tar.gz
Winutils.exe
[install JDK]
Double-click the jdk-8u162-windows-x64.exe installation, and the follow-up is basically "next" all the way.
Configure environment variables
# can be verified by echo% JAVA_HOME% JAVA_HOME: C:\ Program Files\ Java\ jdk1.8.0_162# can be verified by echo% CLASS_PATH% CLASS_PATH: C:\ Program Files\ Java\ jdk1.8.0_162\ lib# can be verified by echo% PATH% Path: C:\ Program Files\ Java\ jdk1.8.0_162\ bin
[install Python3]
Double-click python-3.6.7-amd64.exe installation
To facilitate code completion, it is recommended to install the following two third-party packages
# install ipythonpip3 install ipython-I https://pypi.doubanio.com/simple/# install pyreadlinepip3 install pyreadline-I https://pypi.doubanio.com/simple/
[install Spark]
Download spark-2.3.2-bin-hadoop2.7.tgz from http://spark.apache.org/downloads.html and extract it, and walker is decompressed to the D:\ spark directory.
At this point, pyspark is ready to run, but you will be prompted that winutils.exe cannot be found.
Configure environment variables
SPARK_HOME: d:\ spark\ spark-2.3.2-bin-hadoop2.7PATH: d:\ spark\ spark-2.3.2-bin-hadoop2.7\ bin
[install Hadoop]
Download hadoop-2.7.7.tar.gz from https://archive.apache.org/dist/hadoop/common/hadoop-2.7.7/ and extract it. If you get the error "Can not create symbolic link: the client does not have the required privileges." Run the decompression software as an administrator and then decompress it
Configure environment variables
HADOOP_HOME: d:\ spark\ hadoop-2.7.7PATH: d:\ spark\ hadoop-2.7.7\ bin
Modify the D:\ spark\ hadoop-2.7.7\ etc\ hadoop\ file, otherwise an error like "Error: JAVA_HOME is incorrectly set. Please update F:\ hadoop\ conf\ hadoop-env.cmd" may be reported.
# because the JAVA_HOME environment variable has spaces, modify set JAVA_HOME=%JAVA_HOME%#-- > set JAVA_HOME=C:\ PROGRA~1\ Java\\ jdk1.8.0_162
If you turn on cmd and input hadoop version, the following output is normal.
Hadoop 2.7.7Subversion Unknown-r c1aad84bd27cd79c3d1a7dd58202a8c3ee1ed3acCompiled by stevel on 2018-07-18T22:47ZCompiled with protoc 2.5.0From source with checksum 792e15d20b12c74bd6f19a1fb886490This command was run using / D:/spark/hadoop-2.7.7/share/hadoop/common/hadoop-common-2.7.7.jar
[winutils.exe]
Download the corresponding version of winutils.exe from https://github.com/steveloughran/winutils and put it in D:\ spark\ hadoop-2.7.7\ bin.
[Python script test]
The script is as follows (t.py)
From pyspark import SparkConf, SparkContextconf = SparkConf (). SetMaster ("local"). SetAppName ("My App") sc = SparkContext (conf = conf) print ('*% s'% sc.appName) sc.stop ()
Run the script with the spark-submit t.py command with the following output as correct.
* * My App
[FAQ]
ModuleNotFoundError: No module named 'resource'
Maybe the spark version you are using is too new for Windows 10, just switch to Spark 2.3.2 or below.
[related reading]
Configure the pyspark work environment on windows10
* walker * *