Shulou(Shulou.com)06/03 Report--

1. Download as follows

Put it on disk D.

Add SPARK_HOME = D:\ spark-2.3.0-bin-hadoop2.7.

And add% SPARK_HOME%/bin to the environment variable PATH. Then enter the command line and enter the pyspark command. If successfully carried out. The environment variable is set successfully

Find the pycharm sitepackage directory

Right-click to enter the directory and copy the / python/pyspark directory in D:\ spark-2.3.0-bin-hadoop2.7 above to the sitepackage directory above

Install py4j

Experiment with the following code:

From _ _ future__ import print_functionimport sysfrom operator import addimport os# Path for spark source folderos.environ ['SPARK_HOME'] = "D:\ spark-2.3.0-bin-hadoop2.7" # Append pyspark to Python Pathsys.path.append ("D:\ spark-2.3.0-bin-hadoop2.7\ python") sys.path.append ("D:\ spark-2.3.0-bin-hadoop2.7\ python\ lib\ py4j-0.9-src.zip") From pyspark import SparkContextfrom pyspark import SparkConfif _ _ name__ = ='_ main__': inputFile = "D:\ Harry.txt" outputFile = "D:\ Harry1.txt" sc = SparkContext () text_file = sc.textFile (inputFile) counts = text_file.flatMap (lambda line: line.split ('')) .map (lambda word: (word 1)) .reduceByKey (lambda a, b: a + b) counts.saveAsTextFile (outputFile)

As long as the calculation is successful

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.