Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to realize the Spark program". In the daily operation, I believe that many people have doubts about how to realize the Spark program. The editor has consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how to realize the Spark program". Next, please follow the editor to study!

Import org.apache.spark.sql.SparkSession

Import java.net.URLDecoder

Import org.apache.hadoop.hdfs.DistributedFileSystem

Import org.apache.hadoop

Object SparkMapreduce {

Def main (args: Array [String]) {

Val session = SparkSession.builder () .master ("local [2]") .getOrCreate ()

Val sc = session.sparkContext

Val path = SparkMapreduce.getClass.getResource ("readme.md")

Val pathStr = path.getFile

Println (pathStr)

Val textFile = sc.textFile ("file://" + pathStr)

Val count = textFile.count ()

Println (count)

Val linesWithSpark = textFile.filter (line = > {

/ / println (line)

Line.contains ("UnderConstruction")

})

/ / after removing collect, it will not print, which illustrates the difference between transform and action.

LinesWithSpark.collect () .map {println (_)}

Val url = SparkMapreduce.getClass.getResource ("classified prediction .csv")

/ / Chinese will be encoded, which is reversed here

Val predictionPath = URLDecoder.decode (url.getPath, "utf-8")

Val predictionDF = "file://" + predictionPath"

Val predictionRDD = sc.textFile (predictionDF, 3)

/ / We print out the text.

PredictionRDD.collect () .map {println (_)}

}

}

At this point, the study of "how to implement the Spark program" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.