Spark analysis website ranking heat
Demand:
/ solution is a hot ranking within each sub-module-- "ranking must be done with sortBy--" (it may be a simple sort, or a secondary ranking)-- "
/ / there is a wordCount--- to "calculate the number of times--" what to consider as a key
/ / under a website, the top 2 hottest pages under each sub-module:
/ / under a website, the top 2 hottest pages under each sub-module-"the top 2 of the number of pages under each sub-module
/ / in actual development, the real code time may only account for 20-30%. The rest of the time is to understand the requirements and think about ideas.
Import java.net.URLimport org.apache.spark. {SparkConf, SparkContext} object Data_anlysis {def main (args: Array [String]) {val conf = new SparkConf (). SetAppName ("data-anysis"). SetMaster ("local [2]") val sc = new SparkContext (conf) val linesRDD = sc.textFile ("d://web_data.log") val rdd01 = linesRDD.map (line = > {val lineSplited = line.split ("\ t") (lineSplited (1)) 1)}) val rdd02 = rdd01.reduceByKey (_ + _) val rdd03 = rdd02.map (tup = > {val url = tup._1 Val host = new URL (url). GetHost (host,url,tup._2)}) val rdd04 = rdd03.groupBy (_. _ 1) val rdd05 = rdd04.mapValues (iter = > {iter.toList.sortBy (_. _ 3) .explorse.take (2)}) println (rdd05.collect (). ToBuffer) sc.stop ()}}
The running results will display the log data of the site in accordance with the requirements of a certain module to list the top two results.