Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

Spark analysis website ranking heat

Shulou Source: shulou.com Published: 2022-06-03 07:22:00 11月23日 Update

Demand:

/ solution is a hot ranking within each sub-module-- "ranking must be done with sortBy--" (it may be a simple sort, or a secondary ranking)-- "

/ / there is a wordCount--- to "calculate the number of times--" what to consider as a key

/ / under a website, the top 2 hottest pages under each sub-module:

/ / under a website, the top 2 hottest pages under each sub-module-"the top 2 of the number of pages under each sub-module

/ / in actual development, the real code time may only account for 20-30%. The rest of the time is to understand the requirements and think about ideas.

Import java.net.URLimport org.apache.spark. {SparkConf, SparkContext} object Data_anlysis {def main (args: Array [String]) {val conf = new SparkConf (). SetAppName ("data-anysis"). SetMaster ("local [2]") val sc = new SparkContext (conf) val linesRDD = sc.textFile ("d://web_data.log") val rdd01 = linesRDD.map (line = > {val lineSplited = line.split ("\ t") (lineSplited (1)) 1)}) val rdd02 = rdd01.reduceByKey (_ + _) val rdd03 = rdd02.map (tup = > {val url = tup._1 Val host = new URL (url). GetHost (host,url,tup._2)}) val rdd04 = rdd03.groupBy (_. _ 1) val rdd05 = rdd04.mapValues (iter = > {iter.toList.sortBy (_. _ 3) .explorse.take (2)}) println (rdd05.collect (). ToBuffer) sc.stop ()}}

The running results will display the log data of the site in accordance with the requirements of a certain module to list the top two results.

Tags: Module size heat website web page face time times results requirements sorting code reasonable practical that is ideas data logs development running Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno MySQL Xiaomi Shulou Tech Info Microsoft OPPO Reno