Example Analysis of ALS algorithm for MLlib Collaborative filtering
In this issue, the editor will bring you an example analysis of the MLlib collaborative filtering ALS algorithm. The article is rich in content and analyzed and described from a professional point of view. I hope you can get something after reading this article.
ALS algorithm for collaborative filtering is roughly to establish a user commodity matrix and solve it in the form of Sudoku according to the score.
Import java.text.SimpleDateFormatimport java.util.Dateimport org.apache.spark.mllib.recommendation. {ALS, Rating} import org.apache.spark. {SparkContext, SparkConf} / * Created by hadoop on 2015-7-20. * / object MLlibCF {def main (args: Array [String]) {val time = new SimpleDateFormat ("MMddHHmm") .format (new Date ()) val sparkConf = new SparkConf (). SetAppName ("MLlibCF-" + time) sparkConf.set ("mapreduce.framework.name") "yarn") sparkConf.set ("spark.rdd.compress", "true") / / whether serialized rdd partitions need to be compressed Sacrifice cpu time to improve space utilization sparkConf.set ("spark.serializer", "org.apache.spark.serializer.KryoSerializer") / / configure serialized interfaces sparkConf.set ("spark.storage.memoryFraction", "0.2") sparkConf.set ("spark.scheduler.mode", "FAIR") sparkConf.set ("spark.ui.port", "4042") sparkConf.set ("spark.akka.frameSize") Val sc = new SparkContext (sparkConf) val data = sc.textFile ("hdfs://namenode:9000/data/test_in/mahout1.txt", 1) / / A preprocessing of read files And put val ratings = data.map (_ .split (",") match {case (Array (user, product, rate)) = > Rating (user.toInt, product.toInt, rate.toDouble)}) / / the required value val user1 = sc.parallelize ("1105", "1106", "2105", "2107", "3102") in the Rating container. Map (_ .split (",") match {case (Array (user (user)) Product)) = > (user.toInt, product.toInt)}) val rank = 10 val numIterations = 20 / / build ALS model val model = ALS.train (ratings, rank, numIterations, .01) / / read the required values val predictions = model.predict (user1) .map {case Rating (user, product, rate) = > (user, product) Rate)} predictions.saveAsTextFile ("hdfs://10.207.0.217:9000/data/test_out/zk/MLlib-" + time)}} above is an example of the MLlib collaborative filtering ALS algorithm shared by the editor. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.
