How to realize Fast iterative clustering with spark mllib
Editor to share with you spark mllib how to achieve fast iterative clustering, I hope you will learn something after reading this article, let's discuss it together!
The running code is as follows: package spark.clusteringimport org.apache.spark.mllib.clustering. {PowerIterationClustering} import org.apache.spark. {SparkConf, SparkContext} / * * Fast iterative clustering * basic principle: use undirected lines with weights to connect the sample data in an undirected graph, and then divide them according to similarity. * the data source requires RDD [(Long), (Long), (Double)] * the first parameter and the second parameter are the numbers of the first point and the second point, that is, the ID between them, and the third parameter is the similarity calculation value. * / object PIC {val conf = new SparkConf () / / create the environment variable .setMaster ("local") / / set the localization handler .setAppName ("pic") / / set the name val sc = new SparkContext (conf) Def main (args: Array [String]) {val data = sc.textFile (". / src/main/spark/clustering/pic.txt") val similarities = data.map {line = > val parts = line.split ("") (parts (0) .toLong Parts (1) .toLong, parts (2) .toDouble)} val pic = new PowerIterationClustering () .setK (2) / / set the number of clusters .setMaxIterations (10) / / set the number of iterations val model = pic.run (similarities) model.assignments.foreach {a = > println (s "${a.id}-> ${a.cluster}")}
Pic.txt
01 1.00 2 1.00 3 1.012 1.02 3 1.03 4 0.14 5 1.04 15 1.05 6 1.06 7 1.07 8 1.08 9 1.09 10 1.010 11 1.011 1.012 13 1.013 14 1.014 15 1.012
After reading this article, I believe you have a certain understanding of "how to achieve fast iterative clustering in spark mllib". If you want to know more about it, you are welcome to follow the industry information channel. Thank you for reading!
