Shulou(Shulou.com)06/03 Report--

When writing Spark code, it is often found that rdd does not have a reduceByKey method, which occurs in spark1.2 and its previous versions, because rdd itself does not have a reduceByKey method and needs to be implicitly converted to PairRDDFunctions in order to access it, so Import org.apache.spark.SparkContext._ needs to be introduced.

However, when it comes to the version of spark1.3, the implicit conversion is placed in the object of rdd, so it will be introduced automatically and does not need to be explicitly introduced.

* Defines implicit functions that provide extra functionalities on RDDs of specific types. * For example, `RDD`.`rddToPairRDDFunctions` converts an RDD into a `PairRDDFunctions`PairRDDFunctions`key-value-pair RDDs, and enabling extra functionalities such as `PairRDDFunctions`.`reduceByKey`.* / object RDD {/ / The following implicit functions were in SparkContext before 1.3and users had to / / `import SparkContext._ `to enable them. Now we move them here to make the compiler find / / them automatically. However, we still keep the old functions in SparkContext for backward / / compatibility and forward to the following functions directly. Implicit def rddToPairRDDFunctions [K, V] (rdd: RDD [(K, V)]) (implicit kt: ClassTag [K], vt: ClassTag [V], ord: Ordering [K] = null): PairRDDFunctions [K, V] = {new PairRDDFunctions (rdd)}

As for what is implicit conversion, to put it simply, scala steals a beam for a post and lets Lao Wang next door do what you can't do.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.