The third part of spark streaming testing is stateful received data.
Test ideas:
First of all, use the program of the previous article to send network data
Secondly, run the spark program to observe the effect.
Description:
1. The update function is used here
two。 Use checkpoints to ensure status.
SparkStreamingimport org.apache.log4j. {LoggerLevel} import org.apache.spark.streaming. {SecondsStreamingContext} import org.apache.spark. {SparkContextSparkConf} import org.apache.spark.streaming.StreamingContext._object StatefulWordCount {def main (args:Array []) {Logger. () .setLevel (Level.) Logger. () setLevel (Level.) UpdateFunc = (values: [] state:Option []) = > {currentCount = values.foldLeft () (_ + _) previousCount = state.getOrElse () (currentCount + previousCount)} conf = SparkConf (). SetAppName () .setMaster () sc = SparkContext (conf) ssc = StreamingContext (sc () ssc.checkpoint () lines = ssc.socketTextStream (args () args () toInt) words = lines.flatMap (_ .split) () wordCounts = words.map (x = > (x)) stateDstream = wordCounts.updateStateByKey [] (updateFunc) stateDstream.print () ssc.start () ssc.awaitTermination ()}}