Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

The third part of spark streaming testing is stateful received data.

Shulou Source: shulou.com Published: 2022-06-03 07:07:41 11月23日 Update

Test ideas:

First of all, use the program of the previous article to send network data

Secondly, run the spark program to observe the effect.

Description:

1. The update function is used here

two。 Use checkpoints to ensure status.

SparkStreamingimport org.apache.log4j. {LoggerLevel} import org.apache.spark.streaming. {SecondsStreamingContext} import org.apache.spark. {SparkContextSparkConf} import org.apache.spark.streaming.StreamingContext._object StatefulWordCount {def main (args:Array []) {Logger. () .setLevel (Level.) Logger. () setLevel (Level.) UpdateFunc = (values: [] state:Option []) = > {currentCount = values.foldLeft () (_ + _) previousCount = state.getOrElse () (currentCount + previousCount)} conf = SparkConf (). SetAppName () .setMaster () sc = SparkContext (conf) ssc = StreamingContext (sc () ssc.checkpoint () lines = ssc.socketTextStream (args () args () toInt) words = lines.flatMap (_ .split) () wordCounts = words.map (x = > (x)) stateDstream = wordCounts.updateStateByKey [] (updateFunc) stateDstream.print () ssc.start () ssc.awaitTermination ()}}

Tags: Programs data status testing previous articles functions ideas effects articles checkpoints networks guarantees updates checks observations runs Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno MariaDB Shulou Tech Info MySQL Shulou Technology vpn