Shulou(Shulou.com)06/03 Report--

Test ideas:

First, use a network data sender to send data

Then, run the spark program

Observe the effect.

Description:

1. You also need to set the checkpoint directory here.

two。 Here are four parameters:

The first two are the listening port and how many milliseconds to receive data.

The third parameter is the number of milliseconds of data received; (for more information, please see the specific meaning of window)

The fourth parameter is how often data is received in milliseconds.

SparkStreamingimport org.apache.log4j. {LoggerLevel} import org.apache.spark.storage.StorageLevelimport org.apache.spark.streaming. {SecondsStreamingContext} import org.apache.spark. {SparkContextSparkConf} import org.apache.spark.streaming.StreamingContext._WindowWordCount {def main (args: Array []) {Logger. () .setLevel (Level.) Logger. () setLevel (Level.) Conf = SparkConf (). SetAppName (). SetMaster () sc = SparkContext (conf) ssc = StreamingContext (sc ()) ssc.checkpoint () val lines = ssc.socketTextStream (args (0), args (1). ToInt, StorageLevel.MEMORY_ONLY_SER) words = lines.flatMap (_ .split ()) / / windows operator val wordCounts = words.map (x = > (x.jin1)). ReduceByKeyAndWindow Seconds (args (2) .toInt), Seconds (args (3) .toInt) wordCounts.print () ssc.start () ssc.awaitTermination ()}}

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.