Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

One of the spark streaming tests uses a local data source

Shulou Source: shulou.com Published: 2022-06-03 08:09:19 11月23日 Update

Go straight to the code.

Let's talk about the test idea:

The / tmp/sparkStream/ directory that the code monitors

First, create the directory mkdir-p / tmp/sparkStream

Then, run the spark program

Finally, add data files to the monitoring directory / tmp/sparkStream/

Observe the running effect of spark program.

SparkStreamingimport org.apache.log4j. {LoggerLevel} import org.apache.spark.SparkConfimport org.apache.spark.streaming. {SecondsStreamingContext} import org.apache.spark.streaming.StreamingContext._object HdfsWordCount {def main (args: Array []) {Logger.getLogger ("org.apache.spark") .setLevel (Level.WARN) Logger.getLogger ("org.apache.eclipse.jetty.server") .setLevel (Level.OFF) sparkConf = SparkConf (). SetAppName (). SetMaster () ssc = StreamingContext (sparkConf ()) lines = ssc.textFileStream () words = lines.flatMap (_ .split () wordCounts = words.map (x = > (x)). ReduceByKey (_ + _) wordCounts.print () ssc.start () ssc.awaitTermination ()}}

Tags: Directories code programs monitoring runs data tests ideas effects files observations data sources Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno macOS vpn NVidia Shulou Technology MySQL