One of the spark streaming tests uses a local data source
Go straight to the code.
Let's talk about the test idea:
The / tmp/sparkStream/ directory that the code monitors
First, create the directory mkdir-p / tmp/sparkStream
Then, run the spark program
Finally, add data files to the monitoring directory / tmp/sparkStream/
Observe the running effect of spark program.
SparkStreamingimport org.apache.log4j. {LoggerLevel} import org.apache.spark.SparkConfimport org.apache.spark.streaming. {SecondsStreamingContext} import org.apache.spark.streaming.StreamingContext._object HdfsWordCount {def main (args: Array []) {Logger.getLogger ("org.apache.spark") .setLevel (Level.WARN) Logger.getLogger ("org.apache.eclipse.jetty.server") .setLevel (Level.OFF) sparkConf = SparkConf (). SetAppName (). SetMaster () ssc = StreamingContext (sparkConf ()) lines = ssc.textFileStream () words = lines.flatMap (_ .split () wordCounts = words.map (x = > (x)). ReduceByKey (_ + _) wordCounts.print () ssc.start () ssc.awaitTermination ()}}