Shulou(Shulou.com)06/01 Report--

This article shares with you the content of the sample code about flink batch dataset. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Package hgs.flink_lessonimport org.apache.flink.api.java.utils.ParameterToolimport org.apache.flink.streaming.api.scala._import org.apache.flink.api.scala.ExecutionEnvironmentimport org.apache.flink.core.fs.FileSystem.WriteModeimport org.apache.flink.api.common.accumulators.Accumulatorimport org.apache.flink.api.common.accumulators.IntCounterimport scala.collection.immutable.Listimport scala.collection.mutable.ListBufferimport scala.collection.immutable.HashMap//import StreamExecutionEnvironment.classobject WordCount {def main (args: Array [ String]: Unit = {val params = ParameterTool.fromArgs (args) / / 1. Get an execution environment and replace it with StreamExecutionEnvironment val env = ExecutionEnvironment.getExecutionEnvironment / / if it is Streaming. This will get the configuration env.getConfig.setGlobalJobParameters (params) println (params.get ("input")) println (params.get ("output")) val text = if (params.has ("input")) {/ / 2 in the current environment. Load or create initialization data env.readTextFile (params.get ("input"))} else {println ("Please specify the input file directory.") Return} println ("lines" + text.count ()) val ac = new IntCounter / / 3. Specify the operation type val counts = text.flatMap {_ .toLowerCase (). Split ("\\ W+"). Filter {_ .nonEmpty}} / / this is a little different from the groupBy of spark's operator. Here, an array of similar subscripts is used to determine what groups are based on. Map {(_ 1)} .groupBy (0) .reduceGroup (it= > {val tuple = it.next () var cnt = tuple. 2 val ch = tuple._1 while (it.hasNext) {cnt= cnt+it.next (). _ 2} (ch Cnt)}) / / indicate where to put the calculated data result / / 4.counts.print () counts.writeAsCsv ("file:/d:/re.txt", "\ n", "", WriteMode.OVERWRITE) / / 5. Trigger program executes env.execute ("Scala WordCount Example") / /}} Thank you for reading! This is the end of this article on "sample code for flink batch dataset". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.