Shulou(Shulou.com)06/03 Report--

When we want to customize the output, we first inherit two abstract classes, one is OutputFormat and the other is RecordWriter

. The former is mainly about creating RecordWriter, while the latter is mainly implementing the write method to write kv to a file.

1. Demand

Output the KV of the reduce, and if the key contains a specific string, output it to one file, and the rest of the KV to another file.

2. Source code

Source data

Http://cn.bing.comhttp://www.baidu.comhttp://www.google.comhttp://www.itstar.comhttp://www.itstar1.comhttp://www.itstar2.comhttp://www.itstar3.comhttp://www.baidu.comhttp://www.sin2a.comhttp://www.sin2a.comw.google.comhttp://www.sin2desa.comhttp://www.sin2desa.comw.google.comhttp://www.sina.comhttp://www.sindsafa.comhttp://www.sohu.com

OutputFormat

Public class MyOutputFormat extends FileOutputFormat {@ Override public RecordWriter getRecordWriter (TaskAttemptContext taskAttemptContext) throws IOException, InterruptedException {return new MyRecordWriter (taskAttemptContext);}}

RecordWriter

Public class MyRecordWriter extends RecordWriter {private FSDataOutputStream startOut; private FSDataOutputStream otherOut; public MyRecordWriter (TaskAttemptContext job) {try {FileSystem fs = FileSystem.get (job.getConfiguration ()); startOut = fs.create ("G:\\ test\\ date\ A\\ itstarlog\\ logdir\\ startout.log")) OtherOut = fs.create ("G:\ test\\ date\ A\\ itstarlog\\ logdir\\ otherout.log");} catch (IOException e) {e.printStackTrace ();}} @ Override public void write (Text key, NullWritable value) throws IOException, InterruptedException {String line = key.toString () / / if key contains itstar, write to another file if (line.contains ("itstar")) {this.startOut.writeUTF (line);} else {this.otherOut.writeUTF (line);}} @ Override public void close (TaskAttemptContext taskAttemptContext) throws IOException, InterruptedException {this.startOut.close (); this.otherOut.close ();}}

Mapper

Public class MyOutputMapper extends Mapper {@ Override protected void map (LongWritable key, Text value, Context context) throws IOException, InterruptedException {context.write (value, NullWritable.get ());}}

Reducer

Public class MyOutputReducer extends Reducer {Text k = new Text (); @ Override protected void reduce (Text key, Iterable values, Context context) throws IOException, InterruptedException {String line = key.toString (); line = line + "\ r\ n"; k.set (line); context.write (k, NullWritable.get ());}}

Driver

Ublic class MyDriver {public static void main (String [] args) throws IOException, ClassNotFoundException, InterruptedException {args = new String [] {"G:\\ test\\ date\ A\\ itstarlog\ A\\ other.log", "G:\ test\\ date\ A\\ itstarlog\\ logresult\"}; Configuration conf = new Configuration (); Job job = Job.getInstance (conf); job.setJarByClass (MyDriver.class) Job.setMapperClass (MyOutputMapper.class); job.setReducerClass (MyOutputReducer.class); job.setMapOutputKeyClass (Text.class); job.setMapOutputValueClass (NullWritable.class); job.setOutputKeyClass (Text.class); job.setOutputValueClass (NullWritable.class); / / implementation subclass of custom output, which also inherits FileOutputFormat job.setOutputFormatClass (MyOutputFormat.class); FileInputFormat.setInputPaths (job, new Path (args [0])) / / this path outputs the output path of job's successful successs file, FileOutputFormat.setOutputPath (job, new Path (args [1])); job.waitForCompletion (true);}}

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.