[summary] Combiner practice in Hadoop

The function of Combiner is to merge the output of Mapper, and the output of Combiner is used as the input of Reducer, which reduces the data transfer between map task and reducer task.

1. Set Combiner and no Combiner in Job, and observe the input of Reducer

Set up Combiner using the following code

Job.setCombinerClass (MaxTemperatureReducer.class)

@ Override

Public int run (String [] args) throws Exception {

Job job = new Job ()

Job.setJarByClass (MaxTemperature.class)

Job.setJobName ("Max temperature")

FileInputFormat.addInputPath (job, new Path (args [0]))

FileOutputFormat.setOutputPath (job, new Path (args [1]))

Job.setMapperClass (MaxTemperatureMapper.class)

/ / job.setCombinerClass (MaxTemperatureReducer.class); whether to set Combiner

Job.setReducerClass (MaxTemperatureReducer.class)

Job.setOutputKeyClass (Text.class)

Job.setOutputValueClass (IntWritable.class)

Job.waitForCompletion (true)

/ / output task completion

System.out.println ("Task name:" + job.getJobName ())

System.out.println ("Task successful:" + (job.isSuccessful ()? "Yes": "No"))

System.out.println ("number of input lines:" + job.getCounters () .findCounter ("org.apache.hadoop.mapred.Task$Counter", "MAP_INPUT_RECORDS") .getValue ()

System.out.println ("output lines:" + job.getCounters () .findCounter ("org.apache.hadoop.mapred.Task$Counter", "MAP_OUTPUT_RECORDS") .getValue ())

System.out.println ("output lines:" + job.getCounters () .findCounter ("org.apache.hadoop.mapred.Task$Counter", "REDUCE_INPUT_RECORDS") .getValue ())

Return job.isSuccessful ()? 0: 1

}

2. The following is the output result without setting Combiner. The number of input lines of Reducer is equal to the number of output lines of Mapper.

Task name: Max temperature

Mission success: yes

Number of MAP_INPUT_RECORDS input lines: 1207

Number of MAP_OUTPUT_RECORDS rows: 1190

Number of REDUCE_INPUT_RECORDS rows: 1190

Mission start: 2015-04-24 14:26:00

End of mission: 2015-04-24 14:26:03

Task time: 0.04995 minutes

3. The following is the output result of setting Combiner. After Combiner, the number of input lines of Reducer is greatly reduced.

Task name: Max temperature

Mission success: yes

Number of MAP_INPUT_RECORDS input lines: 1207

Number of MAP_OUTPUT_RECORDS rows: 1190

Number of REDUCE_INPUT_RECORDS rows: 1

Mission start: 2015-04-24 14:28:23

End of mission: 2015-04-24 14:28:25

Task time: 0.030966667 minutes