Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to do the confusion of reduce statistical results summary". The explanation content in this article is simple and clear, and it is easy to learn and understand. Please follow the idea of Xiaobian to study and learn "how to do the confusion of reduce statistical results summary" together.

In hadoop 2.7.1 wrote a MapReduce program, do a word cloud statistics, program execution results have data, but chaos: many identical keywords are not merged together, there are individual merged together, but the aggregate value does not exceed 2, as follows:

China Red 1

China Red 1

China Red 1

Mocha Black 2

Mocha Black 2

Baidu said that most of the problems encountered by other colleagues are that reduce does not output results, the reason is that Reduce should use Iterable or Iterator, and it is still a problem in the 0.19 and 0.20 versions. And I have the output of the result, just not fully performing the summation. Reduce code is as follows:

static class MyReducer extends Reducer { LongWritable result = new LongWritable(); public void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritableval : values) { sum += val.get(); } result.set(sum); context.write(key, result); } }

Is there a problem with Reduce processing Chinese Key? Not likely. Then all keys are transcoded with URLEncoder, the result is still the same, can not be merged. Half a day, no results.

I woke up and suddenly found that there was IntWriteable in the program import. I once changed all Ints to LongWriteable. This program actually leaked! Look at the above code in the Class definition used LongWriteable, calculation time but use IntWriteable, modify, and then run, merge normal.

I see! Map and Reduce data types do not match usually there will be ClassCastException, but LongWriteable and IntWriteable conversion did not report an exception, silent calculation error, unexpectedly!

Thank you for reading, the above is the content of "Reduce statistical results summary chaos how to do," after the study of this article, I believe you have a deeper understanding of the problem of Reduce statistical results summary chaos, the specific use of the situation still needs to be verified by practice. Here is, Xiaobian will push more articles related to knowledge points for everyone, welcome to pay attention!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.