Shulou(Shulou.com)06/03 Report--

I heard my friend say a topic yesterday, but I forgot the specific one! The data goes like this:

To get the result, try to look like this:

1 22 23 34 15 1

The statistics of the data on the left, the de-weighting of the data on the right; when the left is the same, the right is the same, the record is once; when the left is the same, but the right is different, the number of data on the left is cumulative; when the left is different, the right is also different, the data on the left is cumulative.

After understanding the carelessness, I found that this is a small test for the de-repetition of the data! The train of thought does not write, follow the code random reverie, the code is limited to the above situation:

Package com.amir.test;import java.io.IOException;import java.util.Iterator;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapred.FileInputFormat;import org.apache.hadoop.mapred.FileOutputFormat;import org.apache.hadoop.mapred.JobClient;import org.apache.hadoop.mapred.JobConf;import org.apache.hadoop.mapred.MapReduceBase Import org.apache.hadoop.mapred.Mapper;import org.apache.hadoop.mapred.OutputCollector;import org.apache.hadoop.mapred.Reducer;import org.apache.hadoop.mapred.Reporter;public class MapReducer_MulTask {public static class Ma***emovingMap extends MapReduceBase implements Mapper {private Text line = new Text (); public void map (Object key, Text value, OutputCollector output, Reporter reporter) throws IOException {line = value Output.collect (line, new Text (""));}} public static class Ma***emovingReduce extends MapReduceBase implements Reducer {public void reduce (Text key, Iterator value, OutputCollector output, Reporter reporter) throws IOException {output.collect (key, new Text ("")) }} public static class StatisticsMap extends MapReduceBase implements Mapper {private final static IntWritable one = new IntWritable (1); private Text word = new Text (); public void map (Object key, Text value, OutputCollector output, Reporter reporter) throws IOException {StringTokenizer itr = new StringTokenizer (value.toString ()) While (itr.hasMoreTokens ()) {String [] temp = itr.nextToken (). Split (","); String akey = temp [0] .replace ("

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.