What is the method of python map reduce
This article mainly explains "what is the method of python map reduce". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what is the method of python map reduce"?
Execute the shell script run.sh
#! / bin/bashIN_DIR= "/ input/20140221" OUT_DIR= "/ output/20140221" HPHOME=$HADOOP_HOME/bin/JAR_PACKAGE=$HADOOP_HOME/contrib/streaming/hadoop-streaming-0.20.2-cdh4u0.jarMAP_FILE=$ (pwd) / map.pyRED_FILE=$ (pwd) / red.py$ {HPHOME} hadoop fs-rmr $OUT_DIR$ {HPHOME} hadoop jar $JAR_PACKAGE\-numReduceTasks 1\-jobconf mapred.job.queue.name=platform\-input $IN_ DIR\-output $OUT_DIR\-file $MAP_FILE\-file $RED_FILE\-mapper $MAP_FILE\-reducer $RED_FILE\-inputformat SequenceFileAsTextInputFormatexit $?
Map.py
#! / usr/bin/env pythonimport sysfor line in sys.stdin: line = line.strip () if line.find ("str1")! =-1: item = line.split ('\ 01') key = item [3] print'% s\ t% s'% (key 'str3') elif line.find ("str2")! =-1: item = line.split ('\ 01') key = item [2] value = item [5] print'% s\ t% s\ t% s'% (key, 'str3', value) else: pass
Red.py
#! / usr/bin/env pythonfrom operator import itemgetterimport sysword2count = {} total_count = 0total_money = 0.0total_error = 0pre_pid =''pre_money = 0.0is_pre_push = 0for line in sys.stdin: line = line.strip () if line.find (' str3')! =-1: item = line.split ('\ t') total_money + = float (item [2) ]) total_count + = 1 else: passprint total_count Total_error, total_money # # last print
For reference only
Hadoop uses'\ t'by default to separate key and value, and the first field is key
When it comes to reduce, the same key is processed together by the same red.py, so every row of the same key can be read continuously. (the same key in java is put in an array in a reduce, while python still reads multiple rows, but all of the same key are arranged together.)
At this point, I believe you have a deeper understanding of "what is the method of python map reduce". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!
