Streaming executes Python version of WordCount
One: write the map class first
Import sysfor line in sys.stdin:line = line.strip () words = line.split () for word in words:print ('% s\ t% s'% (word, 1))
Two: write reduce class
Import syscurrent_word = Nonecurrent_count = 0word = Nonefor line in sys.stdin:line = line.strip () word, count = line.split try:count = int (count) except ValueError:continueif current_word = = word:current_count + = countelse:if current_word:print ('% s\ t% s'% (current_word,current_count)) current_count = countcurrent_word = wordif current_word = = word:print ('% s\ t% s'%))
Third, use hadoop Streaming to execute the content of Python.
Hadoop jar / home/hadoop/hadoop-2.6.0-cdh6.5.2/share/hadoop/tools/lib/hadoop-streaming-2.6.0-cdh6.5.2.jar-input / user/hadoop/aa.txt-output / user/hadoop/python_output-mapper "python mapper.py"-reducer "python reducer.py"-file mapper.py-file reducer.py
Description:
The input and output paths, which are themselves on the hdfs, do not require a special hdfs.
If you do not add quotation marks in the × × part, an error will be reported:
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads (): subprocess failed with code 2
If you don't add the pink part, an error will be reported:
Error: java.lang.RuntimeException: Error in configuring object