Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

Streaming executes Python version of WordCount

Shulou Source: shulou.com Published: 2022-06-03 06:14:40 11月23日 Update

One: write the map class first

Import sysfor line in sys.stdin:line = line.strip () words = line.split () for word in words:print ('% s\ t% s'% (word, 1))

Two: write reduce class

Import syscurrent_word = Nonecurrent_count = 0word = Nonefor line in sys.stdin:line = line.strip () word, count = line.split try:count = int (count) except ValueError:continueif current_word = = word:current_count + = countelse:if current_word:print ('% s\ t% s'% (current_word,current_count)) current_count = countcurrent_word = wordif current_word = = word:print ('% s\ t% s'%))

Third, use hadoop Streaming to execute the content of Python.

Hadoop jar / home/hadoop/hadoop-2.6.0-cdh6.5.2/share/hadoop/tools/lib/hadoop-streaming-2.6.0-cdh6.5.2.jar-input / user/hadoop/aa.txt-output / user/hadoop/python_output-mapper "python mapper.py"-reducer "python reducer.py"-file mapper.py-file reducer.py

Description:

The input and output paths, which are themselves on the hdfs, do not require a special hdfs.

If you do not add quotation marks in the × × part, an error will be reported:

Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads (): subprocess failed with code 2

If you don't add the pink part, an error will be reported:

Error: java.lang.RuntimeException: Error in configuring object

Tags: Content part error will report special that is quotation marks pink path input output Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno Shulou Information Shulou Tech Info MariaDB Microsoft NVidia