How Spark handles Chinese strings
Recently, we need to use spark to extract Chinese characters. As the environment has always been English, it is found that the print is garbled. After studying, we need to do a UTF-8 conversion like this: val data_file = sc.textFile ("test.txt") val item1 = data_file.map (p = > new String (p.getBytes, 0, p.getLength, "UTF-8")) .map {item = > XXX}