Get the App
SLTechnology News&Howtos  ›  Internet Technology  › 

How Spark handles Chinese strings

Shulou Source: shulou.com Published: 2022-06-03 07:23:19 11月23日 Update

Recently, we need to use spark to extract Chinese characters. As the environment has always been English, it is found that the print is garbled. After studying, we need to do a UTF-8 conversion like this: val data_file = sc.textFile ("test.txt") val item1 = data_file.map (p = > new String (p.getBytes, 0, p.getLength, "UTF-8")) .map {item = > XXX}

Tags: UTF-8 characters Chinese garbled environment English work research string Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno Shulou Information Shulou Tech Info Linux Microsoft macOS