Shulou(Shulou.com)06/01 Report--

This article mainly explains the "MapReduce output to MySQL coding error solution", the article explains the content is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "MapReduce output to MySQL coding error solution" bar!

During the process of filling the search engine log results into mysql after counting with mapreduce, the following error occurred:

16-01-15 16:25:57 INFO mapreduce.Job: Task Id: attempt_1452318091100_0022_r_000000_0, Status: FAILED

Error: java.io.IOException: Incorrect string value:'\ xF0\ x9F\ x98\ x82 'for column' KEYWORD' at row 1

At org.apache.hadoop.mapreduce.lib.db.DBOutputFormat$DBRecordWriter.close (DBOutputFormat.java:103)

At org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close (ReduceTask.java:550)

At org.apache.hadoop.mapred.ReduceTask.runNewReducer (ReduceTask.java:629)

At org.apache.hadoop.mapred.ReduceTask.run (ReduceTask.java:389)

At org.apache.hadoop.mapred.YarnChild$2.run (YarnChild.java:164)

At java.security.AccessController.doPrivileged (Native Method)

At javax.security.auth.Subject.doAs (Subject.java:415)

At org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1657)

At org.apache.hadoop.mapred.YarnChild.main (YarnChild.java:158)

The database connection with mysql server code set to utf-8,mapreduce is: jdbc:mysql://host:port/database?useUnicode=true&characterEncoding=UTF-8,mysql driver: mysql-connector-java-5.1.38

According to the survey, the traditional UTF-8 is self-coded by 3, and the error here is a 4's own Unicode (words like smiley face:

?

Now, the database is defined as UTF-8, and when you try to store it with 3 bytes to hold 4 bytes, of course an error will occur.

You need to change the database to utf8mb4 encoding to accommodate these 4 own text. (MysQL version 5.5.3 or above is required, and jdbc driver 5.1.13 is required)

Because there is more than one database on the server, I try to change only the table code of the database I am using to utf8mb4, and the result is the same.

Then modify the my.cnf:

[mysqld] the same error still occurs after the character_set_server=utf8mb4 collation_server=utf8mb4_bin is modified. Try to restart mysql and run it again, successfully. It is speculated that the storage engine needs to be restarted to reflect the new code. Thank you for your reading, the above is the "MapReduce output to MySQL coding error solution" content, after the study of this article, I believe you have a deeper understanding of the problem of MapReduce output to MySQL coding error solution, the specific use of the situation also needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.