How to troubleshoot flume abnormal problems based on TBDS

This article mainly explains "how to troubleshoot flume abnormal problems based on TBDS". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to troubleshoot flume abnormal problems based on TBDS".

Phenomenon

During the long-term operation, it was found that the disk of the deployed flume cluster was full. After troubleshooting, it was found that the log directory of flume caused.

Specific problems

Specifically, if you look at flume's large file log, it is found that a certain MySQL-related sink continues to throw exceptions and prints a large number of logs.

Analysis process

According to this exception information (exception), namely:

Com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after statement closed

It literally means that when the state (connection) of the MySQL service has been closed, there is still a commit transaction operation and an exception is thrown, but the exception continues to be thrown and still needs in-depth analysis.

Configuration analysis

Since it was thrown by flume and related to MySQL, narrow down the problem and find out who is writing MySQL in flume. (the configuration of flume is generally located at / etc/flume/conf/agent/flume.conf)

The shutdown phase simply checks whether the connection exists.

Possible reasons

From the logic of sink, only in the case of empty connection, the sink state will be BACKOFF, and in other cases, the state will be READY. Moreover, the connection status will not be checked before and after the transaction is submitted to MySQL, and the sink status will not be modified even if SQL throws an exception. As a result, the sink loop executes and throws an exception after the commit throws an exception. This is the root of constantly throwing exceptions. So when exactly did the connection close? There are two reasons: (1) sink does not interact with MySQL for a long time, which exceeds the automatic closing time of the connection; (2) the abnormal shutdown of MySQL.

Problem confirmation

Whether sink has no interaction with MySQL for a long time

The timeout configuration for querying MySQL is as follows:

It can be seen that the disconnection between sink and MySQL is not without interaction for a long time.

Whether the service is artificially disconnected

The query starts MySQL artificially at the following time:

It's a good timing.

Conclusion

The MySQL service exception causes the connection to be broken when flume commits the transaction, and flume does not handle this exception, causing an endless loop to commit the transaction, and in this case, flume can no longer work properly.

Problem recurrence

Based on the above inference, this exception can be verified as follows:

HiveServer generates logs

Execute HiveSQL multiple times in HUE

Manually force the shutdown of MySQL

Manually restart the MySQL instance written by flume.

View flume performance

Flume enters an infinite loop and throws an exception, which is verified successfully.

Thank you for your reading, the above is the content of "how to troubleshoot flume abnormal problems based on TBDS". After the study of this article, I believe you have a deeper understanding of how to troubleshoot flume abnormal problems based on TBDS, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!