How to troubleshoot errors in Fluentd
This article will explain in detail how to troubleshoot errors in Fluentd. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.
View the log
If you feel that Fluentd is running abnormally, please check the log first. After td-agent is installed, the default log is stored in / var/log/td-agent/td-agent.log.
Open detailed log
Fluentd logs support multiple levels, and opening detailed logs can help you find problems. There are several ways to open detailed logs:
Set it through the log_level entry of the configuration file. For more information, please see the Fluentd log.
Use systemd user-level files
Add or modify configuration file / etc/systemd/system/td-agent
Add the following, which will overwrite the
Configuration.
[Service] ExecStart=...existing options... -vv
Modify init.d startup script
Edit / etc/init.d/td-agent, add "- vv" after TD_AGENT_OPTIONS, restart td-agent
# at / etc/init.d/td-agent...TD_AGENT_OPTIONS= "...-vv"...
Command line argument
$fluentd.. -vv
Export fluentd internal information
Fluentd uses sigdump (https://github.com/frsyuki/sigdump) to export internal information to a local file, and can send SIGCONT signals to the fluentd process to export the relevant information.
The reason for the high utilization of CPU
If you find that Fluentd occupies a high amount of CPU, you may have the following reasons:
A race problem occurred in a plug-in
Regular expression error
System call exception, such as inotify monitoring too many files
Some ruby dependency package exception
When these conditions occur, you can use the linux perf tool to detect the problem. You can also use the script pid2line.rb (https://gist.github.com/nurse/0619b6af90df140508c2) to assist with troubleshooting.
Check uncaptured logs
Sometimes you will encounter some abnormal exit situations, when the program returns a value of non-0. For example:
2016-01-01 00:00:00 + 0800 [info]: starting fluentd-0.12.282016-01-01 00:00:00 + 0800 [info]: reading config file path= "/ etc/td-agent/td-agent.conf" [... snip...] 2016-01-01 00:00:02 + 0800 [info]: process finished code=6
Such problems can occur within ruby, such as segment errors, some C extensions with bug, and so on. When fluentd runs as a daemon, we won't get the full log.
By running td-agent in the following way, you can get the complete log when the program is abnormal.
$sudo LD_PRELOAD=/opt/td-agent/embedded/lib/libjemalloc.so / usr/sbin/td-agent-c / etc/td-agent/td-agent.conf-user td-agent-group td-agent
About how to troubleshoot errors in Fluentd to share here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.
