[summary] how to deal with the situation when the filebeat process is full of disk

Using filebeat to collect logs, log files are frequently rotate, resulting in filebeat occupied files do not release, as long as filebeat keeps the deleted file Open state, the operating system does not release disk space, resulting in a gradual reduction of available disk space. The following monitoring diagram shows that after killing the filebeat process, the disk footprint drops sharply.

Using the lsof command to look at the file resources held by filebeat, you can find many invalid files (deleted) files that are occupied by filebeat.

Files in deleted state are not released and always occupy disk space

Solution:

View the filebeat configuration file location: / etc/filebeat/filebeat.yml

Add close_timeout: 5m to the configuration file to ensure that file handler is turned off every 5 minutes, regardless of whether or not the EOF symbol is encountered.

It is important to note that this close_timeout parameter will result in data loss if the Filebeat does not process to the end of the file and the file is deleted.

Filebeat.prospectors:

-type: log

Paths:

-/ opt/apps/ecm/service/storm/1.0.1/package/apache-storm-1.0.1/logs/workers-artifacts/xyz*/*/worker.log

Tail_files: false

Force_close_files: true

Close_timeout: 5m

Processors:

-add_cloud_metadata: ~

Output.logstash:

Hosts: ["10.10.10.10 6667"]

Loadbalance: true

Worker: 1

-

Configuration types such as close_rename,close_removed,close_eof,close_inactive are not valid for (deteled) state processing (tested and verified)