Filebeat 6.4.3 collecting nginx logs
Filebeat 6.4.3 collecting nginx logs
Nginx log format specification:
Add a log configuration combination to the nginx main configuration file
Log_format eslog'$remote_addr-$remote_user [$time_local] "$request"'$status $body_bytes_sent "$http_referer"'"$http_user_agent"$http_x_forwarded_for"; ES cluster installs plug-ins
Ingest-user-agent and ingest-geoip need to be installed to collect nginx logs in the cluster. After the installation, you need to restart the es service, otherwise the data cannot be entered into the es cluster normally.
/ usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip/usr/share/elasticsearch/bin/elasticsearch-plugin list upgrade java to 1.8yum install java-1.8-y install filebeat Import key
Rpm-- import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Create a repo file
More lostash.repo
[elastic-6.x]
Name=Elastic repository for 6.x packages
Baseurl= https://artifacts.elastic.co/packages/6.x/yum
Gpgcheck=1
Gpgkey= https://artifacts.elastic.co/GPG-KEY-elasticsearch
Enabled=1
Autorefresh=1
Type=rpm-md
Installation
Yum install filebeat-y
Modify the filebeat main configuration file
More / etc/filebeat/filebeat.yml
Filebeat.config:
Prospectors:
Path: ${path.config} / prospectors.d/.yml
Reload.enabled: false
Modules:
Path: / etc/filebeat/modules.d/.yml
Reload.enabled: false
Output.elasticsearch:
Hosts: ['10.2.3.30mer 9200']
Setup.kibana:
Host: "10.2.3.13 5601"
Configure the filebeat nginx module Access log: / data/nginxlog/eslog/es-access.*Error log: / data/nginxlog/eslog/es-error.*more / etc/filebeat/modules.d/nginx.yml-module: nginx # Access logs access: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. Var.paths: ["/ data/nginxlog/eslog/es-access.*"] # Error logs error: enabled: true # Set custom paths for the log files. If left empty, # Filebeat will choose the paths depending on your OS. Var.paths: ["/ data/nginxlog/eslog/es-error.*"] add launch systemctl enable filebeat to open the nginx module cd / etc/filebeatfilebeat modules enable nginxfilebeat modules list initialization environment
This automatically imports the filebeat template and nginx dashboard into the es cluster:
Set up the initial environment:Loaded index templateLoading dashboards (Kibana must be running and reachable) Loaded dashboardsLoaded machine learning job configurations./filebeat setup-e startup service systemctl start filebeat startup service error
The contents are as follows:
Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory / usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory / usr/share/filebeat/kibana/6/index-pattern: error loading / usr/share/filebeat/kibana/6/index-pattern/filebeat.json: returned 400 to import file:. Response: {"statusCode": 400, "error": "Bad Request" "message": "Request Timeout after 30000ms"} 2018-10-31T16:35:45.659+0900 INFO kibana/client.go:113 Kibana url: http://10.2.3.13:56012018-10-31T16:37:15.664+0900 ERROR instance/beat.go:743 Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory / usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory / usr/share/filebeat/kibana/6/index -pattern: error loading / usr/share/filebeat/kibana/6/index-pattern/filebeat.json: fail to execute the HTTP POST request: Post http://10.2.3.13:5601/api/kibana/dashboards/import?force=true: net/http: request canceled (Client.Timeout exceeded while awaiting headers). Response: handling method:
Log in to kibana and enter Dev tools to delete.
Delete old template # DELETE _ template/filebeat-6.X.X delete old data # DELETE filebeat-6.4.2-* re-import template: cd / etc/filebeat/filebeat setupLoaded index templateLoading dashboards (Kibana must be running and reachable) Loaded dashboardsLoaded machine learning job configurations restart service: systemctl restart filebeatDashboard effect:
