Use docker swarm to build EFK (elasticsearch, filebeat, kibana)
Elasticsearch installation
Elasticsearch.yml refers to the official document https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
Version: '3'services: elasticsearch: image: elasticsearch:7.4.2 restart: always ulimits: memlock: soft:-1 hard:-1 ports:-9200 networks:-logging volumes:-esdata1:/usr/share/elastcisearch/data -. / elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml environment:- "ES_JAVA_OPTS=-Xms512m-Xmx512m" volumes: esdata1: driver: localnetworks: logging: external: name: logging
The new version encountered two problems in the installation process.
1 the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
Need to create a new elasticsearch.yml file (https://github.com/elastic/elasticsearch/blob/master/distribution/src/config/elasticsearch.yml)
Modify node.name to be consistent with cluster.initial_master_nodes
# = = Elasticsearch Configuration = # # NOTE: Elasticsearch comes with reasonable defaults for most settings.# Before you set out to tweak and tune the configuration, make sure you# understand what are you trying to accomplish and the consequences.## The primary way of configuring a node is via this file. This template lists# the most important settings you may want to configure for a production cluster.## Please consult the documentation for further information on configuration options:# https://www.elastic.co/guide/en/elasticsearch/reference/index.html##-Cluster-- -# # Use a descriptive name for your cluster:#cluster.name: es-cluster##-Node-# # Use a descriptive name for the node:#node.name: "es-master "# # Add custom attributes to the node:##node.attr.rack: R1 Path to directory where to store the data #-Paths-# # Path to directory where to store the data (separate multiple locations by comma): # # ${path.data} # # Path to log files:##$ {path.logs} # #-Memory-# # Lock the memory on startup:##bootstrap.memory_lock: true## Make sure that the heap size is set to about half the Memory available# on the system and that the owner of the process is allowed to use this# limit.## Elasticsearch performs poorly when the system is swapping the memory.##-Network-# # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 0.0.0.0percent # Set a custom port for HTTP:##http.port: 9200percent # For more information Consult the network module documentation.##-- Discovery-- # # Pass an initial list of hosts to perform discovery when this node is started:# The default list of hosts is ["127.0.0.1" "[: 1]"] # discovery.seed_hosts: ["127.0.0.1", "[: 1]"] # # Bootstrap the cluster using an initial set of master-eligible nodes:#cluster.initial_master_nodes: ["es-master"] # # For more information Consult the discovery and cluster formation module documentation.##-- Gateway-- # # Block initial recovery after a full cluster restart until N nodes are started:##gateway.recover_after_nodes: 3 years # For more information Consult the gateway module documentation.##-- Various-- # # Require explicit names when deleting indices:##action.destructive_requires_name: truehttp.cors.enabled: truehttp.cors.allow-origin: /. * / 2 max virtual Memory areas vm.max_map_count [65530] is too low Increase to at least [262144]
When modifying / etc/sysctl.conf in the host, the ruqin of adding vm.max_map_count=262144filebeatfilebeat is smaller than that of flnent code, and there is no need to modify the relevant java projects that have been developed, and the memory footprint is small.
Docker-compose.yml
Version: '3'services: filebeat: image: elastic/filebeat:7.4.2 container_name: filebeat volumes: -. / filebeat.yml:/usr/share/filebeat/filebeat.yml restart: always networks:-logging deploy: replicas: 1networks: logging: external: name: logging
Filebeat.yml
Filebeat.inputs:- type: log paths:-/ var/lib/docker/containers/*/*.logoutput.elasticsearch: hosts: ["elasticsearch:9200"] kibana
Kibana does not have any tedious configuration, just specify ELASTICSEARCH_HOSTS
The docker-compose.yml configuration is as follows
Version: '3'services: kibana: image: kibana:7.4.2 ports:-5601 kibana 5601 networks:-logging environment: ELASTICSEARCH_HOSTS: http://elasticsearch:9200networks: logging: external: name: logging
