EFK tutorial (3)-ElasticSearch Multi-instance deployment
Based on ElasticSearch multi-instance architecture, rational allocation of resources and separation of hot and cold data are realized.
Author: "Little Wolf". Welcome to reprint and contribute.
Catalogue
Use of ▪
▪ architecture
▪ 192.168.1.51 elasticsearch-data deployment of dual instances
▪ 192.168.1.52 elasticsearch-data deployment of dual instances
▪ 192.168.1.53 elasticsearch-data deployment of dual instances
▪ test
Use
Previously on:
In the first EFK tutorial-Quick start Guide, ▷ described the installation and deployment of EFK, in which the architecture of ES is three nodes, that is, the master, ingest, and data roles are deployed on three servers at the same time.
In the second EFK tutorial-ElasticSearch High performance and High availability Architecture, ▷ describes the use of EFK's data/ingest/master role and the deployment of three nodes to maximize performance while ensuring high availability.
In the first two articles, there is only one instance in the ES cluster, but in this article, multiple ES instances will be deployed in a cluster to achieve rational resource allocation. For example, there are SSD and SAS hard drives in data servers, which can store hot data to SSD and cold data to SAS to separate hot and cold data.
In this article, we will create two instances for the data server, one based on SSD and one based on SAS hard disk, and put the September index of nginx on SAS disk and the rest on SSD disk.
Architecture
Architecture diagram
Server configuration
192.168.1.51 elasticsearch-data deploys dual instances
Index migration (this step cannot be ignored): put the index on 192.168.1.51 on the other 2 data nodes
Curl-X PUT "192.168.1.31:9200/*/_settings?pretty"-H 'Content-Type: application/json'-d' {"index.routing.allocation.include._ip": "192.168.1.52192.168.1.53"}'
Confirm the current index storage location: verify that all indexes are not on the 192.168.1.51 node
Curl "http://192.168.1.31:9200/_cat/shards?h=n"
Stop the process of 192.168.1.51, modify the directory structure and configuration: please press SSD and SAS hard disk to mount the data disk.
# download and deploy the installation package, please refer to the first "EFK tutorial-Quick start Guide" cd / opt/software/tar-zxvf elasticsearch-7.3.2-linux-x86_64.tar.gzmv / opt/elasticsearch / opt/elasticsearch-SASmv elasticsearch-7.3.2 / opt/mv / opt/elasticsearch-7.3.2 / opt/elasticsearch-SSDchown elasticsearch.elasticsearch / opt/elasticsearch-*-Rrm-rf / data/SAS/*chown elasticsearch.elasticsearch / data/ *-Rmkdir-p / opt/logs/elasticsearch-SASmkdir-p / opt/logs/elasticsearch-SSDchown elasticsearch.elasticsearch / opt/logs/*-R
SAS instance / opt/elasticsearch-SAS/config/elasticsearch.yml configuration
Cluster.name: my-applicationnode.name: 192.168.1.51-SASpath.data: / data/SASpath.logs: / opt/logs/elasticsearch-SASnetwork.host: 192.168.1.51http.port: 9200transport.port: 930 discovery.seed_hosts and cluster.initial_master_nodes must be accompanied by port numbers Otherwise, you will use the http.port and transport.port ports discovery.seed_hosts: ["192.168.1.31 discovery.seed_hosts", "192.168.1.32 cluster.initial_master_nodes", "192.168.1.32", "192.168.1.33") cluster.initial_master_nodes: ["192.168.1.31", "192.168.1.32" "192.168.1.33 node.master 9300"] http.cors.enabled: truehttp.cors.allow-origin: "*" node.master: falsenode.ingest: falsenode.data: true# only allows 2 instances to be launched on this machine node.max_local_storage_nodes: 2
SSD instance / opt/elasticsearch-SSD/config/elasticsearch.yml configuration
Cluster.name: my-applicationnode.name: 192.168.1.51-SSDpath.data: / data/SSDpath.logs: / opt/logs/elasticsearch-SSDnetwork.host: 192.168.1.51http.port: 9201transport.port: 930 discovery.seed_hosts and cluster.initial_master_nodes must be accompanied by port numbers Otherwise, you will use the http.port and transport.port ports discovery.seed_hosts: ["192.168.1.31 discovery.seed_hosts", "192.168.1.32 cluster.initial_master_nodes", "192.168.1.32", "192.168.1.33") cluster.initial_master_nodes: ["192.168.1.31", "192.168.1.32" "192.168.1.33 node.master 9300"] http.cors.enabled: truehttp.cors.allow-origin: "*" node.master: falsenode.ingest: falsenode.data: true# only allows 2 instances to be launched on this machine node.max_local_storage_nodes: 2
Startup method of SAS instance and SSD instance
Sudo-u elasticsearch / opt/elasticsearch-SAS/bin/elasticsearchsudo-u elasticsearch / opt/elasticsearch-SSD/bin/elasticsearch
Confirm that 2 instances of SAS and SSD have been enabled
Curl "http://192.168.1.31:9200/_cat/nodes?v"
192.168.1.52 elasticsearch-data deploys dual instances
Index migration (this step cannot be ignored): put the index on 192.168.1.52 on the other 2 data nodes
Curl-X PUT "192.168.1.31:9200/*/_settings?pretty"-H 'Content-Type: application/json'-d' {"index.routing.allocation.include._ip": "192.168.1.51192.168.1.53"}'
Confirm the current index storage location: verify that all indexes are not on the 192.168.1.52 node
Curl "http://192.168.1.31:9200/_cat/shards?h=n"
Stop the process of 192.168.1.52, modify the directory structure and configuration: please press SSD and SAS hard disk to mount the data disk.
# download and deploy the installation package, please refer to the first "EFK tutorial-Quick start Guide" cd / opt/software/tar-zxvf elasticsearch-7.3.2-linux-x86_64.tar.gzmv / opt/elasticsearch / opt/elasticsearch-SASmv elasticsearch-7.3.2 / opt/mv / opt/elasticsearch-7.3.2 / opt/elasticsearch-SSDchown elasticsearch.elasticsearch / opt/elasticsearch-*-Rrm-rf / data/SAS/*chown elasticsearch.elasticsearch / data/ *-Rmkdir-p / opt/logs/elasticsearch-SASmkdir-p / opt/logs/elasticsearch-SSDchown elasticsearch.elasticsearch / opt/logs/*-R
SAS instance / opt/elasticsearch-SAS/config/elasticsearch.yml configuration
Cluster.name: my-applicationnode.name: 192.168.1.52-SASpath.data: / data/SASpath.logs: / opt/logs/elasticsearch-SASnetwork.host: 192.168.1.52http.port: 9200transport.port: 930 discovery.seed_hosts and cluster.initial_master_nodes must be accompanied by port numbers Otherwise, you will use the http.port and transport.port ports discovery.seed_hosts: ["192.168.1.31 discovery.seed_hosts", "192.168.1.32 cluster.initial_master_nodes", "192.168.1.32", "192.168.1.33") cluster.initial_master_nodes: ["192.168.1.31", "192.168.1.32" "192.168.1.33 node.master 9300"] http.cors.enabled: truehttp.cors.allow-origin: "*" node.master: falsenode.ingest: falsenode.data: true# only allows 2 instances to be launched on this machine node.max_local_storage_nodes: 2
SSD instance / opt/elasticsearch-SSD/config/elasticsearch.yml configuration
Cluster.name: my-applicationnode.name: 192.168.1.52-SSDpath.data: / data/SSDpath.logs: / opt/logs/elasticsearch-SSDnetwork.host: 192.168.1.52http.port: 9201transport.port: 930 discovery.seed_hosts and cluster.initial_master_nodes must be accompanied by port numbers Otherwise, you will use the http.port and transport.port ports discovery.seed_hosts: ["192.168.1.31 discovery.seed_hosts", "192.168.1.32 cluster.initial_master_nodes", "192.168.1.32", "192.168.1.33") cluster.initial_master_nodes: ["192.168.1.31", "192.168.1.32" "192.168.1.33 node.master 9300"] http.cors.enabled: truehttp.cors.allow-origin: "*" node.master: falsenode.ingest: falsenode.data: true# only allows 2 instances to be launched on this machine node.max_local_storage_nodes: 2
Startup method of SAS instance and SSD instance
Sudo-u elasticsearch / opt/elasticsearch-SAS/bin/elasticsearchsudo-u elasticsearch / opt/elasticsearch-SSD/bin/elasticsearch
Confirm that 2 instances of SAS and SSD have been enabled
Curl "http://192.168.1.31:9200/_cat/nodes?v"
192.168.1.53 elasticsearch-data deploys dual instances
Index migration (this step cannot be ignored): be sure to do this by placing the index on 192.168.1.53 on the other 2 data nodes
Curl-X PUT "192.168.1.31:9200/*/_settings?pretty"-H 'Content-Type: application/json'-d' {"index.routing.allocation.include._ip": "192.168.1.51192.168.1.52"}'
Confirm the current index storage location: verify that all indexes are not on the 192.168.1.52 node
Curl "http://192.168.1.31:9200/_cat/shards?h=n"
Stop the process of 192.168.1.53, modify the directory structure and configuration: please press SSD and SAS hard disk to mount the data disk.
# download and deploy the installation package, please refer to the first "EFK tutorial-Quick start Guide" cd / opt/software/tar-zxvf elasticsearch-7.3.2-linux-x86_64.tar.gzmv / opt/elasticsearch / opt/elasticsearch-SASmv elasticsearch-7.3.2 / opt/mv / opt/elasticsearch-7.3.2 / opt/elasticsearch-SSDchown elasticsearch.elasticsearch / opt/elasticsearch-*-Rrm-rf / data/SAS/*chown elasticsearch.elasticsearch / data/ *-Rmkdir-p / opt/logs/elasticsearch-SASmkdir-p / opt/logs/elasticsearch-SSDchown elasticsearch.elasticsearch / opt/logs/*-R
SAS instance / opt/elasticsearch-SAS/config/elasticsearch.yml configuration
Cluster.name: my-applicationnode.name: 192.168.1.53-SASpath.data: / data/SASpath.logs: / opt/logs/elasticsearch-SASnetwork.host: 192.168.1.53http.port: 9200transport.port: 930 discovery.seed_hosts and cluster.initial_master_nodes must be accompanied by port numbers Otherwise, you will use the http.port and transport.port ports discovery.seed_hosts: ["192.168.1.31 discovery.seed_hosts", "192.168.1.32 cluster.initial_master_nodes", "192.168.1.32", "192.168.1.33") cluster.initial_master_nodes: ["192.168.1.31", "192.168.1.32" "192.168.1.33 node.master 9300"] http.cors.enabled: truehttp.cors.allow-origin: "*" node.master: falsenode.ingest: falsenode.data: true# only allows 2 instances to be launched on this machine node.max_local_storage_nodes: 2
SSD instance / opt/elasticsearch-SSD/config/elasticsearch.yml configuration
Cluster.name: my-applicationnode.name: 192.168.1.53-SSDpath.data: / data/SSDpath.logs: / opt/logs/elasticsearch-SSDnetwork.host: 192.168.1.53http.port: 9201transport.port: 930 discovery.seed_hosts and cluster.initial_master_nodes must be accompanied by port numbers Otherwise, you will use the http.port and transport.port ports discovery.seed_hosts: ["192.168.1.31 discovery.seed_hosts", "192.168.1.32 cluster.initial_master_nodes", "192.168.1.32", "192.168.1.33") cluster.initial_master_nodes: ["192.168.1.31", "192.168.1.32" "192.168.1.33 node.master 9300"] http.cors.enabled: truehttp.cors.allow-origin: "*" node.master: falsenode.ingest: falsenode.data: true# only allows 2 instances to be launched on this machine node.max_local_storage_nodes: 2
Startup method of SAS instance and SSD instance
Sudo-u elasticsearch / opt/elasticsearch-SAS/bin/elasticsearchsudo-u elasticsearch / opt/elasticsearch-SSD/bin/elasticsearch
Confirm that 2 instances of SAS and SSD have been enabled
Curl "http://192.168.1.31:9200/_cat/nodes?v"
test
Move all indexes to the SSD hard drive
# the following parameters will be explained in a later article You can copy curl-X PUT "192.168.1.31:9200/*/_settings?pretty"-H 'Content-Type: application/json'-d' {"index.routing.allocation.include._host_ip": "," index.routing.allocation.include._host ":"," index.routing.allocation.include._name ":", "index.routing.allocation.include._ip": "" Index.routing.allocation.require._name: "*-SSD"}'
Verify that all indexes are on the SSD hard drive
Curl "http://192.168.1.31:9200/_cat/shards?h=n"
Migrate the log index of nginx9 month to SAS hard disk
Curl-X PUT "192.168.1.31:9200/nginx_*_2019.09/_settings?pretty"-H 'Content-Type: application/json'-d' {"index.routing.allocation.require._name": "*-SAS"}'
Confirm that the log index of nginx9 month is migrated to SAS hard disk.
Curl "http://192.168.1.31:9200/_cat/shards"
